Since Try Monad, Progress and Lockdown Rules and In the meanwhile, I wrote an interesting (well I thought so) blog about Enterprise System Integration.

This is having spent the last two years of my life integrating (in more ways than one!). It was an interesting reflection, perhaps therapeutic for me even.

Apart from that, I came across an interesting scenario in some source code recently. The scenario is this:

• There is a set number of database connections available for each database i.e it has a connection pool.
• There are multiple consumers of the connection pool. Each consumer can fork off multiple threads and each one can go off and access the database.
• Each thread that accesses the database would consume one of these connections.
• The scenario has multiple databases, i.e shards and each database has its own connection pool of x connections.

The following code was set up to control/limit the number of threads that can access a shard, such that only x threads can do so i.e only n threads for the connection pool's n connections in the connection pool.

This following specifies that for whichever database shard you're (a thread) is trying to connect to, there is a mutex you can own to isolate that access(and thus db connection) to your thread only. There are as many mutexes per shard as there are db connections in the connection pool.

Once it has a mutex it can lock it out for the duration of its database access (yield).

In code:

@@shard_mutex[shard].sample.synchronize do
        ::Multidb.use(shard) do
          yield if block_given?
        end
end

Basically, it's limiting how many threads can connect at once, as to not run out of database connections. 

The reason I was in this code was that it was deadlocking. This is the hand I was dealt. Also look at the way it randomly provides mutexes to threads to acquire, via the sample function... 

Then I remembered something "Acquire multiple locks in a fixed, global order". This sage advice comes from "Seven Concurrency Models in Seven Weeks" by Butcher which is staring down at me as I write this... But its also described here

And that when I started to realise that the above code randomly distributes mutex locked in no particular order.

There is also a re-entrant issue: If the block that is yielded somehow calls the above code again i.e it's recursive, then you'll get a deadlock - because you might get the same mutex as the one you originally obtained (a sample), only this time you'll be waiting for yourself(the first call which obtained the lock) to release the lock... deadlock 101.

I decided to try it without assigning mutexes randomly to threads and also not have threads potentially waiting on each other (two threads trying to access the same shard). I ended up with a function called wait_for_logical_connection() which drives a thread-safe Queue which blocks thread access to it when it's empty.

shard_result = wait_for_logical_connection shard do
        ::Multidb.use(shard) do
          yield if block_given?
        end
end

Basically, behind wait_for_logical_connection (shown below), this uses a Queue called shard_thread_leases which will block all threads if this queue is empty.

Its no longer is an array of x mutexes(we have x connections in the pool) per shard (which seemed a bit excessive to me to begin with) but is now an array of Queues - one for each shard.

Each thread wanting access to a shard can pick up a lease (an integer, not a mutex) from the front of the queue, and then it can do its database work and give the lease back. This is put back on the back of the queue and now will free up any thread waiting on the queue(if it's empty at this point).

No thread now obtains a mutex, just a number. The Queue handles blocking threads trying to obtain a number(lease). One mutex - the queue.

  def wait_for_logical_connection(shard)
    logger.debug "Waiting to access db on shard: #{shard}, thread leases left: #{@@shard_thread_leases[shard].size})"
    lease = @@shard_thread_leases[shard].pop # threads wait until more leases are available
    result = nil
    begin
      logger.debug  "Performing db access on shard: #{shard}, thread leases left: #{@@shard_thread_leases[shard].size})"
      result = yield
    ensure
     @@shard_thread_leases[shard].push lease # give back lease (add to back of queue)
    end
    result
  end

It now uses an ordered shard connection leases queue instead of possibly sharing mutexes.

The internals of the queue favours a simple lease system of an integer instead of a mutex:

@@shard_thread_leases ||= begin
        c = ThreadSafe::Cache.new
        pool_size = ::ActiveRecord::Base.connection.pool.instance_variable_get(:@size) || 10
        ::Multidb.main_shards.each do |s|
          c[s] = Queue.new # Limit of logical connections is based on limit on physical database connections.
          1.upto(pool_size) { |i| c[s] << i  } # A logical connection lease is object that can be checked in/out  
        end
        c
      end

Any thread can release its connection lease to the pool allowing any other threads (waiting on the pool) to be unblocked.

Any thread that does not release its connection lease(for whatever reason) will not hold up the remaining threads that are waiting. i.e threads wait for the queue(to have more leases), and not specifically individual threads holding a shared mutex which other threads need to be released first.

Each thread fetches a 'lease' of the queue and puts it back once it's done accessing the database. There are no more mutexes - only the queue is responsible for blocking threads which it cannot 'pop' or provide a lease to a thread wanting access to the db. If so whatever reason the calling code passing in a lambda that calls this code again, one of the other threads that returns a lease will unlock it.  

I shall sleep well tonight!

Enterprise system integration is the bringing together of heterogeneous components within an enterprise for them to collaborate to realise and contribute to business goals.

Broadly, models of integration can be distinguished into those that affect physical integration (e.g. hardware), application integration (interoperability of applications across heterogeneous platforms) and business integration, with the view of process integration. ({xbref tag="0" title="Panetto and Cecil, 2013" desctext="Panetto, H., Cecil, J., 2013. Information systems for enterprise integration, interoperability and networking: theory and applications. Enterprise Information Systems: Information Systems for Enterprise Integration, Interoperability and Networking: Theory and Applications 7, 1–6. https://doi.org/10.1080/17517575.2012.684802" }Panetto and Cecil, 2013¹⁵{/xbref})

{xbref tag="0" title="Volkoff et al. (2005)" desctext="Volkoff, O., Strong, D.M., Elmes, M.B., 2005. Understanding enterprise systems-enabled integration. European Journal of Information Systems: Special Issue: Making Enterprise Systems Work 14, 110–120. https://doi.org/10.1057/palgrave.ejis.3000528" }Volkoff et al. (2005)¹⁵{/xbref} identifies degrees of which integration is achieved such as Total, Unification and Federated and can further categorized into types of integration i.e. data, context and process integration. This research also mentions how the relationship and dependencies between integrating participants also typifies integration.

Key technologies centre around the concept of interoperability of Enterprise information systems (EIS), where co-ordination and communication play pivotal roles. Analysis of systems, modelling, architecture, compatibility, co-ordination, communication and systems’ interdependences are also key concerns.

Designs and implementation of loosely coupled systems are idealised through collaborative technologies such as APIs, Web services and other distributed, platform-agnostic technologies such as CORBA/RPC etc.

Architecture such as Service Orientated Architectures (SOA) and Model-driven development exemplify the need to model “reusable actives are services”. ({xbref tag="0" title="Jardim-Goncalves et al., 2013" desctext="Jardim-Goncalves, R., Grilo, A., Agostinho, C., Lampathaki, F., Charalabidis, Y., 2013. Systematisation of Interoperability Body of Knowledge: the foundation for Enterprise Interoperability as a science. Enterprise Information Systems: Information Systems for Enterprise Integration, Interoperability and Networking: Theory and Applications 7, 7–32. https://doi.org/10.1080/17517575.2012.684401" }Jardim-Goncalves et al., 2013¹⁵{/xbref})

Data modelling technologies such as UML and organisational modelling technologies such as UEML/QVT/MOF coupled with network design help provide visibility during analysis of integration projects.

Application integration technologies usually centre around common services such as shared databases and standardised data and include technologies such as JEE/.NET that are used to implement interoperable designs such as web services, standard data format exchange (using XML, WSDL etc) and services across distributed processing environments.

The overarching benefits of integration usually is the standardization of work and data, operational efficiencies and reduced cost. This can be seen through the convergence of functionality, unification of best practises, higher data accuracy, better collaboration between participants and reduction in manual integration effort and overall process time. (Volkoff et al., 2005)

This research also indicates that difficulties usually stem from the degree of dissimilarity between integrating participants and centre around interoperability issues around compatibility differences in format of data, semantics and business processes. These differences can include social implications around orientation towards different integration goals, and the impact the integration has on the organisation at large, including inter-personal orientation and the effect that personal workarounds (in the face of integration) has on downstream systems.

Other difficulties include the cost of integration, the impact that a break in the chain of inter-connected services have on functionality downstream and the lack of consistencies between models from different environments.

Other issues cited are increased interdependence/coupling on system on each other, the increased rigidity of integrated processes and the consequences that a break in the flow has on downstream participants and processes.

An interesting observation is that while interoperability results in compatibility, it’s not necessarily the other way around, nor is a maximum level of interoperability necessarily an optimal one. (Jardim-Goncalves et al., 2013)

In my experience, and as taught in this module, analysis, design and architecture play key roles in integration projects. Particularly the need and emphasis on analysis and architecture such as the need to synchronise analysis models with implementation models when change occurs (e.g. MDD) and how Software engineering also strives for similar goals to integration – common, shared data and reuse of functionality. (Joannou, 2007)

I was able to reach home after 2 failed attempts at running back home over the last week. I managed only to do about half of my planned running. I found that about halfway through the run my tummy started to act up - I was bloated and really uncomfortable. I had to stop and walk back. Still, half is still something.

Thankfully on Saturday, I was able to go out and finish my run, so whatever was up with my tummy is alright now. I managed to do about 10km which was good considering, and should be able to get back into the swing of things.

There is certainly something to be said about the feeling of getting out vs not getting out i.e I find its much better to just even to make it out than to think you did not even get that. A large portion of success is turning up even if that means walking back the rest of the way home :-). I think that this is an interesting aspect of psychology.

I spend the rest of the weekend formulating a research essay that is due in September I think. It requires a summary of 3 papers and an accompanying critical review. I've found that when it comes to essays, the best thing you can do is read things and then leave a period of time to reflect, percolate if you will. The other is to make notes on the first pass. I'd read the 3 papers and made my notes well in advance of me putting pen to paper (or rather, in this new age: fingers to keyboard). 

A large frustration at times to choosing what not to write about because ultimately there is always more to discuss and it comes down to selecting things that you think are more important than others. I always enjoyed creative writing as school though I'm not sure I was any good at it. I still enjoy it. 

I've been investigating the impact of upgrading our database migration process and I've found it really useful to document my findings and results in a wiki. I find this gives me the confidence to sensibly forget things (externalize) so that I can concentrate on the new tasks at hand. I'm then able to refer back to what my findings suggested without 2nd-guessing myself as to what the result was or the situation/context was. I believe that my recent study and focus on research has helped me in this way a great deal and I suspect it will again in the future. Another great thing about it is that you can plan your next steps or predict your next steps as you're updating your current results/work.

It's also great to be given autonomy to 'do whatever it takes to get it done'. I seem to do quite well at those types of projects.

I ventured out for a bit of a walk today and being summer, it's still wonderfully sunny and pleasant. People are still wearing masks but a lot of them I'm sensing are feeling the novelty of it slowly wearing off because many were not wearing masks. Still compulsory however in shopping centres. I wear mine as soon as I leave my apartment. I also think people find it reassuring say If I enter the same lift as them, though I try to avoid this.

I've also started to grow my beard again. It's awesome, I love it.

I started once last year but it got scratchy and I hadn't used beard oil. This time I have and its a lot more 'wavy' and comfortable. Though at times, it does look a little mismanaged and I'm hoping that this is just the intermediary phase I'm in between having a sort of a beard to having a full-on one!

I've got to get myself up to scratch on how I'll need to groom it - going to the barbers in town is far too expensive now that travelling isn't huge on my priority list.