A while back I started reading an interesting book on software architectural design patterns. The broker patterns caught my interest because it looks pretty clever, its about distributed components which is interesting and there happens to be a team within my company that works on a 'broker' which I'm interested to know more about.

Also at the same time I started putting together a C library of useful functions, some code of which were BSD socket functions and data structure implementations. So I thought, "why not build a fairly fully functional representation of the broker pattern in C and at the same time build my library to support it?". So I did - what a cool project.

Lots of lessons learnt.

You can see a nice pictorial representation of the components in the broker pattern here

This is a broker written in C using the broker pattern. The communications are through tcp sockets using an "on-the-wire" data format called msgpack which is similar to Json which serves as a common protocol between the components.

This is the design behind the broker pattern: https://en.wikipedia.org/wiki/Broker_Pattern] And here is a nice picture that shows the components involved in the pattern

To see a picture of the components of the broker pattern see here.

You can browse the full code here

The solution is broken down into the following components, as seen above.

Broker

The idea here is that this component listens for incoming socket connections from either clients or servers and routes the connections between the two, allowing neither of the two components to 'know' the location of each other and they all rely on the broker to 'broker' the connections to the correct destination.

This is the essence of what the broker pattern tries to achieve - lochttp://devel.stuartmathews.com/projects/broker/repository/revisions/master/entry/server/server.cation independence. The broker pattern is used in designing distributed components that need to communicate but offloads managing the communication to the broker component - the client and the server are usually the components that try to talk to each other. Both register with the broker by providing their location details and the broker. Each component (client/server) talks to just the broker that it has registered with. When sending a request for a service say on a server that is registered with the broker, the client will send the request to the broker, the broker will send it on to any server that has registered with the broker(and having specified what services it can fulfil). If a server goes down, the client knows nothing about the server and simply asks the broker to service the request and the broker will do so to another registered server that is capable of servicing that request. The request is sent to the next best server, the response is sent back to the broker and the broker sends it back to the requesting client. This also allows failing components not to bring down others because they are all isolated from each other.

This broker is started from the command line and utilizes stulibc(another c library I've written) to set-up and manage this. The broker starts by setting up a listening socket and then waits for servers to register themselves or clients to make requests for function on those registered servers. The broker keeps a list of in-memory clients, servers and their supported operations.

You can browse the broker code here

Server proxy

This is an extension of the server, which is responsible for communicating with the broker.

While the main server component would contain the code that represents the services(the actual code implementation) that it represents and that client would want to call(like RPC).So it basically determines what operation needs to be called and calls it, and returns the response to the broker.

The interesting thing about this component, is that the code is generated from a set of function declarations that the server will support, and from that generates the code that can call into those functions (its a bit of magic really). It marshals the data(the required function, parameters) then arranges for the actual call to be called in the main server. The server proxy is generated from a server_interface.h file, using a perl script, and this proxy will be the code that retrieves the message for the function form the broker, executes the actual implementation the server.c and sends the response back to the broker. The perl script is here: 

Server

These are the functions that the server implements(and which the client ultimately wants to call) and also which the server proxy will ultimately call on the server component when a connection comes into the broker for that function. Remember: The broker sends the request to the server proxy first and then the server proxy calls the server component.

You can see how it does this here

Client

This is the client that attempts to calls the server function, it does so by sending a message to the broker (via its client-proxy - see below) to say that it want to call that function(on any server that the broker knows can service the function request) with the provided parameters.

Its actually the client-proxy actually constructs the protocol message to that effect, sends it to the broker and waits from a response, upon which it gives this back to the client.

The client just calls function1(int a, int b) say, and client proxy will generate a protocol message, with the parameters included that represents that request to call that function to the broker. The return of this function is the result that the client-proxy receives from the broker.The client-proxy is the 'fake' implementation of the function, and the client doesn't know it - it just things that the client request was fulfilled entirely by the client proxy. The client-proxy actually asked the broker to fulfil it by way of sending the protocol message to the broker that represents a 'service- request'

You can see how the client does all this here

Client Proxy

The client proxy is the brains behind the client just like the server-proxy is the brains behind the server.In both cases these parts actually do the communication and request for their counterparts to the broker.

So in a way it contains the fake-calls that the client calls, but they are actually an implementation that requests that call from the broker(si the client-proxy delegates to the broker) and when the broker

Client Proxy

returns its value, the client-proxy returns the result to this function to the client - as if all the code was executed/done on the client-proxy but really it was the client-proxy that asked the broker to process it and get a response for it!

Similarly to the server, the client proxy is generated from the server_interface.h. You can have a look at the autogenerating c-code generator here

Limitations

I've not implemented the bridge component. I also haven't tried two different server implementations registered with the same broker. Its quite possible that it works, but without namespace clashing support for the server functions, two servers might have the same function and the broker, without some change wouldn't know wher eto send the requests/responses to!

I've pretty much used libmsgpack to store the protocol messages, and I've not made any attempt to genericise the procotol message format, other than put it into "common" functions that both the client/proxy/server can access. In theory its just this common stuff that would need to be changed to support a new protocol message format.

The parts of this project that I enjoyed the most were:

• Protocol design for one was particularly interesting. This is basically using libmsgpack to store the protocol messages and then I send them using BSD sockets via stulibc.
  
  • Have a look at the protocol here
  • The implementation of which is here (function declarations) and here
• The TCP/IP code for the BSD sockets implementation was cool.
  • Its used in the broker here, in the client here and the server here
• The auto generation of C-code via Perl was particularly fun and inspiring.
  • Generate the client-proxy and the server-proxy
• The usage of various data structures and implementation of the C library i wrote was cool.
  • Linked list used by the broker to store registered servers (register_service.c) and clients (register-client_request.c)