A while back I started reading an interesting book on software architectural design patterns. The broker patterns caught my interest because it looks pretty clever, its about distributed components which is interesting and there happens to be a team within my company that works on a 'broker' which I'm interested to know more about.

Also at the same time I started putting together a C library of useful functions, some code of which were BSD socket functions and data structure implementations. So I thought, "why not build a fairly fully functional representation of the broker pattern in C and at the same time build my library to support it?". So I did - what a cool project.

Lots of lessons learnt.

You can see a nice pictorial representation of the components in the broker pattern here

This is a broker written in C using the broker pattern. The communications are through tcp sockets using an "on-the-wire" data format called msgpack which is similar to Json which serves as a common protocol between the components.

This is the design behind the broker pattern: https://en.wikipedia.org/wiki/Broker_Pattern] And here is a nice picture that shows the components involved in the pattern

To see a picture of the components of the broker pattern see here.

You can browse the full code here

The solution is broken down into the following components, as seen above.

Broker

The idea here is that this component listens for incoming socket connections from either clients or servers and routes the connections between the two, allowing neither of the two components to 'know' the location of each other and they all rely on the broker to 'broker' the connections to the correct destination.

This is the essence of what the broker pattern tries to achieve - lochttp://devel.stuartmathews.com/projects/broker/repository/revisions/master/entry/server/server.cation independence. The broker pattern is used in designing distributed components that need to communicate but offloads managing the communication to the broker component - the client and the server are usually the components that try to talk to each other. Both register with the broker by providing their location details and the broker. Each component (client/server) talks to just the broker that it has registered with. When sending a request for a service say on a server that is registered with the broker, the client will send the request to the broker, the broker will send it on to any server that has registered with the broker(and having specified what services it can fulfil). If a server goes down, the client knows nothing about the server and simply asks the broker to service the request and the broker will do so to another registered server that is capable of servicing that request. The request is sent to the next best server, the response is sent back to the broker and the broker sends it back to the requesting client. This also allows failing components not to bring down others because they are all isolated from each other.

This broker is started from the command line and utilizes stulibc(another c library I've written) to set-up and manage this. The broker starts by setting up a listening socket and then waits for servers to register themselves or clients to make requests for function on those registered servers. The broker keeps a list of in-memory clients, servers and their supported operations.

You can browse the broker code here

Server proxy

This is an extension of the server, which is responsible for communicating with the broker.

While the main server component would contain the code that represents the services(the actual code implementation) that it represents and that client would want to call(like RPC).So it basically determines what operation needs to be called and calls it, and returns the response to the broker.

The interesting thing about this component, is that the code is generated from a set of function declarations that the server will support, and from that generates the code that can call into those functions (its a bit of magic really). It marshals the data(the required function, parameters) then arranges for the actual call to be called in the main server. The server proxy is generated from a server_interface.h file, using a perl script, and this proxy will be the code that retrieves the message for the function form the broker, executes the actual implementation the server.c and sends the response back to the broker. The perl script is here: 

Server

These are the functions that the server implements(and which the client ultimately wants to call) and also which the server proxy will ultimately call on the server component when a connection comes into the broker for that function. Remember: The broker sends the request to the server proxy first and then the server proxy calls the server component.

You can see how it does this here

Client

This is the client that attempts to calls the server function, it does so by sending a message to the broker (via its client-proxy - see below) to say that it want to call that function(on any server that the broker knows can service the function request) with the provided parameters.

Its actually the client-proxy actually constructs the protocol message to that effect, sends it to the broker and waits from a response, upon which it gives this back to the client.

The client just calls function1(int a, int b) say, and client proxy will generate a protocol message, with the parameters included that represents that request to call that function to the broker. The return of this function is the result that the client-proxy receives from the broker.The client-proxy is the 'fake' implementation of the function, and the client doesn't know it - it just things that the client request was fulfilled entirely by the client proxy. The client-proxy actually asked the broker to fulfil it by way of sending the protocol message to the broker that represents a 'service- request'

You can see how the client does all this here

Client Proxy

The client proxy is the brains behind the client just like the server-proxy is the brains behind the server.In both cases these parts actually do the communication and request for their counterparts to the broker.

So in a way it contains the fake-calls that the client calls, but they are actually an implementation that requests that call from the broker(si the client-proxy delegates to the broker) and when the broker

Client Proxy

returns its value, the client-proxy returns the result to this function to the client - as if all the code was executed/done on the client-proxy but really it was the client-proxy that asked the broker to process it and get a response for it!

Similarly to the server, the client proxy is generated from the server_interface.h. You can have a look at the autogenerating c-code generator here

Limitations

I've not implemented the bridge component. I also haven't tried two different server implementations registered with the same broker. Its quite possible that it works, but without namespace clashing support for the server functions, two servers might have the same function and the broker, without some change wouldn't know wher eto send the requests/responses to!

I've pretty much used libmsgpack to store the protocol messages, and I've not made any attempt to genericise the procotol message format, other than put it into "common" functions that both the client/proxy/server can access. In theory its just this common stuff that would need to be changed to support a new protocol message format.

The parts of this project that I enjoyed the most were:

• Protocol design for one was particularly interesting. This is basically using libmsgpack to store the protocol messages and then I send them using BSD sockets via stulibc.
  
  • Have a look at the protocol here
  • The implementation of which is here (function declarations) and here
• The TCP/IP code for the BSD sockets implementation was cool.
  • Its used in the broker here, in the client here and the server here
• The auto generation of C-code via Perl was particularly fun and inspiring.
  • Generate the client-proxy and the server-proxy
• The usage of various data structures and implementation of the C library i wrote was cool.
  • Linked list used by the broker to store registered servers (register_service.c) and clients (register-client_request.c)

I’m currently reading this book: http://shop.oreilly.com/product/9780596005269.do?sortby=bestSellers which is basically an old version of “C# In a Nutshell”. It’s still relevant even though there is like 4 updates to the book. I got it from the bookshelf at the office and funnily enough I actually just opened it and started reading it and well, I ‘got hooked’. So I’ve been reading it while waiting for the bus in the evening and on the bus in the morning. Yesterday I discovered some very interesting aspects of c#: usage of pointers such as those found in C. Very cool. My ultimate goal is to start reading the section on interop because I’d like integrate my C Library into a WPF application (C#) and while I get there I’m enjoying the process of getting there….

I worked a bit more on adding logging to my broker, so now if you want to set the loglevel=0, you just get log0() messages, if you set the loglevel=1, you’ve increased the loglevel and so, you get log0() and log1() calls meaning you get more detailed logging. This makes sense because you;ve increased the logging level and thus log1() is more verbose than log0().

I had a good weekend this weekend. The most obvious thing about this weekend was that I designed a logging mechanism for my broker component. The whole idea is to log at different levels and allow the configuration to specify which levels you’d like to show. So If you want high-level logs you’d say so and the other detailed logs will not be shown. So you add different logging calls such as log0(),log1(), or whatever and you can turn those logging instructions off depending on what you tell the program to log.

Anyway, I spend the whole day doing it and then while standing in the shower afterwards, realised that I’d over engineered it and replaced it with a simple solution in 5 minutes! I was thinking, “Surely there has to be an easier way than this way I’m doing?”. I wonder if I’d not over engineered it, would I have come up with a similar solution or could I have just started there(perhaps I should have started there?). On the plus side was I implemented a function that uses va_list as a argument which allows me to pass in variable parameters to other functions, which in itself was a carry over from my complex design which as it happens was necessary to implement the simpler design – so not all lost!

This is a print statement very much like vsprintf which allows you to pass in a va_list, and it will print out all the characters and replace those that need replacing with the variable parameters indicated by the va_list parameter :

void VPRINT(char*msg, va_list argp)	
{	
	for(char *p=msg;*p;p++) {	
		if(*p != '%') { putchar(*p); continue; }	
		switch(*++p) {	
			case 'd':	
				printf("%d", va_arg(argp,int));	
				break;	
			case 's':	
				printf("%s", va_arg(argp,char*));	
		}
	
	}
	
}

This can only interpret %s and %d for strings and integer parameters passed as the additional args to the VPRINT() function.

Which now can be called like this from as variadic function, and use the extra parameters itself to do whatever it want to do with them:

void log0(int loglevel, char* message, ...)
{
	if(loglevel > 0)
	    return;
	char *buffer;
	asprintf(&buffer, "log0: %s\t%s", TIM_GetDateTimeStr(),message);
	va_list argp;
	va_start(argp, buffer);
		VPRINT(buffer, argp);
	va_end(argp);
	free(buffer);
	buffer = 0;	
}

So that was my Saturday wiped out. Its not usual that I do that, but I was feeling inspired to work on my broker because I found new inspiration when looking at the proxy-generation routine which I wrote in perl(I look at it now and go huh? what?), and I was like – hmm that's actually a pretty cool thing I created. What it is, is you can generate a bunch on c-functions from a defined list of function declarations so you can arrange to call one of the generated functions dynamically when the protocol message states that you should do, and you usher with the provided parameters into the call – so you need the call to exist, so i generate it automatically each time I add a new function to the interface. Here is a typical protocol message that can be sent to the server, to which the server needs to call the function with the params in the message:

{ "request-type" : SERVICE_REQUEST }
{ "message-id": 3456789 }	
{ "sender-address" : "127.0.0.1" }
{ "reply-port": 8090 }
{ "op"=>"getServerDate" }
{ "params" => [ buffer, length, ... ] }

This allows me to add functions to the interface and then have the proxy-generation script read those and generate the function bodies and then arrange for the parameters passed over the wire to be loaded into those function call invocations. This kind of stuff you just take for granted with C# and WSDL.

So when I get told that I need to run the getServerDate() function with the parameters, I can call that function dynamically – this isn’t as easy in C as in a higher language that does OOP and reflection easily.

I also decided to go to the gym on Sunday, Not bad eh.