Exploring WebSockets and Evergreen III : Proof of Concept

We’ve seen what Websockets are and how their persistent connections could provide considerable speed benefits and reduction of network traffic.  In this post, I’ll introduce some proof of concept code so that we can bring the discussion out of the world of theory and into the world of practice.

All of the code I’m discussing today lives in the OpenSRF working repository.;a=shortlog;h=refs/heads/collab/berick/websockets

What do we need to start experimenting with Websockets in Evergreen?

1. An OpenSRF Websocket gateway

2. An OpenSRF Javascript Websocket client library

1. The Gateway

First, a little history…

The code library to use for the server gateway was a point of debate when we discussed this at the developer’s meeting in September.  One option is to create the Websocket gateway using Apache.  There are a number of reasons to take this approach, but Apache uses a lot of RAM, which is a problem when managing large numbers of long-lived connections.  An alternative solution is to use a standalone Websocket server library, like libwebsockets [].  The hope of a standalone server is that it will have a much smaller memory footprint.

Coming into this, I had a strong preference for using Apache.  Every Evergreen administrator knows Apache and my fear was that adding a new technology stack to the existing mix of technology used by Evergreen would inhibit adoption.  Also, Apache has been around forever, it’s stable, and comes with a number of goodies out of the box, like access control, request/environment inspection/munging, data compression tools, etc.,

Finally, in regard to comparing Apache to Libwebsockets specifically, Libwebsockets does not operate as a forking server.  It was developed, I presume, mainly to support threaded applications.  However, the OpenSRF C library is not thread safe, nor would a single (per brick) process be sufficient to relay messages to/from all connected clients without congestion. To use libwebsockets, we’d have to implement a forking / process management layer.  I was not particularly interested in tackling this bit of code just for my proof of concept experiments.

Though I was eager to use Apache for the gateway, I had a hard time convincing myself it was worth it if Apache required too much RAM to be a viable option in the long run.  To tackle this problem, I considered ways to reduce the amount of RAM Apache used out of the box and ultimately decided to try running Apache with no modules loaded and no services running.  It turns out, Apache by itself is very lightweight.  After some experimentation, I was able to get a Websocket gateway running successfully on Apache with only 3 modules loaded: mime, SSL, and mod_websocket.

For comparison, how does the trimmed down Apache stack up to a single-process Libwebsockets implementation?  The image below shows memory information for both implementations.  What you are seeing is the baseline footprint.  Neither implementation (at this point) has been taught to communicate over OpenSRF.  They only have enough code to understand Websockets.


(RSS is in kilobytes)

Libwebsockets beats the Apache baseline memory usage by 92K.  However, as I mentioned before, libwebsockets does not have any process control built in, so that would require the addition of a decent chunk of new code and memory consumption from inter-process communications (pipes, etc.), which is already accounted for in Apache.  Even from a memory perspective, if Apache doesn’t win outright (and I think it might), it at least ties libwebsockets, and probably any other websockets library.

In case you’re wondering, the last Apache entry in the screenshot, with the higher memory footprint, is the parent process, which requires more RAM to manage open pipes to all of its child processes, etc.

This was all I needed to continue developing the gateway code for Apache.  To take advantage of the reduced memory footprint created by disabling all of those Apache modules, we have to run a secondary, trimmed-down Apache instance to host the Websocket gateway.  Otherwise, each Websocket connection (i.e. Apache process) would be sitting atop 30+ Apache modules and a large pile of RAM.  Running a secondary Apache instance is not exactly ideal, but it seems that hosting a Websocket gateway is going to require running some other process, regardless.  Running a secondary instance of a service for which we already have a solid grasp seems like the lesser of evils.  Also, running a secondary Apache instance is apparently not that uncommon.  (For example, Debian includes a script to automate the setup).  For reference, here are my current install notes:;a=blob;f=README.websockets;h=c9f2391d40857e8764ad162ec8afa3e92ce7df8e;hb=HEAD

Now that we have a baseline, let’s add support for OpenSRF communication.  The addition of OpenSRF and its dependencies (e.g. libxml2) is a fixed cost incurred regardless of the underlying technology.

What is the RAM picture after Apache is taught how to handle OpenSRF websockets requests?

This screenshot shows the memory consumption of a set of trimmed down Apache websocket processes alongside a set of regular Evergreen Apache processes.  All processes shown at this point have handled several requests, so we are seeing their operating RAM levels, not the RAM usage at startup. The websocket processes use less than 1/15th the RAM of a regular Evergreen process.

As an aside, we should consider reviewing the default Apache modules used for Evergreen.  There may be some room to remove a few modules and regain some memory.

For a websocket gateway, we’re still talking about needing more RAM, right?  Not necessarily.  Managing API traffic through a separate set of processes means it would no longer necessary to use the 50+MB Apache processes to handle translator requests.  (Or, at least, it would be much less common).  Those large processes would be limited to more traditional web activities like serving files and generating web content (e.g. TPAC).  At any given time, many, if not most, of the active Apache processes today are handing translator requests.  Eliminating those processes opens up more RAM space for running the websocket Apache instance.

As a data point, if the average websocket process consumes 4MB of RAM, one thousand staff connections would require roughly 4G of RAM.  Of course, a sane admin would add a good bit of overflow RAM, but you get the idea.

2. The Client

Javascript already supports Websockets, so this code simply needs to be able to open a Websocket connection, send OpenSRF requests over the connection, then translate Websocket responses into OpenSRF responses (and potentially other types of out-of-band messages, like broadcasts) and pass them back to the caller.;a=commit;h=aab6f533c8b63ae19c49df0661bb0ec35c58bf5e

The API is practically identical to the existing XMLHttpRequest/Translator API.

var ses = new OpenSRF.ClientSession('');ses.request({method : 'opensrf.system.echo',params : ['hello', 'world'],onresponse : function(r) {var result = r.recv().content();alert(result);},oncomplete : function() {// all responses received}}).send();

The only real difference is that all requests are required to be asynchronous, so onresponse/oncomplete handlers are required for receiving responses.

It’s important to note that because Websockets are always asynchronous, we cannot simply replace the Evergreen communication layer with Websockets.  There are numerous instances of synchronous communication used in Evergreen today and each would have to be modified to work properly in a fully asynchronous environment.

Experimenting With the Code

I’ve set up a few tests and so far it’s all very promising.  I can successfully communicate with the Evergreen server, manage sets of open conversations among multiple services, and manage stateful/connected sessions.  To be clear, though, the code, installer process, and documentation as they exist today are alpha quality at best.  I have not attempted to integrate the code into the staff client in any meaningful way, so it only works today in the browser.  Much testing and smoothing is required before we could leverage the code, but that doesn’t mean it can’t be installed and run by some adventurous coders.

I look forward to testing the code more in the coming days and creating some benchmark scripts for comparing Websockets to XMLHttpRequest in the context of Evergreen and OpenSRF.  When I have data to share, I may post an epilogue to this series.  As always, feel free to ping me in IRC (berick), here, or the Evergreen developer mailing list if you have any questions about the code or my approach to Websockets in general.