Why Erlang Gateways? - Part 3

After protocol decoding and client implementation, the third major challenge was making sure the gateways were scalable. Each gateway instance must handle a large number of active client sessions and multiple instances must be clusterable using some form of session-based load balancing.

Every heysan user has a jabber account that is registered with one or more gateways. Upon login the jabber server sends a presence probe to each gateway which then connects the user to the appropriate legacy network. A client session consists three Erlang processes that communicate via message passing: socket reader, FSM, and translator of protocol-specific events into a common format. This style of concurrency oriented programming is very quite and elegant, but is not possible in many languages due to the overhead of native OS threads.

Erlang's SMP VM distributes runnable processes across all available CPUs. IM clients spend most of their time waiting on network IO, so the gateways run with the +Ktrue command line option which instructs the VM to use epoll/kqueue/poll rather than select() to efficiently determine which processes are runnable. We have yet to explore the limits of a single gateway instance, but our most popular network has reached > 700 active clients and load on the gateway server was very low.

Of course at some point a single gateway instance will become constrained by the finite amount of CPU power, RAM, IO bandwidth, etc available on a single server. So the gateways have been designed to support multiple instances running in a cluster. Persistent data such as user credentials are stored in OTP's distributed database mnesia and accessible from every instance. Our jabber server, ejabberd, provides built-in support for load-balancing sessions across multiple instances of a gateway. Bringing a new instance online is a simple matter of telling it to join an existing cluster and updating the ejabberd config file.

So, in the final analysis, Erlang/OTP served as an ideal platform for building jabber gateways. Which isn't particularly surprising since such a system isn't far from Erlang's original telcom applications, and ejabberd has already proven itself in large deployments. In upcoming posts I intend to discuss some of Erlang's other libraries as well as some of the difficulties encountered due to MSNP v15's extensive use of SOAP and XML.

Why Erlang Gateways? - Part 2

Erlang's bit syntax provides a very convenient method of decoding and encoding binary protocols like OSCAR and YMSG. MSNP is a textual protocol that is also easily parsed with gen_tcp's {packet, line} and {active, once} options. Each protocol implementation has a process which reads incoming packets, does preliminary decoding, and forwards the data to a finite state machine.

The language and VM are great, but one of the major benefits to writing programs in Erlang is OTP which provides a large standard library and design principles that have been developed and tested in real-world use. One example of this is gen_fsm, a generic behavior for event-based FSMs which is particularly useful for implementing network protocols. The gen_fsm module handles initialization, synchronous and asynchronous event delivery, error reporting, debugging, etc. All the programmer must do is implement a callback module which handles events and makes state change decisions.

Execution of a gen_fsm begins with a call to gen_fsm:start or gen_fsm:start_link, which invokes the callback module's init/1 function. init performs any necessary initialization and determines the FSM's starting state. Here is msnp_fsm:init/1:

init([Username, Password, Client, Opts]) ->
 process_flag(trap_exit, true),

 Host     = proplists:get_value(host, Opts, ?MSNP_HOST),
 Port     = proplists:get_value(port, Opts, ?MSNP_PORT),

 State = #state{client   = Client,
                user     = #user{username = Username, password = Password},
                contacts = [],
                pending  = dict:new(),
                sessions = []},
 gen_fsm:send_event(self(), {connect, Host, Port}),
 {ok, protocol_negotiation, State}.

The FSM begins in state protocol_negotiation, and immediately receives an asynchronous event {connect, Host, Port} telling it to connect to the server. Next, gen_fsm will call msnp_fsm:protocol_negotiation/2, passing it the event and FSM state:

protocol_negotiation({connect, Host, Port}, State) ->
  {ok, Sock} = msnp_sock:start_link(self(), Host, Port),
  msnp_sock:send_cmd(Sock, "VER", ["MSNP15", "CVR0"]),
  {next_state, protocol_negotiation, State#state{sock = Sock}};

The FSM remains in protocol_negotiation until it receives a matching VER command from the server, then transitions through sso_auth, waiting_for_profile, synchronizing, and eventually to the ready state. Each state function is quite simple, and with multiple function clauses to handle different events the code is easy to read. Here are two ready/1 event handlers that handle contact status changes:

ready(#cmd{cmd = "ILN", args = [_, Code, Name, _, Nick | _]}, State) ->
   State2 = send_status_update(Name, url_util:decode(Nick), Code, State),
   {next_state, ready, State2};

ready(#cmd{cmd = "NLN", args = [Code, Name, _, Nick | _]}, State) ->
   State2 = send_status_update(Name, url_util:decode(Nick), Code, State),
   {next_state, ready, State2};

msnp_sock parses incoming packets and sends the resulting event to the msnp_fsm as a cmd record. Erlang's pattern matching provides a very concise way to determine which clause to invoke and binds the parameters needed to variables such as Name and Nick.

OTP behaviors like gen_fsm also provide much useful debugging functionality. When an event handler fails for any reason a log message is generated with the FSM's state, the last message received, the cause, etc. gen_fsm:start_link can also be called with the option {debug, [trace]} which will log every incoming event and state change. Running systems can be inspected in real-time thanks to functions such as sys:get_status/1 which displays the state of running processes.

With the combination of Erlang's bit-syntax and OTP library it is pretty easy to implement an IM client. Indeed most of the difficulty is due to lack of official documentation and incomplete reverse engineering. In the next post I'll cover one of the most interesting and important aspects of the gateways: scalability.

Why Erlang Gateways? - Part 1

Jabber gateways (XEP-0100, also known as transports) provide a bridge from XMPP to legacy IM network protocols like OSCAR, used by AIM and ICQ; YMSG, used by Yahoo Messenger; and MSNP, used by MSN/Windows Live Messenger. Unfortunately these protocols are all proprietary and lacking comprehensive documentation. AOL has released partial specs for parts of OSCAR but neglected to include the sections necessary for ICQ compatibility, and there is an obsolete draft RFC for MSNP. Both of those protocols have been reverse engineered and documented fairly well, but there are no open source implementations suitable for use as jabber gateways. The various Python transports (PyAIMt, PyMSNt, PyYIMt) appear fairly unmaintained and libpurple's API is better suited to clients than gateways, not to mention supporting very outdated versions of MSNP and YMSG.

Fortunately Erlang is an excellent language for implementing network protocols! In this post I'll cover one reason: a data type and syntax for manipulating binary data. The data type is called a binary (which as of R12B is a subset of a new type, bitstring) and stores bytes, very similar to a byte array in C, C++, Java, etc. However, unlike a byte array, the bit syntax provides access to an arbitrary number of bits. Let's jump right into an example using the syntax for a FLAP header which frames all OSCAR packets:

<<16#2A:8, Chan:8, Seq_Num:16, Size:16>>

The FLAP header is 6 bytes long. Byte 1 is an asterisk (character code 0x2A), byte 2 is a channel number, bytes 3-4 are a sequence number, and bytes 5-6 contain the size of the remaining payload. When this Erlang expression is matched against an incoming FLAP packet it assigns the channel number to Chan, sequence number to Seq_Num, and size to Size. Size:16 is actually a shortcut, taking advantage of bit syntax defaults, for Size:16/big-unsigned-integer which means a 16 bit unsigned integer, encoded as big-endian.

Most OSCAR packets are sent on channel 2 which indicates another level of framing called SNAC. When a FLAP header arrives on channel 2 we read Size bytes, which start with a SNAC header:

<<Family:16, Subtype:16, Flags:16, Req_Id:32, Bin/binary>>

This looks quite similar to FLAP, except for the last part. Until now we've only been pulling integers of various size from of the binary, but the bit syntax also supports floats and binaries. When matched against a SNAC packet, this expression stores the first 10 bytes in Family, Subtype, Flags, and Req_Id, and all remaining bytes in Bin. The values of Family and Subtype determine the format of the data in Bin.

So far we've been looking at the bit syntax patterns used to decompose binary data. What does the data actually look like? Well it's just a sequence of bytes, which Erlang displays as a comma-separated list of numeric values surrounded by << and >> This is easier to demonstrate using another common OSCAR data type, string16, which is a UTF-8 string, prefixed with a 2 byte length.

Here is what the string "hello world" would look like as a string16 binary: <<0,11,104,101,108,108,111,32,119,111,114,108,100>>. The first two bytes are the length, and the remaining bytes are UTF-8 character codes. When matched against the pattern

<<Len:16, Str:Len/binary>>

Len will contain 16 and Str will contain <<"hello world">>, which is a shortcut for <<104,101,108,108,111,32,119,111,114,108,100>> (Erlang does not have a string data type, but sequences of Latin-1 characters can be enclosed in quotes in both lists and binaries).

These examples of the bit syntax have been taken from my actual OSCAR implementation, but are only a small part of the puzzle. In the next post I'll discuss modeling protocols using finite state machines, via Erlang's gen_fsm behavior. Here's a teaser snippet:

ready(#packet{family = 16#13, subtype = 16#19, data = Data}, State) ->
  <<Len:8, Contact:Len/binary, _Rest/binary>> = Data,
  State#state.client ! {incoming_contact_add, Contact},
  {next_state, ready, State};

Welcome!

Hello, and welcome to Erlang at Work! My name is Will Glozer and in this blog I'll be describing my use of Erlang in production applications at heysan. Heysan is a mobile, web-based, instant messaging client supporting the AIM, Google Talk, ICQ, MSN, and Yahoo networks as well as chat rooms and instant messaging between heysan users. These features are powered by ejabberd and IM network gateways that I wrote in Erlang.