Over the years middleboxes have become a fundamental part of today’s networks. Despite their usefulness, they come with a number of problems, many of which arise from the fact that they are hardware-based: they are costly, difficult to manage, and their functionality is hard or impossible to change, to name a few. To address these issues, there is a recent trend towards network function virtualization (NFV), in essence proposing to turn these middleboxes into software-based, virtualized entities. Towards this goal we introduce ClickOS, a high-performance, virtualized software middlebox platform. ClickOS virtual machines are small (5MB), boot quickly (about 30 milliseconds), add little delay (45 microseconds) and over one hundred of them can be concurrently run while saturating a 10Gb pipe on a commodity server. We further implement a wide range of middleboxes including a firewall, a carrier-grade NAT and a load balancer and show that ClickOS can handle packets in the millions per second.
The graph above reports throughput results for various middleboxes. Overall, ClickOS performs well, achieving almost line rate for all configurations for 512-byte and larger packets (the BRAS and CG-NAT middleboxes have rates slightly below the 2.3 Mp/s line rate figure). For smaller packet sizes the percentage of line rate drops, but ClickOS is still able to process packets in the millions/second. To get and idea of how this relates to a real-world traf- fic matrix, compare this to an average packet size of 744 bytes reported a study done on a tier-1 OC192 (about 10Gb/s) backbone link: if we take our target to be packets of around this size, all middleboxes shown can sustain line rate. Naturally, some of these middleboxes fall short of being fully functional, and different configurations (e.g., a large number of firewall rules) would cause their performance to drop from what we present here. Still, we believe these figures to be high enough to provide a sound basis upon which to build production middleboxes. The carrier-grade NAT, for instance, is proof of this: it is fully functional, and in stress tests it is still able to handle packets in the millions/second.
Building the ClickOS image is not trivial. MiniOS is intended to be
built with standard GCC and as such we can link any standard C library
to it, provided we handle for missing functionality
appropriately. However, Click is written in C++, and so it requires
special precautions. We developed a new build tool which creates a
Linux-independent C++ cross-compiler using
newlibc. Our build tool
redesigns the standard MiniOS toolchain so that it is possible to
quickly and easily build arbitrary, MiniOS-based VMs by simply linking
an application’s entry point so that it starts on VM boot; this is
useful for supporting middleboxes that cannot be easily supported by
Click. Regarding libraries, we have been conservative in the number of
them we link, and have been driven by need rather than
experimentation. In addition to the standard libraries provided with
the out-of-the-box MiniOS build (lwip, zlib, libpci) we add support
for libpcre, libpcap and libssl, libraries that certain Click elements
We developed our own Domain-0 tool called
cosmos. Cosmos is similar
xl and is built directly on top of the Xen UI libraries. To
simplify development and user interaction,
cosmos implements a SWIG
wrapper, enabling users to automatically generate bindings for
any of the SWIG supported languages. For convenience, we have also
implemented a Python-based ClickOS CLI.
In order to reduce domain creation times even further, Cosmos can also
be built on top of our own domain management library called
being less complex and by eliminating some rarely used Xenstore
entries, this library makes the creation process much faster compared
libxl. However, this is done at the cost of having a less generic
toolstack; as a result, we also provide some patches to accelerate
We carried out a few modifications to Click in order to have it run in the MiniOS environment. For example, we added a new target to Click for MiniOS, and a tiny glue, around 300 lines of code, that connects Click to MiniOS and exposes the element handlers and middlebox initialization. We further included a couple of new Click elements to act as Xen virtual network adapters.
|Middlebox||Key Click Elements|
Key Click elements available for developing a wide range of middleboxes.
Click comes with over 300+ stock elements which make it possible to construct middleboxes with minimal effort (see table above). It is also extensible, so we are not limited to the functionality provided by the stock elements. We can easily extend this framework and introduce new elements for new classes of packet processing. Click is of course no panacea: it does not cover all types of middlebox processing, for instance middleboxes that need a full-fledged TCP stack. In such cases, it is better to use a standard Linux VM. It is worth mentioning that while MiniOS represents a difficult development environment, programming for ClickOS is relatively painless: development, building and testing can take place in user-space Click, and the resulting code/elements simply added to the ClickOS build process when ready.