Fast and Lightweight Network Function Virtualization



Over the years middleboxes have become a fundamental part of today’s networks. Despite their usefulness, they come with a number of problems, many of which arise from the fact that they are hardware-based: they are costly, difficult to manage, and their functionality is hard or impossible to change, to name a few. To address these issues, there is a recent trend towards network function virtualization (NFV), in essence proposing to turn these middleboxes into software-based, virtualized entities. Towards this goal we introduce ClickOS, a high-performance, virtualized software middlebox platform. ClickOS virtual machines are small (5MB), boot quickly (about 30 milliseconds), add little delay (45 microseconds) and over one hundred of them can be concurrently run while saturating a 10Gb pipe on a commodity server. We further implement a wide range of middleboxes including a firewall, a carrier-grade NAT and a load balancer and show that ClickOS can handle packets in the millions per second.

The graph above reports throughput results for various middleboxes. Overall, ClickOS performs well, achieving almost line rate for all configurations for 512-byte and larger packets (the BRAS and CG-NAT middleboxes have rates slightly below the 2.3 Mp/s line rate figure). For smaller packet sizes the percentage of line rate drops, but ClickOS is still able to process packets in the millions/second. To get and idea of how this relates to a real-world traf- fic matrix, compare this to an average packet size of 744 bytes reported a study done on a tier-1 OC192 (about 10Gb/s) backbone link: if we take our target to be packets of around this size, all middleboxes shown can sustain line rate. Naturally, some of these middleboxes fall short of being fully functional, and different configurations (e.g., a large number of firewall rules) would cause their performance to drop from what we present here. Still, we believe these figures to be high enough to provide a sound basis upon which to build production middleboxes. The carrier-grade NAT, for instance, is proof of this: it is fully functional, and in stress tests it is still able to handle packets in the millions/second.


Building the ClickOS image is not trivial. MiniOS is intended to be built with standard GCC and as such we can link any standard C library to it, provided we handle for missing functionality appropriately. However, Click is written in C++, and so it requires special precautions. We developed a new build tool which creates a Linux-independent C++ cross-compiler using newlibc. Our build tool redesigns the standard MiniOS toolchain so that it is possible to quickly and easily build arbitrary, MiniOS-based VMs by simply linking an application’s entry point so that it starts on VM boot; this is useful for supporting middleboxes that cannot be easily supported by Click. Regarding libraries, we have been conservative in the number of them we link, and have been driven by need rather than experimentation. In addition to the standard libraries provided with the out-of-the-box MiniOS build (lwip, zlib, libpci) we add support for libpcre, libpcap and libssl, libraries that certain Click elements depend on.


We developed our own Domain-0 tool called cosmos. Cosmos is similar to xl and is built directly on top of the Xen UI libraries. To simplify development and user interaction, cosmos implements a SWIG wrapper, enabling users to automatically generate bindings for any of the SWIG supported languages. For convenience, we have also implemented a Python-based ClickOS CLI.

In order to reduce domain creation times even further, Cosmos can also be built on top of our own domain management library called xcl. By being less complex and by eliminating some rarely used Xenstore entries, this library makes the creation process much faster compared to libxl. However, this is done at the cost of having a less generic toolstack; as a result, we also provide some patches to accelerate libxl.


We carried out a few modifications to Click in order to have it run in the MiniOS environment. For example, we added a new target to Click for MiniOS, and a tiny glue, around 300 lines of code, that connects Click to MiniOS and exposes the element handlers and middlebox initialization. We further included a couple of new Click elements to act as Xen virtual network adapters.

Middlebox Key Click Elements
Load Balancer RatedSplitter,HashSwitch
Firewall IPFilter
DPI Classifier, IPClassifier
Traffic Shaper BandwidthShaper, DelayShaper
Tunnel IPEncap, IPsecESPEncap
Multicast IPMulticastEtherEncap, IGMP
BRAS PPPControlProtocol, GREEncap
Monitoring IPRateMonitor, TCPCollector
DDoS prevention IPFilter
IDS Classifier, IPClassifier
IPS IPClassifier, IPFilter
Congestion Control RED, SetECN
IPv6/IPv4 ProtocolTranslator46

Key Click elements available for developing a wide range of middleboxes.

Click comes with over 300+ stock elements which make it possible to construct middleboxes with minimal effort (see table above). It is also extensible, so we are not limited to the functionality provided by the stock elements. We can easily extend this framework and introduce new elements for new classes of packet processing. Click is of course no panacea: it does not cover all types of middlebox processing, for instance middleboxes that need a full-fledged TCP stack. In such cases, it is better to use a standard Linux VM. It is worth mentioning that while MiniOS represents a difficult development environment, programming for ClickOS is relatively painless: development, building and testing can take place in user-space Click, and the resulting code/elements simply added to the ClickOS build process when ready.