Tiny, Agile Virtual Machines for Network Processing

An overview over the guest operating system.

Looking for ClickOS examples? Check out Getting started section for more details

ClickOS


A minimalistic, tailor-made, virtualized operating system to run Click-based middleboxes.

The recent trend towards network function virtualization (NFV) proposes to shift middlebox processing from hardware-based appliances to software running on inexpensive, commodity hardware (e.g., x86 servers with 10Gb NICs). Towards this goal we developed ClickOS, a high-performance, virtualized software middlebox platform. It consists of the Click modular router software running on top of MiniOS (a minimalistic OS available with the Xen sources), plus optimizations to network I/O in order to drive 10 Gb/s throughput for almost all packet sizes. These virtual machines are small (6MB), boot quickly (in about 30 milliseconds) and add little delay (45 microseconds). In the rest of this page we describe the toolchain required to build the OS image; the toolstack to boot VMs in milliseconds; and a high level discussion of Click and the modifications we did to it in order to run in the MiniOS environment.

Ping delays for ClickOS, a Linux Xen VM (Domain-0), and KVM using the e1000 or virtio drivers.

Toolchain


Building such the ClickOS image is not trivial. MiniOS is intended to be built with standard GCC and as such we can link any standard C library to it, provided we handle for missing functionality appropriately. However, Click is written in C++, and so it requires special precautions. We developed a new build tool which creates a Linux-independent C++ cross-compiler using newlibc. Our build tool redesigns the standard MiniOS toolchain so that it is possible to quickly and easily build arbitrary, MiniOS-based VMs by simply linking an application’s entry point so that it starts on VM boot; this is useful for supporting middleboxes that cannot be easily supported by Click. Regarding libraries, we have been conservative in the number of them we link, and have been driven by need rather than experimentation. In addition to the standard libraries provided with the out-of-the-box MiniOS build (lwip, zlib, libpci) we add support for libpcre, libpcap and libssl, libraries that certain Click elements depend on.

Toolstack


We developed our own Domain-0 tool called cosmos. Cosmos is similar to xl and is built directly on top of the Xen UI libraries. To simplify development and user interaction, cosmos implements a SWIG [39] wrapper, enabling users to automatically generate bindings for any of the SWIG supported languages. For convenience, we have also implemented a Python-based ClickOS CLI.

Boot times and middlebox instantiation times.


In order to reduce domain creation times even further, Cosmos can also be built on top of our own domain management library called xcl. By being less complex and by eliminating some rarely used Xenstore entries, this library makes the creation process much faster compared to libxl. However, this is done at the cost of having a less generic toolstack; as a result, we also provide some patches to accelerate libxl.

Click

We carried out a few modifications to Click in order to have it run in the MiniOS environment. For example, we added a new target to Click for MiniOS, and a tiny glue, around 300 lines of code, that connects Click to MiniOS and exposes the element handlers and middlebox initialization. We further included a couple of new Click elements to act as Xen virtual network adapters.

Middlebox Key Click Elements
Load Balancer RatedSplitter,HashSwitch
Firewall IPFilter
NAT IP UDP TCPRewriter
DPI Classifier, IPClassifier
Traffic Shaper BandwidthShaper, DelayShaper
Tunnel IPEncap, IPsecESPEncap
Multicast IPMulticastEtherEncap, IGMP
BRAS PPPControlProtocol, GREEncap
Monitoring IPRateMonitor, TCPCollector
DDoS prevention IPFilter
IDS Classifier, IPClassifier
IPS IPClassifier, IPFilter
Congestion Control RED, SetECN
IPv6/IPv4 ProtocolTranslator46

Key Click elements available for developing a wide range of middleboxes.

Click comes with over 300+ stock elements which make it possible to construct middleboxes with minimal effort (see table above). It is also extensible, so we are not limited to the functionality provided by the stock elements. We can easily extend this framework and introduce new elements for new classes of packet processing. Click is of course no panacea: it does not cover all types of middlebox processing, for instance middleboxes that need a full-fledged TCP stack. In such cases, it is better to use a standard Linux VM. It is worth mentioning that while MiniOS represents a difficult development environment, programming for ClickOS is relatively painless: development, building and testing can take place in user-space Click, and the resulting code/elements simply added to the ClickOS build process when ready.

ClickOS middlebox throughput performance using a single CPU core for the VM.