Getting Started

A quick guide into the deployment and usage

VALE Extensions

Get acquainted with our fast, modular software switch. Start playing with software ports, and try building new modules to enable more complicate scenarios.

VALE, is provided as a part of the netmap distribution. For Linux, it is a third-party kernel module. For FreeBSD, it has been part of the operating system since FreeBSD 10.

Compiling (FreeBSD)

You just need to compile the applications. Assuming the source code is at /usr/src/, then:

 $ cd /usr/src/tools/tools/netmap
 $ make

For convenience, we recommend to set the $PATH environmental variable to /usr/src/tools/tools/netmap. For bash or zsh, type

 $ export PATH=$PATH:"/usr/src/tools/tools/netmap"

Compiling (Linux)

You need the Linux kernel sources that exactly correspond to the installed kernel. The kernel version should be 2.6.3 - 3.13 (though perhaps newer ones work too). We assume the kernel sources to be at /usr/src/linux-X.X.X.

Download the netmap sources here. To build (without NIC support, since we are using only virtual ports for now):

$ git clone https://code.google.com/p/netmap/ netmap-release
$ cd /usr/
$ cd netmap-release/LINUX
$ make clean; make KSRC=/usr/src/linux-X.X.X NODRIVERS=yes
$ insmod netmap_lin.ko;
$ cd ../examples;
$ make clean; make

For convenience, please add /usr/src/netmap-release/examples to $PATH:

 $ export PATH=$PATH:"/usr/src/netmap-release/examples"

Connecting two virtual ports

Having installed the kernel module, we are ready to play with a switch by attaching the pkt-gen packet generater/receiver to a switch port.

First, setup a receiver port named rxp on a switch named vale0::

 $ pkt-gen -f rx -i vale0:rxp

The -f rx option makes pkt-gen wait for packets on vale0:rxp In another terminal, setup a sender port named txp on the (existing) switch "vale0:":

 $ pkt-gen -f tx -i vale0:txp

The -f tx option specifies that packets should be sent out on vale0:txp. With this in place, you should be able to observe packets on the receiver's terminal.

Note that the virtual interface and the switch instance are dynamically created. The virtual interface is detached from the switch and deleted when the process dies. The switch instance is destroyed when the final interface is detached.

As you may have noticed, the netmap/VALE frameworks uses a combination of , where the switchname starts with "vale" and can be followed by arbitrary characters, and portname can be an arbitrary string, unless we are referring to an actual NIC.

Attach and detach NICs to/from the switch

In Linux, you will first need to compile the netmap/VALE kernel module with patched device drivers. First, make sure that the target NIC drivers are compiled as a module in your system. For IGB and IXGBE drivers, enable CONFIG_IXGBE_DCA. Here is an example in .config:

  CONFIG_E1000=m
  CONFIG_E1000E=m
  CONFIG_IGB=m
  CONFIG_IXGBE=m
  CONFIG_IXGBE_HWMON=y
  CONFIG_IXGBE_DCA=y
  CONFIG_IXGBEVF=m

To compile the netmap/VALE kernel module with these patched drivers, follow the compiling process outlined in the "Compiling (Linux)" section above while applying following changes:

  • Third line: (make clean; make KSRC=…), remove NODRIVERS=yes;
  • After the fourth line: insmod …, add insmod e1000/e1000.ko (if you use other drivers, please replace e1000 with their name)

  • For both FreeBSD and Linux, turn off flow control and TCP segmentation offload (TSO):

    In Linux:

    $ ethtool -A ethX autoneg off tx off rx off
    $ ethtool -A ethX autoneg off tx off rx off
    $ ethtool -K ethX tso off gro off gso off tx off rx off
    

    In FreeBSD (for ix0):

     $ sysctl dev.ix.0.fc=0
     $ ifconfig ix0 -tso
    

    You are now ready to attach/detach NICs to/from a VALE switch using the vale-ctl command (which is somewhat similar to brctl for the Linux bridge).

  • vale-ctl -a attaches an interface to the switch. The example below attaches an interface ethX to a switch named "vale0:". Note that if "vale0:" doesn't exist, it is dynamically created and subsequently destroyed when the last interface is detached:

    $ vale-ctl -a vale0:ethX
    
  • vale-ctl without any option prints the current configuration. An example below shows a switch instance named "vale0:" with a port whose index is 0 and name eth0:

    $ vale-ctl
    bdg_ctl [149] bridge:0 port:0 vale0:eth0
    
  • vale-ctl -d detaches an interface from the switch. The example below detaches ethX from vale0:

    $ vale-ctl -d vale0:ethX
    

    When you attach NICs to the switch, they will be disconnected from the TCP/IP stack. If you wish to have the host stack still attached you can use the -h option instead of -a:

    $ vale-ctl -h vale0:ethX
    

You can write your own packet switching logic as a kernel module for either Linux or FreeBSD.

In this section we demonstrate how to write a module that implements a simple packet switching logic. Writing a kernel module may sound daunting but don't worry, under the VALE framework it just requires small, mechanical work. Below is a simple module that you can extend to create your own modules. It routes packets to destinations ports using the least significant byte of the destination IP address:

Downloading a sample module

Assuming the netmap files are at /usr/src/netmap-release and that you place your modules at /usr/src/mymodule.

 $ cd /usr/src/
 $ wget http://example.com/mymodule.tar.gz
 $ tar xzf mymodule.tar.gz

Compiling (FreeBSD)

Assuming your kernel sources are at /usr/src

 $ cd mymodule/sys/contrib/mymodule
 $ make KSRC=/usr/src

Compiling (Linux)

Assuming your kernel sources are at /usr/src/linux-kernel

 $ cd mymodule/LINUX
 $ make KSRC=/usr/src/linux-kernel NSRC=/usr/src/netmap-release

Checking a bit of code

Before running, let's have a look at the code that implements the packet switching logic. Open mymodule/sys/contrib/mymodule/mymodule.c with your favorite viewer, and have a look at the "my_lookup()" function:

u_int
my_lookup(struct nm_bdg_fwd *ft, uint8_t *hint,
    const struct netmap_vp_adapter *vpna)
{
  uint16_t ether_type;
  struct ip *iph;
  char *buf = ft->ft_buf;

  ether_type = ntohs(*((uint16_t *)(buf + ETHER_ADDR_LEN * 2)));
  if (ether_type != ETHERTYPE_IP)
    return NM_BDG_NOPORT;
  iph = (struct ip *)(buf + ETHER_HDR_LEN);
  return ntohl(iph->ip_dst.s_addr) & 0xff;
}

This function is called for every packet and decides which port they should go to. As described before, it returns the least significant byte of the destination IP address. If this is a non-IPv4 packet, it returns NM_BDG_NOPORT, indicating that this packet should be dropped. While it is not shown in the listing above, we can return NM_BDG_BROADCAST to indicate broadcast.

Attaching a module to a switch

The module above will be part of the VALE switch instance (switching fabric) named "vale0:" at load time. To instantiate the switch, first make a network interface:

 $ pkt-gen -i vale0:vi0 -f rx

Please keep this process active until the end of the experiment:

Now, let's load the module and attach it to this switch in another terminal:

Linux (you should be at /usr/src/mymodule/LINUX):

 $ insmod ./mymodule_lin.ko
 (to unload, do "rmmod mymodule_lin")

FreeBSD (you should be at /usr/src/mymodule/sys/contrib/mymodule):

 $ kldload ./mymodule.ko
 # (to unload, do "kldunload mymodule")

Set up a receiver port:

 $ pkt-gen -i vale0:rxport -f rx

Now we should have two ports in the switch. To confirm this, in another terminal, type:

 $ vale-ctl

You will see the new port rxport attached to port 1 (you should also see the port "vi0" located at port 0) With this in place, we expect to receive packets at rxport depending on the destination IP address. To confirm this, let's send packets from another port (use another terminal):

 $ pkt-gen -i vale0:txport -f tx -d 10.10.0.1

You should see the packets at vale0:rxport, because the least significant byte of the packet's destination IP address is 1. Next, kill the sender with Ctrl+C (in the second terminal), and send packets to another destination IP address, for instance 10.10.0.2:

 $ pkt-gen -i vale0:txport -f tx -d 10.10.0.2

This time you will NOT see the packets at vale0:rxport, because the destination port is identified as port 2 which does not exist (thus packets are dropped).


In the xennet repository you can already find some sample switch extensions/modules. These are merely prototypes that have been created to exploit the modularity of the switch and to control traffic going through our middleboxes. An additional tool is bundled to let you control which extension is registered with the switch.

 $ cd /path-to-xennet/
 $ cd bdgfn
 $ make NETMAP_SRC=/path/to/netmap

In the following example, 257 means no port in case of unmatched packets, and 0 is the bridge index. ips refers to the switch extension name. This command will register the static source IP extension/module with the vale0 bridge:

 $ ./bdgfnctl ips regfn --bridge=0 --port=257

This tells the extension that all packets with source 10.10.0.4 go to port 1:

 $ ./bdgfnctl ips addif --bridge=0 --ip4 10.10.0.4 --dest 1

And the same applies to the static MAC extension:

 $ ./bdgfnctl macs regfn --bridge=0 --port=257
 $ ./bdgfnctl macs addif --bridge=0 --mac 00:00:00:00:00:01 --dest 1

You can also list current IP/MAC -> switch port assignments with the following commands:

 $ ./bdgfnctl macs listif --bridge=0 --mac 1
 $ ./bdgfnctl ips listif --bridge=0 --ip4 1

We assume that netmap is already installed.

OVS-VALE code can be cloned from GitHub:

 $ git clone https://github.com/cnplab/ovs-vale.git

For installation, see INSTALL.NETMAP

Xen Optimizations

A fast network backend to accelerate Xen packet I/O for our ClickOS virtual machines and netmap-based applications on Linux VMs.
  • Linux kernel version >= 3.6 installed
  • Kernel sources that exactly correspond to your installed kernel
  • Kernel headers/symbols for the kernel version you are against
  • Compilation of the target NIC (only ixgbe supprted for now), backend and frontend as modules
  • Example .config:
    CONFIG_IXGBE=m
    CONFIG_IXGBE_HWMON=y
    CONFIG_IXGBE_DCA=y
    CONFIG_IXGBEVF=m
    CONFIG_XEN_NETDEV_BACKEND=m
    CONFIG_XEN_NETDEV_FRONTEND=m
    

Grab the sources:

 $ git clone https://github.com/cnplab/xennet
 $ cd xennet 
 $ git submodule update --init

If your kernel isn't built already, you will need at least to do a prepare:

 $ cp /boot/config-`uname -r` /usr/src/linux-source-X.X.X/.config
 $ cd /usr/src/linux-source-X.X.X
 $ make prepare
 $ cd -

By default it compiles everything (netmap + netback/netfront)

 $ cd LINUX
 $ make KSRC=/usr/src/linux-source-X.X.X prepare
 $ make KSRC=/usr/src/linux-headers-X.X.X

Then just load the modules

 $ insmod ../netmap/LINUX/netmap_lin.ko;
 $ insmod xen-netback/xen-netback.ko;

Copy the Xen hotplug script:

 $ cp ../scripts/vif-vale /etc/xen/scripts/

Finally, the hotplug scripts require a netmap tool called vale-ctl. We need to build it and either copy to your /usr/local/bin or export it to your PATH

 $ cd ../netmap/examples
 $ make    
 $ export PATH=$PATH:`pwd`/../netmap/examples

The network backend provided with kernels <= 3.11 cannot be removed. Before inserting the patched netback driver you need to blacklist the old module at startup time, and then reboot the machine; only then can you insert our modified netback.

 $ vim /etc/modprobe.d/blacklist.conf
 blacklist xen-netfront
 blacklist xen-netback

You can improve latency if you set the scheduler parameters to the minimum, specially in the case of booting VMs on demand. We decrease the timeslice to 1 ms and rate limit to 100 usecs:

 xl sched-credit -s -t 1 -r 100

The linux frontend is experimental and only supports applications that use the netmap API. To use this modified netfront all you need is to change the hotplug script. The logic in the hotplug is the same as the ones provided by Xen, but instead it uses vale-ctl to manage the bridge.

 $ cp path-to-xennet/scripts/vif-vale /etc/xen/scripts/

While it will work with Linux guests, our main focus so far has been MiniOS guests, which is what we used to conduct most of our benchmarks. However, if you would like to try the experimental Linux frontend, you simply need to load the previously-compiled kernel modules in the Linux guest and fire up the pkt-gen netmap application:

First boot a domain without network:

$ xl create path-to-xen-config
$ xl console linux

On your domU, add the following entry to /etc/modprobe.d/blacklist.conf:

blacklist xen-netfront

Remove the xen-netfront:

rmmod xen-netfront;

Then you just load your patched xen-netfront:

$ insmod path-to-xennet/netmap/LINUX/netmap_lin.ko
$ insmod path-to-xennet/LINUX/xen-netfront/xen-netfront.ko
$ ifconfig eth0 up
$ cd ../netmap/examples
$ ./pkt-gen -i eth0 -f rx -b 1024

Afterwards on your Domain 0 try run the packet generator:

$ cd path-to-xennet/netmap/examples
$ ./pkt-gen -w 2 -i eth0 -f tx -l 60 -b 1024

ClickOS

Bootstrap your ClickOS environment, and play with your network traffic. We hope you explore Click packet processing framework, adapting to your needs.

Requirements

You need to have the following packages installed (in Debian):

    swig

Export the following environment variables (replace with the Xen version you are running):

$ export XEN_ROOT=/path-to-xen-sources

Note: To build the javascript binding you need to use this version of swig

Grab the latest image here or build it from sources as shown below.

To build ClickOS from source, we first need to build a cross-compilation toolchain. Since the toolchain is rather big and takes a while to buildm we provide prebuilt binaries so you can start building ClickOS right away. For this you will need the Xen sources, though you won't need to build these since only the header files are necessary. The steps are as follows:

$ git clone https://github.com/cnplab/mini-os.git
$ export MINIOS_ROOT=`pwd`/mini-os
$ git clone https://github.com/cnplab/toolchain
$ cd toolchain
$ make

$ cd -
$ export NEWLIB_ROOT=`pwd`/toolchain/x86_64-root/x86_64-xen-elf
$ export LWIP_ROOT=`pwd`/toolchain/x86_64-root/x86_64-xen-elf

# or the upstream https://github.com/kohler/click
$ git clone https://github.com/cnplab/clickos.git

$ cd clickos
$ ./configure --enable-minios --with-xen=$XEN_ROOT
$ make minios

Thats it! You should be able to find the ClickOS image at:

$ ls -lah /path-to-clickos/clickos/minios/build/clickos_x86_64{,.gz}
-rwxr-xr-x 1 root root 6.8M Mar 18 01:06 clickos_x86_64*
-rw-r--r-- 1 root root 1.6M Mar 18 01:07 clickos_x86_64.gz

We tend to prefer the uncompressed image since uncompression takes about 40 msecs at boot time, a significant portion of time when the rest of the boot process takes about 30 msecs. Once we have the image, all that remains is to build the ClickOS toolstack, called cosmos. If you're installing with the Xen sources you do not need any additional libraries; if not, you'll need to install the libxen-dev (on Debian-based systems).

For the toolstack, just clone the toolstack and run make.


To run ClickOS you need two things: a Xen configuration and a Click configuration.

Here's a sample Xen configuration file for ClickOS :

    name   = 'clickos'
    kernel = 'path-to-clickos/minios/build/clickos_x86_64'
    vcpus  = '1'
    # pinning your VCPU helps performance
    #cpus   = '3'
    memory = '8'

    # uncoment this line if you wish to use our backend
    #vif    = ['bridge=vale0,script=vif-vale']
    vif    = ['mac=00:15:17:15:5d:74,bridge=xenbr0']

    on_poweroff = 'destroy'
    on_reboot   = 'restart'
    on_crash    = 'preserve'                        
    click       = 'mirror.click'

    # uncomment this line to enable PVH mode for ClickOS
    #pvh=1

The Click configuration could be something really simple that just grabs packets, swaps src and dst MAC addresses, and sends them back to the bridge:

    FromDevice -> EtherMirror -> ToDevice

Save the click config as mirror.click and the Xen config as example.cfg. You can either use xl to boot the ClickOS guest, and later on cosmos to start the click config within that guest:

    $ xl create example.cfg
    $ DOMID=`xl list | grep clickos | awk -F' ' '{ print $2 }'`
    $ cosmos start $DOMID mirror.click

You can choose a DOMLIB cosmos will be able to manage domains on its own. You have tree options available:

  • DOMLIB=xcl enables our toolstack to manage MiniOS-based domains. It is made for MiniOS, and should not be used for other types. Attaching block devices are not supported at the moment.
  • DOMLIB=xl enables the standart libxl toolstack to manage domains.
  • DOMLIB=none is the default option and disables domain management in cosmos toolstack

If you intend to orchestrate the booting process in a more high-level language, or rather simplicity of the deployment, you definitely should enable these options.

To build cosmos with our domain library (libxcl):

    cd $XEN_ROOT
    ./configure
    cd tools
    make -C include
    cd /path/to/cosmos
    make DOMLIB=xcl

If you built cosmos with a Dom library you will be able to boot ClickOS VMs with the following command:

    $ cosmos create example.cfg

After booting the VM, the output from the console will look like the following:

    $ xl console clickos
        Xen Minimal OS!
    start_info: 0x232000(VA)
        nr_pages: 0x800
    shared_inf: 0x5f76c000(MA)
         pt_base: 0x235000(VA)
    nr_pt_frames: 0x5
        mfn_list: 0x22e000(VA)
    mod_start: 0x0(VA)
         mod_len: 0
           flags: 0x0
    cmd_line: -d standard
      stack:      0x1ebce0-0x20bce0
    MM: Init
          _text: 0x0(VA)
    _etext: 0x173a7b(VA)
       _erodata: 0x1d1000(VA)
         _edata: 0x1d2ec0(VA)
    stack start: 0x1ebce0(VA)
           _end: 0x22c518(VA)
    start_pfn: 23d
        max_pfn: 800
    Mapping memory range 0x400000 - 0x800000
    setting 0x0-0x1d1000 readonly
    skipped 0x1000
    MM: Initialise page allocator for 23f000(23f000)-800000(800000)
    MM: done
    Demand map pfns at 801000-2000801000.
    Heap resides at 2000802000-4000802000.
    Initialising timer interface
    Initialising console ... done.
    gnttab_table mapped at 0x801000.
    Initialising scheduler
    Thread "Idle": pointer: 0x2000802050, stack: 0x250000
    Thread "xenstore": pointer: 0x2000802800, stack: 0x260000
    xenbus initialised on irq 1 mfn 0x5ef6b
    Thread "shutdown": pointer: 0x2000802fb0, stack: 0x270000
    Dummy main: start_info=0x20bce0
    Thread "main": pointer: 0x2000803760, stack: 0x280000
    sparsing 0MB at 1ea000
    "main" "-d" "standard"
    [on:82] * 18 for clickos/0/elements*
    [on:82] * 17 for clickos/0/control*
    [on_status:247] status change to Running
    Thread "click": pointer: 0x2000804c60, stack: 0x290000
    ************************ NETFRONT for device/vif/0 **********


    net TX ring size 256
    net RX ring size 256
    backend at /local/domain/0/backend/vif/5/0
    mac is 00:15:17:15:5d:74
    **************************
    [router_thread:205] Starting driver...

Using Click is pretty much like building legos. The basic module is called an element, which is a basic unit that performs an action on packets such as modifying MAC addresses, verying IP checksums, decreasing TTLs or rate limiting traffic, to name a few. Here are a couple of more examples of Click configurations to get you started.

Firewall

A stateless firewall:

    // WAN Interface
    wani :: FromDevice(0);
    wano :: ToDevice(0);

    // LAN Interface
    lani :: FromDevice(1);
    lano :: ToDevice(1);

    // Classifier element
    cw :: Classifier(
        12/0806, // ARP Packets
        12/0800, // IP Packets
        -        // Other
    );

    // IP Filter element
    f :: IPFilter(
        0 dst host 10.0.0.1,
        1 all
    );


    wani -> cw;

    cw[0] -> CheckARPHeader(14) -> lano;                  // ARP Packets
    cw[1] -> CheckIPHeader(14) -> f;                      // IP Packets
    cw[2] -> Print("Drop (Blocked protocol)") -> Discard; // Other

    f[0] -> lano;                                         // allow
    f[1] -> Print("Drop (Blocked by filter)") -> Discard; // drop

    lani -> wano;

Ponger

A Click configuration to process ARP/ICMP requests:

    define($IP 10.10.0.3);
    define($MAC 00:15:17:15:5d:75);

    source :: FromDevice;
    sink   :: ToDevice;
    // classifies packets 
    c :: Classifier(
        12/0806 20/0001, // ARP Requests goes to output 0
        12/0806 20/0002, // ARP Replies to output 1
        12/0800, // ICMP Requests to output 2
        -); // without a match to output 3

    arpq :: ARPQuerier($IP, $MAC);
    arpr :: ARPResponder($IP $MAC);

    source -> c;
    c[0] -> ARPPrint -> arpr -> sink;
    c[1] -> [1]arpq;
    Idle -> [0]arpq;
    arpq -> ARPPrint -> sink;
    c[2] -> CheckIPHeader(14) -> ICMPPingResponder() -> EtherMirror() -> sink;
    c[3] -> Discard;

For information about how to run ClickOS with these click configurations check the previous section. Once it is up and running, configure the dom0 bridge as follows and test your ponger configuration:

    $ ifconfig xenbr0 10.10.0.1 netmask 255.2