Live Chat Software by Kayako
ProApps routing in netmap mode with netmap-rt
Posted by Alex Lemos on 26/Apr 17:24

Netmap mode routing in ProApps with netmap-rt

ProApps Enterprise Appliance starting on ProApps Routing 2.3.23 (routing-2.3.23) includes netmap mode L3 router via netmap-rt service. This feature leverages from netmap framework to do layer 3 packet routing, which is ready for primary border routing usage, with both BGP and OSPF services.

Before using netmap-rt there are quite a few concepts you need to understand. First, we strongly suggest this feature to be used by ProApps users running ProApps Routing component on ServerU hardware, specially L-800 or superior models, to take most benefit of performance and stability. If you run ProApps routing in a different hardware (other vendors) we suggest you use Chelsio T5 or Intel igb(4) and ix(4) cards which offer the best support for netmap framework.

When running in netmap mode it's important to understand that packets are processed and handled directly by the application. It means kernel has limited function in this mode. It's true to say that kernel is skipped and ignored, almost completely, in netmap mode. Except that it's the kernel which provides and controls access to netmap pseudo device and control plane exported by netmap framework as well as memory allocation and handling of /dev/netmap file descriptors. But once the application has open a network interface in netmap mode, the application takes full control of that port and skips kernel for any kind of packet processing from that moment.

It means in a practical sense that most of the kernel statistics you are used to checking with netstat(1) like packets per second and bits per second rates as well as error and retransmission rates, won't be counted by kernel anymore. They will be frequently zeroed or indicating low values, referring only to host bridge communications or pseudo device communication-related. Only hardware directly provided stats will be shown by netstat(1). Therefore, monitoring tools like nload(8), rate(8) or tcpdump(1) won't work as you are used to. But it doesn't mean you can use them anymore. Keep reading.

We have some concepts you need to understand:

  • kernel-path: is the packet processing flow inside the kernel, meaning, the normal processing mode within FreeBSD where a packet goes on different processing subroutines like driver related processing, kernel routing, vlan code, IPSec, packet mbuff tagging, firewall (ipfw, pf), queueing and subsystems like netgraph, etc.
  • user-path: in netmap mode, kernel-path is fully ignored, therefore vlan, ipsec, and filtering features need to be supported and implemented directly in the netmap application, this is what we call user-path; it means features not supported on netmap-rt(1) or any other netmap application won't be supported at all.
  • host-stack bridge: netmap framework provides a communication interface between application (user-path) and kernel (kernel-path) on host local, it's called host-stack bridge, with this feature the application does not need to explicitly support services which will be processed by the local host itself, so bgpd, ospfd, bird and sshd daemons may be used as usual when the application leverages on host-stack bridge features in netmap mode.

The netmap-rt service

On ProApps, netmap mode routing is supported by netmap-rt(1), a multithreaded daemon which offers dataplane (where routing happens in fact), controlplane (service management, threads sync, CLI and ctl application handling, dataplane threads control and interaction, ...), specifically netmap-rt provides:

  • dataplane: are the routing threads, the ones which actually do packet forwarding among configured interface ports; it offers different RX/TX threads identified as netmap-rt{dataplane} or netmap-rt{netmap-rt} when inspected with top -H.
  • controlplane: manage dataplane threads and access from CLI client, as well as netmap-rtctl utility, are identified as netmap-rt{controlplane} or netmap-rt{netmap-rt} when inspected with top -H.

  • monitor: monitors kernel FIB activities on FreeBSD and manages other convergence threads, they are identified as netmap-rt{monitor} when inspected with top -H.

  • fib-trigger-qX: are the multiple threads which may exist to manage the many convergence queues between kernel FIB and netmap FIB, meaning while monitor threads do monitor and feed the pending FIB changes in different queues, the queue threads will empty while syncing and updating netmap FIB and making it available to all dataplane (routing engine) threads; they are identified as netmap-rt{fib-trigger-q0} and inspect with top -H whereas qX may range from q0 up to the number of available CPUs in the system.

When you run top -H you will see threads details, for example:

11958 root 23 0 1411M 87896K kqread 2 0:02 85.42% netmap-rt{netmap-rt}
11958 root 23 0 1411M 87896K kqread 4 0:02 32.19% netmap-rt{netmap-rt}
11958 root 21 0 66804K 10004K select 1 18:42 2.20% snmpd
11958 root 23 0 1411M 87896K nanslp 3 0:00 11.71% netmap-rt{fib-trigger-q0}
11958 root 20 0 1131M 146M nanslp 1 0:21 0.00% netmap-rt{monitor}

About netmap-rt

netmap-rt, an IP router implementing FreeBSD routing system on top of netmap, the very fast networking packet I/O framework. This software is based on netmap-fwd, the original PoC for an IPv4 router on top of Netmap, designed by Patrick Tracanelli and PoC (pre-alpha) developed by L. Souza. Due to a couple of reasons, we decided to rename it to a more unique and non-conflicting name, and it's an evolution of the original PoC. This software is also a DPDK router, it means it can operate in DPDK mode and benefit differently from Intel networking cards when compared to Netmap mode. This software is developed and maintained by ServerU Inc and FreeBSD BR and shipped with ProApps Enterprise Appliance. However, it's designed to run on any compatible FreeBSD system or with ProApps Base without ProApps GUI.

This software was designed to transparently replace the kernel routing path while still fully compatible with it, allowing for user and daemon handling of kernel FIB and interfaces out of netmap mode while it's always reflected to netmap mode. It means it's designed to have the kernel-path as a safe fall back operation mode for a router without depending on netmap-only settings or configurations. It can also combine multiple networking interfaces in netmap routing mode with other kernel-path interfaces doing different jobs.

With a smart kernel FIB integration and sync and the ability to bridge pkts to host stack, using your every-day routing daemon like BIRD, OpenBGP or quagga is transparent and won't need absolutely any specific setting.

The main goal for netmap-rt is to leverage on netmap (and dpdk) technologies to reach unprecedented performance and packet forwarding rates on commodity networking hardware, taking best results from great professional networking cards with or without specific hardware acceleration features. We focus on Intel and Chelsio hardware but better performance is expected on several other cards. We want to reach the closest the possible to 10G and 40G line rate forwarding numbers with a full BGP prefix table on cost effective routing devices such as ServerU Netmap series.

Why rename:

The first and main reason is non-technical, the previous name (netmap-fwd) was the original name for netmap bridge example application by Luigi Rizzo, and although they shared the same name they are completely unrelated applications and do different things. However, performance numbers and related statistics were published with the same name making a huge and unwanted confusion for scientific and academical references of related works.

Second, netmap-fwd was just a proof of concept. Now it's time for the whole thing and netmap-rt is not just a forwarder anymore. With ACL filtering, PBR capabilities, metric collector for network observability, other actual and planned features we decided to rename it.

Available features on netmap-rt:

The include version of netmap-rt on ProApps Routing offers the following features:

  • Netmap IPv4 routing on top of Netmap framework
  • Multithreaded
  • RTM_MESSAGE trigger monitor thread to sync kernel FIB with netmap-rt FIB
  • Multi-threaded queue for parallel routing sync with smart adaptive batched queueing (individual queued entries or volumetric are handled differently)
  • netmap-rtctl control utility for automation and interaction
  • FIB couple, decouple and recouple controls
  • ACL (Access Control List) for basic filtering and PBR
  • PBR (Policy Based Routing) on top of ACL action
  • RADIX or DXR (Direct / Range Routing Lookups) for routing lookup method
  • DXR lookup method for ACLs
  • Ready for BGP, OSPF kernel-path based routing daemons (host-stack bridge)
  • IEEE 802.1q support (VLAN)
  • local socket based console for interaction
  • FIB couple, decouple and recouple controls
  • route dump to file in a dedicated thread for noninterrupting full routing dump, inspection and debug
  • smart "route get XXX" command which compares and monitors coupled status
  • Fully transparent interaction with kernel FIB and command line allowing users & daemons to inspect and modify kernel routing table as always done
  • IEEE 802.1ad (Q-in-Q) support
  • Intel DPDK mode (via DPDK Netmap Compatibility Layer), compat_netmap
  • idpdk-fwd specific enhancements when in DPDK mode (yet to be finished)
  • Application control over wanted hardware acceleration and offload features

Availability of netmap-rt

You can install and use netmap-rt shipped with ProApps Enterprise Appliance. You can also use it from provided packages. To check your ProApps Routing component version, type:

% pkg info routing
Name : routing
Version : 2.3.23
Installed on : Mon Apr 6 18:37:31 2016 BRT
Origin : proapps/routing
Architecture : freebsd:10:x86:64
Prefix : /usr/local
Categories : proapps
Licenses :
Maintainer :
Comment : ProApps Routing meta port
Annotations :
repo_type : binary
repository : ProApps
Flat size : 5.52MiB
Description :
ProApps Routing meta port

And to check if netmap-rt is installed, check with:

% pkg info netmap-rt
Name : netmap-rt
Version : v0.3c
Installed on : Thu Apr 14 18:18:59 2016 BRT
Origin : net/netmap-rt
Architecture : freebsd:10:x86:64
Prefix : /usr/local
Categories : net
Licenses : EULA
Maintainer :
Comment : IP router implementing FreeBSD routing on top of Netmap
Shared Libs required:
Annotations :
Flat size : 108KiB
Description :
IP router implementing FreeBSD routing on top of Netmap

Using netmap-rt

To install in a system without ProApps Routing (ProApps Base), use pkg(1) utility:

pkg install libucl libevent2
pkg add <url_to_this_package>

Add to /etc/rc.conf:

#netmap_rt_flags="-v" #optional additional flags



and adjust it according to your needs. In particular, you may want to keep LRO acceleration features (not disable it) on some particular NICs. You may also try DXR and Radix lookup methods and pick the one that performs better for you.

The most important configuration on netmap-rt.conf is the list of networking interfaces used by netmap-rt. Interfaces not listed will be left for kernel path routing only while listed interfaces will be handled in netmap mode:

interfaces="ix0 ix1 igb1 igb2 igb3 vlan10 vlan12"

Start the service as usual:

# service netmap-rt start

Log messages are handled by syslog and written to /var/log/messages.

You should always manipulate routes directly from CLI as you are used to, using route(8) command and your daemons of choice. You may manipulate routes on netmap-rt but routes added to netmap-rt won't be feed to kernel fib, however, routes on kernel FIB will always be in sync on netmap-rt, so we strongly recommend you to only inspect netmap-rt routes directly.

Using netmap-rtctl

Know and explore netmap-rtctl:

usage: netmap-rtctl -f [netmap-rt.conf] -s [socket] <command> <command arg>

Commands and example usage for netmap-rtctl:

netmap-rtctl arp

show cached arp list

netmap-rtctl ifconfig

show observed interface running configuration

netmap-rtctl route get <destionation>
netmap-rtctl route delete <destination>
netmap-rtctl route add <destination> <gateway>
netmap-rtctl route add <destionation> -iface <interface>

route manipulation commands, similar to route(8)

netmap-rtctl route

print full routing table, BEWARE: it's a BLOCKING thread implementation on Radix or DXR walktree, will cause routing engine to temporarily pause, DON'T USE IT in production, use it for debug or troubleshooting only, instead, use the netmap-rtctl route dump command to dump routing table to file

netmap-rtctl route dump

dump full routing table as converged into netmap-rt to a file which defaults to /var/db/netmap-rt-dump (per netmap-rt.conf)

netmap-rtctl fib couple|decouple|recouple

manage convergence from kernel FIB to netmap-rt

netmap-rtctl status

show counters and status information

netmap-rtctl cli

show interactive cli and netmap-rtctl counter

netmap-rtctl shutdown

cleanly shutdown and exit netmap-rt router

netmap-rtctl acl list

print running acl rules and stats

netmap-rtctl acl reload

reload acl rules file

netmap-rtctl acl enable|disable

enabled by default if acl_conf file exists and has valid rules disable acl will completely skip loaded rules and packets are passed directly to the router engine

netmap-rtctl help

basic help (command listing)

Management port

We strongly recommend you to have an extra management port for sshd and ProApps GUI access. This management GUI should not be in netmap mode (ie, not in netmap-rt_iflist) and should be directly reachable (do not depend on static route or the default route to reach it) especially because, upon start or shutdown of a netmap application, connectivity will be interrupted while switching kernel-path and netmap modes. Some BGP sessions may be reset upon.

Known Limitations

  • IPv6 support is experimental and not intended to use in BGP/OSPF production environments, use kernel-path instead with dedicated VLAN or LAN ports and keep that outside netmap-rt.
  • (*)It's not possible to use IPSEC in netmap mode with netmap-rt.
  • Using netmap-rt in DPDK mode requires explicit and manual configurations on the loader.conf file, please open a ticket to be instructed on how to do it if you want to test it.
  • ACL support is considered beta and is mostly tested for blackhole and PBR features.
  • A statistic column is missing on the routing tree in both RADIX and DXR modes, this stat info should behave like netstat(1) use column.
  • Telemetry hooks for networking visibility for NetFlow need to be completed to include AS numbers.

(*) this task is not on the TODO list, everything else is planned work.

Report feedback:


For software version v0.3c

Early adoption on ProApps and Current State

Currently, netmap-rt(1) activation on ProApps can not be done from ProApps GUI (graphical interface), since the current state is considered BETA. We have been running netmap-rt since December/15 on several customers and on our own border routing (BGP). ProApps early availability is aimed for testing and early adoption for users who are comfortable enough with the command line (CLI). ProApps Routing will continue working as usual since netmap-rt integration is transparent and only configuration (netmap-rt.conf) and runtime troubleshooting demand CLI access.

We strongly recommend you open a ticket on before configuring netmap-rt on your system so you can be followed by our staff which will provide a more confident comfortable experience.


netmap-rt offers up to 10x performance improvements compared to kernel routing. In a testing environment with ServerU Netmap L-800 with 4x10GbE and 6x1GbE bilateral tests running pkt-gen in both RX and TX modes, we observed the following performance numbers:


  # CPU ix(4) pps ix(4) bps ncxl(4) pps ncxl(4) bps vlan(4) pps vlan(4) bps
ProApps netmap-rt 1 1.7Mpps 20Gbps 1.7Mpps 20Gbps 1.5Mpps 20Gbps
FreeBSD kernel (fastforwarding) 1 200Kpps 2Gbps 200Kpps 2Gbps 180Kpps 1.8Gbps
FreeBSD kernel (melifaro routing) 1 400Kpps 4Gbps 440Kpps 4Gbps 393Kpps 4Gbps
Brocade vRouter 5600 (DPDK) 1 1.8Mpps 20Gbps - - 1.6Mpps 20Gbps
ProApps netmap-rt 8 9Mpps 23Gbps 11Mpps 28Gbps 8Mpps 23Gbps
FreeBSD kernel (fastforwarding) 8 900Kpps 18Gbps 1.1Mpps 20Gbps 740MKpps 18Gbps
FreeBSD kernel (melifaro routing) 8 3.6Mpps 20Gbps 3.6Mpps 20Gbps 3.1Mpps 20Gbps
Brocade vRouter 5600 (DPDK) 8 6.8Mpps 23Gbps - - 6.2Mpps 20Gbps

Configuration file reference for netmap-rt

The conf file used by netmap-rt is /usr/local/etc/netmap-rt.conf and it looks like:

# Copyright (c) 2016, ServerU Inc.
# All rights reserved.
# netmap-rt configuration file.
# pidfile - Set the pidfile path.
pidfile = "/var/run/";
# event_backend - netmap event backend
# Only kqueue and poll are supported.
# Only poll works with 11-CURRENT due to a bug in kqueue support.
event_backend = "kqueue";
# cli_max_clients - Maximum number of cli connections.
cli_max_clients = 10;
# cli_socket_path - Path to cli socket.
cli_socket_path = "/var/run/netmap-rt.sock";
# arp_max_tries - Number of ARP request retransmits.
# Retransmit not faster than one request per second.
arp_max_tries = 5;
# disable_hw_acceleration - Disable selected hardware acceleration features
# which should not be used in netmap mode. Supported vales are:
# txcsum rxcsum tso4 tso6 lro.
disable_hw_acceleration = "txcsum rxcsum tso4 tso6 lro";
# acl_conf - Path to access control list configuration file.
acl_conf = "/usr/local/etc/netmap-rt-acl.conf";
# route_dump_file - where `netmap-rtctl route dump` will dump routing table to.
route_dump_file = "/var/db/netmap-rt-dump";
# lookup_method - supported values: radix, dxr (default = radix), only used
# on routing lookups, ACL lookups is always dxr.
lookup_method = "radix";
interfaces = "ix0 ix1 vlan101 vlan190 vlan900 ncxl0 ncxl1";

While the ACL conf file to be used on /usr/local/etc/netmap-rt-acl.conf has the following syntax:


One entry per line, example:

;;;;udp;0-19;drop log

The accepted delimiter is ; while # is a comment, both per line or at the end of each line. The expression any or the absences (blank) of match criteria tells ACL subsystem not to test these criteria. Possible actions are deny|drop, allow|pass and nexthop <gw>, they may follow the log expression to generate logs (syslog) as per the above example.

Troubleshooting and Monitoring

Since netstat(1) and other monitoring utilities which leverage on kernel interfaces won't provide much relevant information, you would always use netmap-rtctl show stats to check for general health statistics of the whole netmap router subsystem. If you need to monitor pps rate, bps rate or to inspect traffic, you need to use pcap enabled applications.

Since 2013 ProApps provides it's own Netmap support, it means advanced features which are not available on the latest -STABLE or -CURRENT systems will be available on ProApps netmap support. Among those features, one feature early adopted on ProApps is Netmap Monitor mode. To use utilities like tcpdump(1), nload(8), rate(1), one may leverage on the special monitor interface and use:

tcpdump -nv -i netmap:ncxl0/rtz
rate -i netmap:ix0/rtz -R -b

The general syntax for netmap monitor mode is netmap:<if>/<monitor_mode> while supported modes are r for RX, t for TX and z for zerocopy. Note that performance may be impacted by inspecting network interfaces so you should really select r or t mode and always prefer z access mode when it works. Since zerocopy approach copies packet in a different moment within netmap framework it may not always work, but for general pcap interception it will usually work so always use it.

(6 vote(s))
Not helpful

Comments (0)
Post a new comment
Full Name:
Help Desk Software by Kayako