Netfilter Simulation Environment

Jeremy Kerr

$LastChangedDate: 2005-11-03 04:49:05 +1100 (Thu, 03 Nov 2005) $, released under GPL.

Abstract

This document provides an overview of the Netfilter Simulator system, used for testing kernel netfilter code.


Table of Contents

1. Simulator Framework
2. Building the simulator
2.1. Configuring and importing the netfilter code
2.2. Compiling
3. Running the simulator
3.1. Command-line arguments
3.1.1. --failtest
3.1.2. --failpath path
3.1.3. --failtest-no-report function
3.1.4. --echo, -x
3.1.5. --quiet, -q
3.1.6. --exit, -e
3.1.7. --no-modules
3.1.8. --version, -V
3.1.9. --help
3.1.10. --linear-packets
3.1.11. --ignore-proc-issues
3.2. Commands
3.2.1. help
3.2.2. exit, quit
3.2.3. log
3.2.4. expect
3.2.5. iptables
3.2.6. route
3.2.7. time
3.2.8. hook
3.2.9. gen_ip
3.2.10. proc
3.2.11. gen_err
3.2.12. config
3.2.13. echo
3.2.14. strace
3.2.15. ifconfig
3.2.16. queue
3.2.17. version
3.2.18. lsmod
3.2.19. insmod
3.2.20. rmmod
3.2.21. tcpsession
3.2.22. info

1. Simulator Framework

The netfilter simulator provides an environment to run kernel netfilter code, by providing a kernel-like 'environment'. The simulator consists of:

Kernel Environment Code

Provide generic kernel structures and functions, such as spinlocks. This code is in the kernelenv/ directory.

Simulator core

Provides the protocol independent netfilter structures and functions, which are used by the protocol module and the netfilter code under test. The core functionality is provided by core/*.c.

Message-passing interface for userspace tools

Allows userspace programs to talk to the simulated kernel using the {s,g}etsockopt() function. This allows tools such as iptables to be used without modification.

A protocol module

Provides all the protocol-specific functions for the current protocol - at present there is only an IPv4 protocol module. This provides the routing code, address formats, etc, and is where the netfilter hooks are called from. The protocol module is in the core/<protocol>/ directory.

The imported netfilter code

The netfilter code under test. This is imported from an external kernel tree into the netfilter/ directory. Also, the .config and Makefile files are copied to allow building.

User interface tools

These tools add functionality to the user interface. Each tool provides one or more commands, such as ifconfig or route. Tools can be added by adding new files into the tools/ directory.

2. Building the simulator

2.1. Configuring and importing the netfilter code

First, the netfilter code needs to be configured - the source directory is specified to ./configure, using the [--kerneldir] flag. By default, it is set to the current kernel's build directory: /lib/modules/$(uname -r)/build

If you are reusing your source tree, it is recommended you do a make distclean before running ./configure.

[Note] Note

At present, the imported .config file is not used - instead, the packaged file .config.sample is copied to .config during the install stage. If you need to add any directives to the .config file, do so after the import

2.2. Compiling

Next, the simulator can be compiled with make.

Because the simulator is in development, the kernel environment may not yet provide all functionality required to the imported netfilter code. If there are any errors during compilation, email me.

[Note] Note

Each build of the simulator is dependent on the netfilter code under test; the binary is not 'installed' as would be the norm with other applications.

3. Running the simulator

Run the simulator with ./simulator.

The simulator will start, initialising the following items (in order):

  1. simulator core

  2. external message interface

  3. user interface

  4. logging system

  5. user interface tools

  6. kernel environment

  7. protocol module

  8. netfilter testing code

During initialisation (or at any other time), calls to printk() will be shown, as would occur on boot or kernel module loading.

By default, the simulator is started with the three interfaces (eth0, eth1 and lo) interface configuration:

lo      addr:  127.0.0.1
mask:  255.0.0.0
bcast: 127.255.255.255
RX packets: 0 bytes: 0
TX packets: 0 bytes: 0

eth0    addr:  192.168.0.1
mask:  255.255.255.0
bcast: 192.168.0.255
RX packets: 0 bytes: 0
TX packets: 0 bytes: 0

eth1    addr:  192.168.1.1
mask:  255.255.255.0
bcast: 192.168.1.255
RX packets: 0 bytes: 0
TX packets: 0 bytes: 0

In short - two ethernet interfaces on different networks, and a loopback interface.

Routing tables are initialised also, with the appropriate network available from each device:

        network    iface
127.0.0.0/8  lo
192.168.0.0/24 eth0
192.168.1.0/24 eth1

3.1. Command-line arguments

3.1.1. --failtest

Try every combination of external failures

The --failtest option runs the script, but deliberately inserts failures (currently allocation failures and user-copying failures). At each potential failure, both failure and success will be simulated: where we fail, the script will no doubt fail later which is OK, but we ensure that the entire system doesn't fall over or leak memory. If an error is found, the path of successes/failures will be printed out, which can be replayed using --failpath

3.1.2. --failpath path

Replay a failure path

Given a failure path, (from --failtest), this will replay the sequence of sucesses/failures, allowing debugging. The input should be the same as the original which caused the failure.

This testing can be slow, but allows for testing of failure paths which would otherwise be very difficult to test automatically.

3.1.3. --failtest-no-report function

Exclude a function from excessive failure reports

Sometimes code deliberately ignores the failures of a certain function. This suppresses complaints about that for any functions containing this name.

3.1.4. --echo, -x

Echo commands as they are executed

nfsim will echo each command before it is executed. Useful when commands are read from a file

3.1.5. --quiet, -q

Run quietly

Causes nfsim to reduce its output to the minimum possible - no prompt is displayed, and most warning messages are suppressed

3.1.6. --exit, -e

Exit on error

If --exit is specified, nfsim will exit (with a non-zero error code) on the first script error it encounters (eg an expect command does not match). This is useful when nfsim is invoked as a non-interactive script

3.1.7. --no-modules

Don't load all available modules on startup

Usually all netfilter modules are loaded on startup, this option will suppress this behaviour. To load modules after invoking nfsim with this argument, use the insmod command

3.1.8. --version, -V

Print the version of the simulator and kernel

3.1.9. --help

Print usage information

Causes nfsim to print its command line arguments and then exit

3.1.10. --linear-packets

Only use linear packets for testing

Prior to 2.6.10, there were many bugs involving nonlinear packets. Suppressing them allows testing for other bugs.

3.1.11. --ignore-proc-issues

Ignore issues with /proc files leaking

The --ignore-proc-issues can be used to suppress failure due to removing non-existent proc entries and leaving proc files behind, which happened in 2.6.10 and previous.

3.2. Commands

Inline help is available on most commands, and is reproduced here.

3.2.1. help

Displays general help, or help for a specified command

help [command]

With no arguments, help will show general system help, and list the available commands. If an argument is specified, then help will show help for that command, if available.

3.2.2. exit, quit

Exit the simulator

exit

quit

The exit and quit commands are synonomous. They both exit the simulator.

3.2.3. log

Manage logging settings

log [ = | + | - ] {type, ...}

Each log message is classified into one of the following types:

KERNEL

Kernel messages (including printk() calls)

UI

Messages from the user interface (deprecated?)

ROUTE

Routing information

PROTOCOL

Information from the protocol (IPv4) layer

PACKET

Information about packet movements (including netfilter hook results)

The log command allows you to select which messages are displayed. By default, all messages will be shown.

If the types argument is prefixed with a +, - or = character, those types will be added, removed or set as the current types of messages to be logged (repectively). If none of these characters is specified, = is assumed (the types are set to only those specified)

Messages generated as a result of user input are always logged.

3.2.4. expect

Catch logging information for automated testing

expect

expect [!] {command} {pattern}

expect will set up a set of patterns to expect in logging messages for a particular command. If that command finishes without matching the specified pattern, the simulator will exit with a non-zero return value. After the command is run, all expectations on that command are deleted.

expect with no arguments will print out the current list of expectations, as a command and a pattern.

expect command pattern will expect the specified pattern to occur the next time command is invoked. If an '!' appears before the command, then the expectation is negated - if the pattern appears in the output, then the simulator will exit with an error

The pattern itself is similar to a simple shell wildcard, except whitespace is loosely matched. The * character will match any a string of any length.

3.2.5. iptables

Run the iptables command on the simulator

iptables [options]

The external iptables binary will be invoked, with its operations redirected to the simulator. If the NFSIM_IPTABLES_PREFIX environment variable is set, the command in that directory will be executed (useful for testing specific variants). Otherwise, the current PATH is searched, then /usr/local/sbin, /sbin, /usr/sbin.

You do not need to be root to use iptables in this way. If the iptables command fails, that will be reported as

iptables: command failed

; with the [-e] to nfsim, nfsim will abort.

3.2.6. route

Manipulate the IPv4 routing table.

route [print]

route add { (1) network } {interface}

route del { (1) network }

(1) {{ ip_addr / netmask_bits } | { ip_addr mask netmask }}

route with no arguments, or a single argument "print" will display the current routing table.

route del network will delete the route to the sepecified network.

route add network interface will add a route to the sepecified network, via the specified interface.

3.2.7. time

Query or advance the kernel time

time

time {time} {+advance}

time {+infinity}

time with no arguments will show the current kernel time (in seconds).

time time will set the kernel time to time - which must be greater than the current time value. time +advance will increment the time by the value of advance, which must be positive.

When the kernel time is incremented, kernel timers may be triggered. If so, the simulator will show logging messages:

> time +30
running timer to ip_conntrack_core.c:init_conntrack()
>

The time +infinity command will forward time past all the currently pending timers, leaving a clean slate.

3.2.8. hook

Insert simple netfilter hooks

hook

hook {hookname} {priority} {verdict}

The hook command allows new hooks to be inserted. These hooks will simply return the specified verdict for all packets.

All hooks are cleared after the next gen_ip command.

The hookname argument specifies which list to insert the hook into. One of IP_PRE_ROUTING, IP_INPUT, IP_FORWARD, IP_OUTPUT, IP_POST_ROUTING

The priority argument specifies the priority of the hook, from INT_MIN to INT_MAX.

The verdict argument specifies the verdict to return from the hook itself: ACCEPT, DROP, QUEUE, STOLEN, REPEAT

3.2.9. gen_ip

Generate IP packets and send through the protocol layers

gen_ip [IF=interface] [FRAG=] [DF] [MF] [CE] [MAC=ethaddr] [TOS=diffserv field] [TTL=ttl] {src} {dest} {datalen} {protocol} {protocol-opts}

gen_ip [options] {src} {dest} {datalen} TCP {srcpt} {dstpt} {flags} [SEQ=seqnum] [ACK=seqnum] [WIN=wsize] [OPT=options] [DATA args]

gen_ip [options] {src} {dest} {datalen} UDP {srcpt} {dstpt}

gen_ip [options] {src} {dest} {datalen} ICMP {type} {code} [id] [sequence]

gen_ip will construct an IP packet and send it on the specified interface, or if no interface is specified, the packet will be sent from as if from the local host.

If [protocol] is TCP, then the ports and TCP flags must be specified. The format of the [flags] is 'SYN/ACK' etc. The format of the [OPT=] option is a comma-separated list of numbers, and there must be a multiple of four of them, such as:

1,2,3,4

. If DATA is specified, then the data of the packet will be populated with the rest of the arguments, with escapes for \n, \r, \t and \0.

If [protocol] is UDP then the ports must be specified.

If [protocol] is ICMP then the type and code must be specified: the id and sequence fields are optional.

3.2.10. proc

Displays and set /proc filesystem information

proc

proc {cat} {file}

proc {write} {file} {args...}

proc with no arguments will print the heirachy of the currently-registered proc files.

proc cat file will display the contents of the proc file specified.

proc write file args.. will write args to the proc file.

3.2.11. gen_err

Generate an ICMP error packet and send it through the protocol layers

gen_err [IF=interface] [LEN=length] {srcip} {icmp-type} {icmp-code} {gen-ip-options}

gen_err will use gen_ip to construct an IP packet, generate an ICMP error for that packet, as would be generated by a host at [srcip], then send it on the specified interface. If no interface is specified, the packet will be sent from as if from the local host. The [length] is the number of bytes to include from the original packet, default 28.

See gen_ip for how to specify the options for the initial packet.

3.2.12. config

Print out the kernel configuration used to build nfsim

config

This command is mainly used for scripts to determine what modules and/or features are available.

3.2.13. echo

Print a message

echo {text...}

echo prints a message.

3.2.14. strace

Report information about all system calls and user-kernel copies

strace

strace {off}

strace will report all system calls from iptables, and all copy_to_user and copy_from_user results. This is useful for debugging iptables, and checking that the kernel is returning the right error values.

Tracing will continue until the [off] argument is given.

3.2.15. ifconfig

Administer the simulated network interfaces. Interfaces can be brought up or down, or reconfigured while up.

At present, interfaces can be assigned only one interface (this limitiation is due solely to the ifconfig tool; the simulated kernel environment can handle multiple interface addresses)

ifconfig [interface]

ifconfig {interface} { (1) if_options }

ifconfig {interface} { (1) if_options } up

ifconfig {interface} down

(1) {{ ip_addr / netmask_bits } | { ip_addr mask netmask }} {broadcast_addr}

If ifconfig is invoked with no arguments, it will show the configuration of all devices. If a single argument is specified, it is assumed to be a device name, and the configuration for that device is displayed.

To bring a new interface up, specify the interface name and a network specification:

ifconfig eth2 10.0.0.1 8 10.255.255.255 up

Alternatively, the netmask can be specified in dotted-decimal notation. This command is equivalent to the previous:

ifconfig eth2 10.0.0.1 mask 255.0.0.0 10.255.255.255 up

Interfaces can be brought down with the command

ifconfig {interface} down

To reconfigure an interface, use the syntax

ifconfig {device} [ip_addr [netmask_bits [broadcast_addr] ] ]

3.2.16. queue

Manage the userspace packet queue

queue

queue {verdict} [id]

When a netfitler hook returns NF_QUEUE, it is stored within the simulator's own packet queue. When a packet enters this queue, a logging message is printed:

> gen_ip IF=eth0 192.168.0.2 192.168.0.1 50 udp 10 20
rcv:eth0
queue:added {IPv4 192.168.0.2 192.168.0.1 50 17 10 20}

The packet is assigned a unique id number (monotonically increasing over the life of the simulator session), which can be used to reinject packets in any order

The contents of the packet queue can be printed with the queue command with no arguments. Each packet in the queue is prefixed with its id:

> queue
0: {IPv4 192.168.0.2 192.168.0.1 50 17 10 20}
1: {IPv4 192.168.1.10 192.168.0.1 50 6 1025 80 SYN}

To reinject a packet from the queue, specify a verdict (one of the netfilter hook return values, for example NF_ACCEPT), and optionally, the id of the packet. If no id is specified, the packet at the front of the queue is reinjected.

3.2.17. version

Displays version information about the kernel simulator.

version

version kernel

The version displays information about the nfsim version and the version of the kernel it's simulating. With the [kernel] argument, it only displays the version of the kernel (useful for testsuites).

3.2.18. lsmod

List modules (both available and loaded)

lsmod

lsmod lists all the loaded modules, one to a line.

3.2.19. insmod

Insert a module

insmod [modulename]


insmod {-a}

insmod loads a module, or all modules. The caller must ensure that any other module this one expects are already loaded.

3.2.20. rmmod

Remove a module

rmmod

insmod {-a}

rmmod removes a loaded module. The caller must ensure that any module which is using this is already unloaded.

3.2.21. tcpsession

Generate TCP session

tcpsession {OPEN} {src} {dest} {srcpt} {dstpt}

tcpsession {OPEN} {src} {dest} {srcpt} {dstpt} {reply-src} {reply-dest} {reply-srcpt} {reply-dstpt}

tcpsession {DATA} {direction} {args}

tcpsession {LENCHANGE} {number}

tcpsession {CLOSE} {direction}

tcpsession {RESET} {direction}

tcpsession {ABANDON}

tcpsession is shorthand for creating a TCP session using multiple gen_ip commands. The four-argument [OPEN] form creates a simple connection (no NAT expected), the eight-argument form creates a connection which might be NATted. You can only have one open connection at a time: this is used for all operations. Sequence numbers will be 1001 for the sender, and 2001 for the recipient. Initial packet comes in eth0, replies come in eth1.

The [DATA] form sends data on the current connection, in a single packet (and sends an ACK in reply). [direction] is either 'original' or 'reply'.

The [LENCHANGE] command sets expected change in length of the next packet. It is used to tell tcpsession that the next packet is going to expected to be shortened or lengthened.

The [CLOSE] form sends the required FIN and ACK packets to close the connection: [direction] specifies who initiates the close.

The [ABANDON] argument causes the connection to simply be forgotten, so you can open a new one.

3.2.22. info

Displays information about the kernel simulator.

info {subject}

The subject argument to info must be one of the following:

hooks

displays the currently registered netfilter hooks

sockopts

displays the currently registered netfilter sockopts

timers

shows the current kernel timers - when each is set to expire, as well as the function where the timer was registered.

rcache

the contents of the kernel routing cache

failpoints

show how many failure points have been passed: with --failtest, each of these alternatives would be attempted.