Erebos Tester
Developed for testing the Erebos protocol implementation
and C++ library, generally intended mainly
for testing networking code – it provides an easy way to execute the tested
program simultaneously on multiple nodes within a virtual network. These nodes
can be configured in a single or multiple subnets, and their properties, like
whether they are connected, can be changed during test run according to a given
script.
The test framework uses two components from the tested project:
-
Test tool – executable that accepts commands on standard input, executes the
tested functionality based on those, and provides output on standard output.
The output can be asynchronous, i.e. some events may trigger a message to be
generated while waiting for output of a command; interpretation is up to the
provided test script.
The test tool can be set either:
- by the
--tool command-line parameter of erebos-tester, or
- in the
erebos-tester.yaml configuration file, or
- by the
EREBOS_TEST_TOOL environment variable.
-
Test script – defines how to run the instances of test tool and in what kind
of network topology. Contains commands to send to the test tool instances
and rules to interpret the responses. The script is written in a custom
language described below.
Usage
The erebos-tester tool, when executed without any arguments,
looks for a erebos-tester.yaml file in the current or any parent directory (see below for details).
Run erebos-tester --help for details about command-line parameters.
The tester can be installed from sources or directly via cabal:
cabal install erebos-tester
When available in the PATH, it can be run to test the Haskell Erebos implementation:
git clone git://erebosprotocol.net/erebos
cd erebos
cabal build
erebos-tester --tool="$(cabal list-bin erebos) test" --verbose
or the C++ one:
git clone git://erebosprotocol.net/cpp
cd cpp
cmake -B build
cmake --build build
erebos-tester --verbose
To run tests from a given test file, pass it as command-line argument:
erebos-tester path/to/script.test
To select single test from a file, use : separator:
erebos-tester path/to/script.test:TestName
Configuration
To allow running erebos-tester without the need to supply project-specific configuration on command line,
per-project configuration can be done using erebos-tester.yaml file placed in the root of the project
(or other directory from which the tests will be executed).
This is a YAML file with following fields:
tool: path to the test tool, which may be overridden by the --tool command-line option.
tests: glob pattern that expands to all the test script files that should be used.
Script language
The test script language uses indentation to define the command blocks, e.g. to
define a test body or denote the scope of variables. Each command is on its own
line, terminated by newline. Commands accept arguments preceded by name/keyword
in arbitrary order, to make the behavior clear without the need to know
the expected order of the parameters.
For examples, see tests within the
Erebos implementation repository.
Each test script file consists of one or more test cases, started with test
keyword, with its body within indented block:
test [<name>]:
<test block>
Test name is optional, but if present can be used to run the single test from
a file that contains multiple tests.
Types
The script language is strictly typed without any implicit conversions,
although types can not be (as of now) declared explicitly and are always inferred.
Each expression has specific concrete type, polymorphic types are not supported (yet).
integer
Integer numbers. Entered as decimal literals and used in arithmetic expressions:
let x = 2
let y = 3
let z = x * 2 + y
number
Arbitrary-precision numbers. Entered as literals with decimal point or percentage and used in arithmetic expressions:
let x = 2.1
let y = 34%
let z = x * 2.0 + y
string
String literals are enclosed in double quotes ("),
using backslash to escape special characters (", \ and $)
and to represent some others (\n for newline).
let s = "some text"
Dollar sign ($) can be used to expand variables (numbers are expanded to decimal representation).
let a = "abc"
let b = 4
let c = "$a $b" # = "abc 4"
Arbitrary expression can be used within additional curly braces:
let a = 2
let b = 3
let s = "abc ${2*a + b}" # = "abc 7"
regex
Regular expression literals are enclosed in slash characters (/):
let re = /a.*/ # match any string starting with 'a'
Dollar-expansion can be used here as well.
Strings expand to regular expressions matching the exact string,
regular expressions expand are used directly.
let str = "."
let re1 = /./
let re2 = "$str$re1" # match '.' followed by any character
boolean
Result of comparison operators == and /=.
network
Represents network/subnet, created by subnet command and used by subnet, node, spawn and network configuration commands.
node
Represents network node, created by node command and used by spawn or network configuration commands.
Members:
ip: string representation of node's IP address.
process
Represents running process. Created by spawn, used by send and expect commands.
Members:
node: node on which the process is running
list
Lists are written using bracket notation:
let numbers = [1, 2, 4]
List elements can be of any type, but all elements of a particular list must have the same type.
Used in the for command.
Build-in commands
subnet <name> [of <network>]
Create a subnet within a <network> (or context network if omitted) and assign the new network to the variable <name>.
node <name> [on <network>]
Create a node on network <network> (or context network if omitted) and assign the new node to the variable <name>.
spawn as <name> [on (<node> | <network>)]
Spawn a new test process on <node> or <network> (or one from context) and assign the new process to variable <name>.
When spawning on network, create a new node for this process.
The process is terminated when the variable <name> goes out of scope (at the end of the block in which it was created) by closing its stdin.
When the process fails to terminate successfully within a timeout, the test fails.
send <string> to <process>
Send line with <string> to the standard input of <process>.
expect <regex> from <process> [capture <var1> [, <var2> ... ]]
Check whether <process> produces line matching <regex> on standard output, and if this does not happen within current timeout, the test fails.
Output lines produced before starting this command and not matched by some previous expect are accepted as well.
Output lines not matching <regex> are ignored by this expect call, i.e. do not cause the expect call to fail.
Regular expressions are anchored on both sides, so must match the entire line.
If e.g. only the beginning should be matched, the passed regular expression needs to end with .*.
The regular expression can contain capture groups – parts enclosed in parentheses ((, )).
In that case the expect command has to have the capture clause with matching number of variable names.
Results of the captures are then assigned to the newly created variables as strings.
flush [from <proc>]
Flush memory of <proc> output, so no following expect command will match anything produced up to this point.
guard <expr>
Check whether boolean expression <expr> is true; if not, the test fails.
disconnect_node [<node>]
Disconnect <node> from network – state of the veth network link from the node is set to down.
The effect lasts until the end of the block.
disconnect_nodes [<network>]
Disconnect all nodes of <network>. The network bridge interface state is set to down.
The effect lasts until the end of the block.
disconnect_upstream [<network>]
Disconnect network upstream – state of the veth network link connecting network bridge to the upstream network is set to down.
The effect lasts until the end of the block.
packet_loss <rate> [on <node>]
Set the packet loss rate on the node's veth link to <rate> as decimal number or percentage, e.g. 0.2 or 20% for 20% packet loss rate.
The effect lasts until the end of the block.
for <var> in <expr>:
<test block>
Execute <test block> for each element of list <expr>, with current element assigned to <var>.
local:
<test block>
Execute <test block> in a new local scope. Used to restrict scope of variables or duration of effects.
with <expr>:
<test block>
Execute <test block> with <expr> as context.
wait
Wait for user input before continuing. Useful mostly for debugging or test development.
Optional dependencies
The test framework can use some other tools to help with debugging or development.
GDB
If GDB is installed, it's possible to use --gdb command-line switch of the erebos-tester tool to use the debugger.
The GDB session is started in background and tester uses the GDB machine interface to communicate with it.
Whenever a new process is spawned, it is attached to the debugger as a new inferior.
In case any process is terminated by a signal, e.g. crashes with segfault, interactive GDB session is opened.
tcpdump
If tcpdump binary is found in the PATH,
it is used to generate network log in the pcap capture format within the test directory,
separately for each virtual network (specifically its bridge interface).
