Performance tests

The objective of the performance tests is to measure end-to-end forwarding performance.

Topology

The physical testbed topology consists of three nodes: a Traffic Generator (TG) and two Devices Under Test (DUT1 and DUT2). DUT1 and DUT2 are Anapaya appliances interconnected via a SCION link.

note

The Traffic Generator and DUT2 configurations remain fixed across all test scenarios:

• Traffic Generator: A host running TRex within a container environment.
• DUT2: A fixed hardware platform, specifically the Supermicro SYS-110D-16C-FRAN8TP.

Traffic flow

The Traffic Generator injects bidirectional traffic. The packet path is defined as follows:

• Direction 1: TG → DUT1 → [SCION Link] → DUT2 → TG
• Direction 2: TG → DUT2 → [SCION Link] → DUT1 → TG

Throughput rates and forwarding performance are measured by the Traffic Generator at the ingress/egress points. The offered load is distributed symmetrically (50/50) between the two directions.

Test scenarios

The following test cases are executed to benchmark forwarding plane performance:

• SCION: Native forwarding of SCION packets.
• IP-in-SCION tunneling: Forwarding of IP packets encapsulated within SCION headers.
• IP-in-SCION tunneling with encryption: Forwarding of IP packets encapsulated within SCION headers, with payload encryption enabled.

Traffic Profiles

Traffic is generated using the following profiles:

• SCION4 IMIX: IPv4 packets encapsulated in SCION, following a standard IMIX distribution (28:16:4 ratio):
  • ~58% packets of 136 bytes
  • ~33% packets of 570 bytes
  • ~8% packets of 1518 bytes
• IPv4 IMIX: Native IPv4 packets following a standard IMIX distribution (28:16:4 ratio):
  • ~58% packets of 64 bytes
  • ~33% packets of 570 bytes
  • ~8% packets of 1518 bytes

Reported rates

The results tables show a reported rate for each test. The reported rate is a packet rate that the platform forwards with a packet loss of 0.1% or less. We derive this rate from repeated Multiple Loss Ratio (MLR) search runs.

Multiple Loss Ratio search

RFC 9971 describes this method in detail.

We use this method to find the highest packet rate at which the packet loss stays at or below a loss ratio of 0.1%.

Our CI system runs the MLR tests automatically on our rack. We store the result of each run. Our CI system runs the tests on this schedule:

• All tests every weekend.
• A small set of tests every night.

The nightly runs help us to identify regressions more precisely.

Calculation of the reported rate

To get a stable rate, we use the results of many MLR runs:

1. We collect the MLR results of each test over a period. Before we publish the first results for a platform, we collect at least 10 consistent runs for each test.
2. We calculate the 25th percentile (first quartile) of the rates of these runs.