Appliance Release v0.41

This page contains the release notes for the v0.41 Anapaya appliance software release. The appliance software release is applicable for the following Anapaya products:

We recommend always upgrading to the latest available patch release. Please refer to Upgrade notes for steps to be taken when upgrading. For general information on how to upgrade your appliance, please refer to software updates.

Upgrade Notes

Known issues

Refer to the Known issues for known issues in this release.

This release introduces breaking changes. Please read the Breaking Changes section carefully before upgrading.
This release requires the installation of the anapaya-system package v2.20.0 (or later). The installer will automatically reject the installation if the required anapaya-system package is not available.
The IP-in-SCION tunneling (SGRP) semantics have changed significantly in this release. Please read IP-in-SCION tunneling: redesigned routing domains and the Migration to v0.41 guide before upgrading. The appliance performs an automatic configuration migration on upgrade; in the event that migration fails the installer will roll back to the previous version.
This software release also contains updated Grafana dashboards and alerting rules, please follow Recommended alerts and Recommended Grafana dashboards for more information on how to update your monitoring setup.

Features

SCION Secure Access Groups

SCION Secure Access Groups (SAGs) introduce a new paradigm of private networking over the public SCION Internet. SAGs allow an organization to create and manage invite-only groups of trusted Autonomous Systems (ASes). SCION path segments published within a group are only visible to and accessible by other members of that same group, effectively creating an invisible, private network overlay on top of the SCION Internet.

This is achieved through the Anapaya Hidden Segment Directory (HSD), which acts as a private path segment registry for the group. Members register their path segments exclusively with the HSD; non-members cannot discover these paths and therefore cannot reach the group's infrastructure. Malicious actors on the public and SCION Internet are structurally blocked from scanning or attacking infrastructure that participates only in a Secure Access Group.

This release adds the ability to configure the appliance's interaction with the HSD. Three new configuration sections have been introduced:

config.scion.segment_registries — configure the HSD endpoint(s) to register path segments with.
config.scion.ases[].segment_sources — configure where the appliance should look up path segments for a given AS (public SCION or private HSD).
config.scion.ases[].interfaces[].segment_registrations — configure which segment registries each SCION interface publishes its path segments to.

For more information, including how to create and administer Secure Access Groups, please refer to the SCION Secure Access Groups product page.

note

Using Secure Access Groups requires an Anapaya EDGE Pro license and an active subscription to the Secure Access Groups service in the Anapaya CONSOLE. Group membership and policies are managed exclusively through the Anapaya CONSOLE.

IP-in-SCION tunneling: redesigned routing domains

The semantics of routing domains in IP-in-SCION tunneling have been redesigned to be more consistent and BGP-like. These changes address a number of long-standing edge cases and give operators more predictable control over how IP prefixes are accepted and announced across SCION ASes.

In the previous design the IP address space was partitioned into mutually exclusive domain slices and incoming prefixes were split at domain boundaries. Starting with v0.41, domains are no longer required to cover mutually exclusive IP ranges. Instead, each incoming prefix from a remote AS is matched in full against an ordered list of SGRP rules (derived from the routing domain configuration) and is either accepted as a whole or rejected as a whole. Prefix splitting is no longer performed. Packet forwarding continues to use standard longest-prefix-match routing.

The new model introduces the following behaviors:

Non-fallback domains generate reject rules for their IP ranges, preventing domains with higher sequence_id (i.e., lower priority) values from hijacking traffic. Prohibit routes are also installed to prevent unexpected failover.
Fallback domains (new concept) omit the security reject rules and prohibit routes, explicitly allowing traffic to fall over from a lower-sequence_id (i.e., higher priority) domain when no matching prefix is available.
The appliance-cli inspect tunneling sgrp command displays the generated SGRP rules for inspection and debugging.

Prefix filters in the domain configuration support ge (greater-than-or-equal) and le (less-than-or-equal) qualifiers, similar to BGP prefix-list syntax. For example, the prefix 172.20.5.0/24 ge 24 le 32 matches any subnet of 172.20.5.0/24 with a prefix length between 24 and 32 bits. Omitting ge or le matches only the exact prefix length given. This makes it possible to concisely express "accept any more-specific route within this range" without enumerating each subnet individually.

The appliance performs an automatic migration of the existing routing domain configuration on upgrade. It is strongly recommended to verify the migration outcome before and after the upgrade using the procedure described in the Migration to v0.41 guide.

For the full configuration reference and detailed examples, see the updated IP-in-SCION tunneling documentation.

Experimental OSPF support

important

OSPF support is an experimental feature. Use in production environments is not recommended without prior consultation with Anapaya Support.

The appliance now supports experimental OSPF-based AS-internal dynamic routing. This allows operators to use OSPF for intra-AS IP routing without the need to deploy an external router.

To enable OSPF, set the enable-frr-ospf feature flag in the experiments section of the appliance configuration and provide an advanced FRR configuration template that includes the OSPF configuration.

Other features

Added support for static onlink routes in the appliance configuration. The onlink property can be set on a route to indicate that the gateway is directly reachable on the link even if it does not match any interface subnet. This is useful in cloud environments that use onlink gateways for the default route.
The appliance can now expose a health endpoint on a separate address from the management API. This is useful for cloud load balancers that require an unauthenticated health endpoint:
```
management:
  health:
    address: "<IP_ADDRESS>:<PORT>"
```
A new POST /system/reboot management API endpoint allows rebooting the appliance host programmatically. A corresponding health check reports a notice when a reboot is required (e.g., after a kernel command line parameter change).

The appliance management configuration now accepts optional description and annotations fields for attaching arbitrary metadata (e.g., location, owner, asset tag) to the device:

management:
  hostname: edge-geneva-corpbank
  description: "Geneva office edge gateway, rack 3"
  annotations:
    - key: location
      value: "Geneva, Rack 3"
    - key: asset-tag
      value: HW-2024-0042

VPP now supports the native Virtio driver, which can be used for virtual machine deployments. Configure interfaces with driver: VPP_VIRTIO to select this driver.
Added support for forwarding intra-AS SCION packets with an empty path.
SCION interface and sibling uptime/downtime is now shown by appliance-cli info and appliance-cli info scion. The new metric router_interface_up_since_timestamp_seconds exposes the last state-change timestamp.

Breaking Changes

Interface address validation

IPv4 interface addresses are now validated to reject network addresses (e.g., 10.0.0.0/24) and broadcast addresses (e.g., 10.0.0.255/24). This validation applies to all IPv4 addresses with a prefix length shorter than /31. Existing configurations using such addresses will fail validation after the upgrade.

Mole service removed

The mole service has been removed. Any metrics, dashboards, or APIs associated with it are no longer available.

v0.41.0

Improvements

Improved CPU core allocation for VPP workers

The default algorithm for allocating CPU cores to VPP workers has been revised. The new algorithm produces more balanced assignments across the range of supported hardware platforms: it uses fewer VPP worker cores on resource-constrained devices, freeing up cores for the operating system, while fully utilizing available cores on higher-end platforms. Example assignments with the new algorithm:

Platform	Cores	VPP workers	VPP main thread	OS threads
Lanner	4	1	1	2
Supermicro L (HT)	8+8 HT	3+3	1+1	4+4
Supermicro XL (HT)	16+16 HT	8+8	1+1	4+4

Previously, on the Lanner with 4 cores, 2 VPP workers were created, leaving only 1 OS thread. The new allocation avoids such over-provisioning of VPP at the expense of OS responsiveness.

DNAT now applies only to inbound traffic

DNAT translation is now restricted to inbound sessions, i.e., packets that originate from outside the network. Packets originating from the internal destination host are no longer translated, ensuring that the source address of locally-initiated traffic is preserved.

For the following DNAT rule:

{
    "nat": {
        "dnats": [
            {
                "address": "192.0.0.1",
                "destination_address": "10.0.0.1"
            }
        ]
    }
}

Traffic direction	Previous behavior	New behavior
Inbound to 192.0.0.1	Translated to 10.0.0.1	Translated to 10.0.0.1
Outbound from 10.0.0.1	Source translated to 192.0.0.1	Not translated (keeps 10.0.0.1)
Reply packets	Translated correctly	Translated correctly

This change also makes it possible to configure multiple DNAT rules with different address values mapping to the same destination_address.

Configuration migration on upgrade

Configuration migration is now performed as part of the appliance installation process. If the migration fails for any reason, the installer automatically rolls back to the previously installed version, ensuring that the appliance remains fully operational.

Previously, migration was handled by the appliance controller after installation. A failure in that model could leave the appliance with an empty configuration — with the particular risk that a subsequent reboot would bring up a deconfigured appliance.

Other improvements

The timeout for service configuration validation has been standardized to 10 seconds across all appliance services, improving startup reliability on low-resource platforms and in the training environment.
OpenTelemetry Collector updated to version 0.144.0. The upstream contrib image is used again, providing a wider selection of receivers, processors, and exporters. A pprof endpoint is now enabled on the telemetry container.
The appliance API health check endpoint (GET /api/v1/health) now accepts a fail_level query parameter, which causes the endpoint to return HTTP 503 when health status is at or below the specified severity. This is useful for cloud load balancer health checks.
The appliance-cli info tunneling command now reports the remote gateways of each tunneling domain, including their health status and whether encryption is supported.
The appliance-cli info shuttle command now reports common network interface statistics (bytes/packets received and transmitted, errors, drops, and multicast packets) for each shuttle interface.
New IP-in-SCION tunneling debug endpoints provide detailed visibility into SGRP prefix state for local and remote ASes:
```
GET  debug/scion-tunneling/sgrp/local/announce
GET  debug/scion-tunneling/sgrp/local/receive
GET  debug/scion-tunneling/sgrp/remote/announce
GET  debug/scion-tunneling/sgrp/remote/receive
```
A debug/scion-tunneling/security endpoint has also been added to show the number of encrypted and non-encrypted sessions per routing domain.
The SCION CA-frontend now reports unhealthy when it is not authenticated to the upstream CA.
Shuttle server configurations now support a disabled flag per client. Setting it to true closes any existing connection from that client and prevents reconnection. The flag is available both in the appliance configuration and via the shuttle server management API.
VPP upgraded to v25.10, including VRRP improvements (VMAC as Ethernet source in GARP/ND, unicast GARP/ND mode) and a DPDK passive device option useful in Azure VMs with accelerated networking.

Fixes

The appliance now rejects configurations that contain conflicting interface addresses — overlapping or identical addresses across and within interfaces. Previously only exact duplicates were rejected.
Direct neighbor path monitoring is now restricted to PARENT and CORE neighbors. CHILD neighbors are excluded because they may not be visible in deployments using Secure Access Groups.
Fixed a bug where configuring a VLAN on a virtual function with the IGBVF driver could prevent the MAC address from being configured correctly.
Fixed a panic in the appliance controller firewall validator on invalid configuration input.
Fixed non-deterministic active license selection when multiple licenses are valid for the same product and tier. The license with the latest expiry date is now always preferred.
The debug/vpp/vrrp endpoint no longer reports an error when VRRP peers are configured in unicast mode.
The Caddy listener addresses are now correctly set. Previously, listener addresses could not be set up correctly under certain configurations, causing connectivity issues.
The appliance now correctly detects configurations pushed via the migration API endpoint (PUT /api/v1/migrations/:version/config) and no longer applies the standard configuration migration to them.
The appliance now validates that a neighbor AS is not the same as the local AS, in both the scion and cluster configuration sections.
The appliance now rejects attempts to add secrets with empty plaintext values.
Default values from system.vpp.connection are now correctly applied to all services using VPP.
The external interface up health check now correctly reports the remote interface identifier in its detail string. Previously it always reported remote interface 0.
Only UP administrative state is accepted for SCION interfaces. Configurations with any other value are automatically migrated to UP state on upgrade.
When applying network configurations, the appliance now removes any unexpected files from the netplan configuration directory, preventing obsolete files (e.g., from cloud-init or AWS VM import) from causing conflicts.
DNAT combined with SNAT using an address pool that overlaps with an interface address now works correctly.
Two DNAT rules sharing the same destination_address are now correctly rejected by configuration validation.
Changing the NAT pool configuration no longer causes the gateway or dataplane-control process to crash.
IP prefixes received as multipath routes (from multiple BGP peers in the same AS) are no longer incorrectly retracted from announced routes.
The 5001-2008 (DomainHasReachableEncryptedRemote) health check has been removed. The 5001-2004 (DomainHasReachableRemoteGateways) check now reports full details about each remote gateway, including non-encrypted remotes when the domain has encryption enabled.
The gateway no longer crashes when a state update coincides with an HTTP API request inspecting the current state.
BFD enabled status is now correctly reported in the health check API when a sibling interface is added for an existing peer with BFD enabled. Previously, BFD was always reported as disabled until a service restart.
The router now correctly sends an InternalConnectivityDown SCMP error (instead of ExternalInterfaceDown) when a sibling is unreachable.
The shuttle server and client now correctly use the ISD-AS specified in the endpoint or local address field when multiple ISDs are configured. Previously, the default AS was used regardless.
The shuttle client now creates the TUN interface only after a connection to the server has been successfully established.
The control service now correctly handles external CA URLs that do not have a trailing slash.
The control service now throttles log messages when beacon registration repeatedly fails (e.g., while waiting for an AS certificate).
IP packets and bytes sent through IP-in-SCION tunneling are now counted correctly.
Initiating a configuration reload no longer clears IPv6 link-local addresses from VPP interfaces.
Bond interface MAC addresses are now preserved across reconfiguration when no explicit MAC address is set in the appliance configuration.

Upgrade Notes​

Features​

SCION Secure Access Groups​

IP-in-SCION tunneling: redesigned routing domains​

Experimental OSPF support​

Other features​

Breaking Changes​

Interface address validation​

Mole service removed​

v0.41.0​

Improvements​

Improved CPU core allocation for VPP workers​

DNAT now applies only to inbound traffic​

Configuration migration on upgrade​

Other improvements​

Fixes​