Skip to main content

Configure shuttle

experimental

Shuttle is an experimental feature of the Anapaya appliance. It was introduced in version v0.40.0.

Shuttle is a feature of the Anapaya appliance that allows operators to establish secure tunnels between appliances and route IP traffic over these tunnels. The main use case of Shuttle is the management of appliances. Shuttle follows the client-server model. Typically, the appliance to be managed should configure a shuttle client and connect to a server operated by the managed service provider.

A Shuttle tunnel is an HTTP3-over-QUIC-over-SCION tunnel between the Shuttle client and server. To create such a tunnel, Shuttle leverages the connectip standard protocol, a protocol to proxy IP packets in HTTP3-over-QUIC tunnels, while using SCION as its underlying network protocol (L3). Shuttle follows the client-server model. Both client and server are configured in the appliance configuration in dedicated sections. A new top-level section is introduced for configuring shuttle servers. Shuttle clients are configured as a new type of interface in the interfaces section. The figure below provides an overview of the shuttle client-server model:

address: 10.0.0.2
 address: 10.0.0.2
address: 10.0.0.1/24
 address: 10.0.0.1/24

allowed_destinations
[10.0.1.0/24]
 allowed_destinations...
endpoint:
[1-ff00:0:2,168.0.0.1]:49420
 endpoint:...
local:
[1-ff00:0:3,172.0.0.1]
 local:...
Shuttle    Server
 Shuttle    Server
Shuttle    Client
 Shuttle    Client
Shuttle tunnel
 Shuttle tunnel
Appliance
 Appliance
IP traffic
 IP traffic
IP network
 IP network Text is not SVG - cannot display

This guide provides the necessary steps to configure shuttle client and server.

Configure shuttle server​

Configure a shuttle server by following the steps below:

Add the shuttle server configuration by appending the configuration under the shuttle-servers.servers list. Choose a name, and fill in the endpoint, address, allowed_destinations, and advertised_routes.

  • name is the name of the tunnel interface created by the shuttle process.

  • endpoint is the SCION address and port on which the shuttle server is listening for connections from clients (in the form [<ISD-AS>,<IP>]:<port>, e.g., [1-ff00:0:2,168.0.0.1]:49420). The reachability of the address can be tested using the scion ping command.

  • address is an IP address with the prefix length which is the address of the tunnel interface created by the shuttle process.

  • allowed_destinations is the list of IP prefixes to which the traffic from the shuttle interface is allowed to be forwarded. The appliance installs firewall rules to allow forwarding from the shuttle interface specified with the name to this list of destinations. It can be empty only if the only intended destination is the server itself.

  • advertised_routes is the list of prefixes that the server advertises to the clients. The clients should forward traffic to those routes over the shuttle interface, i.e., the clients install Linux routes for these prefixes on their local host via the shuttle interface.

    appliance-cli edit config 'config.shuttle_servers.servers[]: {
      "name": "shuttle0",
      "endpoint": "[1-ff00:0:2,168.0.0.1]:49420",
      "address": "10.0.0.1/24",
      "allowed_destinations": ["10.0.0.0/24", "10.0.1.0/24"],
      "advertised_routes": ["10.0.0.0/24", "10.0.1.0/24"]
    }'

Configure appliance with shuttle client​

Configure shuttle client​

Configure a shuttle client by following the steps below:

  1. Add the shuttle client configuration under the interfaces.shuttles section. Fill the following parameters in the command below and run it:

    • name is the name of the tunnel interface created by shuttle process.
    • local is the SCION address to use (in the form <ISD-AS>,<IP>, e.g., 1-ff00:0:3,172.0.0.1). It must be able to reach the server endpoint. The reachability of the address can be tested using the scion ping command.
    • servers.endpoint is the SCION address and port of the shuttle server you want to connect to (in the form [<ISD-AS>,<IP>]:<port>, e.g. [1-ff00:0:2,168.0.0.1]:49420), which is configured on the server side as Configure shuttle server describes. The reachability of the address can be tested using the scion ping command.
    • disable is a boolean flag to disable the shuttle interface. By default it is false. If set to true, the running shuttle client will be stopped and disconnected from the server.
    appliance-cli edit config 'config.interfaces.shuttles[]: {
      "name": "shuttle0",
      "disable": false,
      "local": "1-ff00:0:3,172.0.0.1",
      "servers": [{"endpoint": "[1-ff00:0:2,168.0.0.1]:49420"}]
     }'

  2. Retrieve the client's certificate.

    • Run the following command. Here, shuttle0 should be replaced with the actual name of the shuttle interface.

      appliance-cli info shuttle shuttle0

    • Sample execution output:

      - shuttle0
          Connected: ❌
          ID (Subject Key ID): BB1emzrNbWpe1tbq3k2dGJgI3L4a2WB14jYBkZzXY1_2r6NKIHdFLrqZeWuhKB5IjU6YB79_tZU2y0cfMP8UuX8
          Certificate: MIIBdDCCARqgAwIBAgIRAKYmAruOnswqnIdMexx0P4QwCgYIKoZIzj0EAwIwADAgFw0yNTA3MDkwOTM1NTBaGA8yMTI1MDYxNTA5MzU1MFowADBZMBMGByqGSM49AgEGCCqGSM49AwEHA0IABB1emzrNbWpe1tbq3k2dGJgI3L4a2WB14jYBkZzXY1/2r6NKIHdFLrqZeWuhKB5IjU6YB79/tZU2y0cfMP8UuX+jczBxMA4GA1UdDwEB/wQEAwIHgDATBgNVHSUEDDAKBggrBgEFBQcDAjBKBgNVHQ4EQwRBBB1emzrNbWpe1tbq3k2dGJgI3L4a2WB14jYBkZzXY1/2r6NKIHdFLrqZeWuhKB5IjU6YB79/tZU2y0cfMP8UuX8wCgYIKoZIzj0EAwIDSAAwRQIgI3z3bkl5sUQpmsapo9SepI3y+NAhS4Y8QooOriAjJwcCIQDZyg8jYmMzH0jKo3Q+piWOROrCCkZGpSaiyMc76ZnsIg==
          Candidate Servers: [1-ff00:0:110,172.20.2.4:52810]

    • The ID (Subject Key ID) is used by servers to uniquely identify this client.

    danger

    As soon as a client with a certain name, e.g., shuttle0, is removed from the configuration, its associated ID and Certificate are permanently erased. They can never be recovered.

    Re-adding the client later with the same name, e.g., shuttle0, only results in creation of a new ID and Certificate, which must be communicated again to operators of all servers in the servers list.

    Editing a client does not remove the ID and Certificate.

    To disable a client without removing its ID and Certificate, set the disable flag to true.

  3. Copy the Certificate and provide it to the operators of all shuttle servers listed in Candidate Servers.

Configure the shuttle interface to accept traffic​

Configure management API​

Configure the appliance to expose the management API on the shuttle tunnel. Add the interface listener configuration under the management.interfaces section of the appliance configuration. The name is the name of shuttle interface configured above. The port is the tcp on which the management listener should listen.

appliance-cli edit config 'config.management.api.interfaces[]: {
  "name": "shuttle0",
  "port": 443,
  "description": "Accept management traffic on shuttle interface"
}'

Configure other use cases​

Configure the appliance to accept traffic on the shuttle interface for other use cases, e.g., for SSH access.

prepend mode

This configuration only works if the firewall mode is set to prepend mode.

Add a firewall rule under firewall.tables.chains.rules section of the appliance configuration.

  1. Find the index of the default_input chain, which we denote with $CHAIN_IDX. The list is 0-indexed, i.e., the index of the first entry is 0.

    appliance-cli get config -f 'body.config.firewall.tables[0].chains[].{name}'

  2. Add the following rule under the chain. The rule accepts traffic coming from a shuttle interface, i.e., iifname <name>, where the name is the name of a shuttle interface in the appliance configuration, with a specific protocol and destination port, i.e., <dport>.

    appliance-cli edit config 'config.firewall.tables[0].chains[$CHAIN_IDX].rules[]:{
      "rule": "iifname \"shuttle0\" <protocol> dport <dport> accept",
      "sequence_id": <sequence-id>
    }'

Add clients to shuttle server configuration​

  1. Define a set of IP addresses for the clients of the new server (only one IP per client). The set is not explicitly defined in the appliance configuration.

    Non-overlapping IP sets

    If configuring multiple servers on a single appliance, allocate non-overlapping IP sets across servers.

  2. Receive shuttle clients' credentials from shuttle clients operators. The credential of a client is its certificate generated on the client appliance.

  3. Assign a shuttle tunnel IP address to each client from the pre-defined IP set for the clients of this server.

  4. Add all clients' credentials, i.e., their certificates, and assigned IPs to the shuttle server section of the appliance configuration. Append each client to the clients list. You can add clients individually or all at once. Fill address and credential for each client. address is the assigned address to the client, and credential is the certificate received from the client operator.

    appliance-cli edit config  'config.shuttle_servers.servers[$SERVER_IDX].clients[]: {
      "address": "10.0.0.2",
      "credential": "MIIBdDCCARqgAwIBAgIRAKYmAruOnswqnIdMexx0P4QwCgYIKoZIzj0EAwIwADAgFw0yNTA3MDkwOTM1NTBaGA8yMTI1MDYxNTA5MzU1MFowADBZMBMGByqGSM49AgEGCCqGSM49AwEHA0IABB1emzrNbWpe1tbq3k2dGJgI3L4a2WB14jYBkZzXY1/2r6NKIHdFLrqZeWuhKB5IjU6YB79/tZU2y0cfMP8UuX+jczBxMA4GA1UdDwEB/wQEAwIHgDATBgNVHSUEDDAKBggrBgEFBQcDAjBKBgNVHQ4EQwRBBB1emzrNbWpe1tbq3k2dGJgI3L4a2WB14jYBkZzXY1/2r6NKIHdFLrqZeWuhKB5IjU6YB79/tZU2y0cfMP8UuX8wCgYIKoZIzj0EAwIDSAAwRQIgI3z3bkl5sUQpmsapo9SepI3y+NAhS4Y8QooOriAjJwcCIQDZyg8jYmMzH0jKo3Q+piWOROrCCkZGpSaiyMc76ZnsIg=="
    }'
    Finding SERVER_IDX

    If you have multiple shuttle servers you need to replace the [0] in the command above with the correct index.

    To list your shuttle servers use the following command:

    appliance-cli get config -f body.config.shuttle_servers.servers

    If you want to only list the name of the shuttle servers, you can use the following command:

    appliance-cli get config -f 'body.config.shuttle_servers.servers[].{name}'

    Find the index of the intended server from the list and replace the index in the command that appends clients. The list is 0-indexed, i.e., the index of the first entry is 0.

Advanced configuration on appliance with shuttle server​

Configure NAT​

If your setup requires NAT (e.g., clients' addresses must be translated on traffic leaving the server toward other networks), configure NAT in the firewall section of appliance config.

prepend mode

This configuration only works if the firewall mode is set to prepend mode.

Add a NAT chain under the firewall.tables section for the appliance configuration. Set the chaintype to "NAT". Set the hook to "POSTROUTING". Add a NAT rule under the chain. The rule sets the oifname or iifname to the name of the shuttle interface, and set the masquerade.

Existing NAT chain

If you already have a NAT chain in the appliance configuration, you can add the rules to the existing chain instead of creating a new one. Find the chain index using

appliance-cli get config -f 'body.config.firewall.tables[0].chains[].{name}'
SNAT on traffic initiating from shuttle clients
appliance-cli edit config 'config.firewall.tables[0].chains[]:{
  "chaintype": "NAT",
  "hook": "POSTROUTING",
  "name": "postrouting_chain",
  "policy": "ACCEPT",
  "priority": 0,
  "rules": [
    {
      "comment": "snat on traffic initiating from shuttle clients",
      "rule": "iifname \"shuttle0\"  masquerade",
      "sequence_id": 0
    }
  ]
}'
warning

When applying the source NAT to the traffic going to the shuttle clients, the NAT address must be put in the advertised_routes of the shuttle server config.

SNAT on traffic initiated from outside to shuttle clients
appliance-cli edit config 'config.firewall.tables[0].chains[]:{
  "chaintype": "NAT",
  "hook": "POSTROUTING",
  "name": "postrouting_chain",
  "policy": "ACCEPT",
  "priority": 0,
  "rules": [
    {
      "comment": "snat on traffic initiated from outside to shuttle clients",
      "rule": "oifname \"shuttle0\"  masquerade",
      "sequence_id": 0
    }
  ]
}'