Skip to main content

Troubleshooting

This page describes the common operations you can perform on the SCION-CA. They are mostly a reference for actions that are needed during the runbooks, but they can also be used as a reference for other operations.

Start the vaultca service​

  1. SSH to the given machine.

  2. Run the following command:

    systemctl start vaultca.service

  3. Ensure that there are no errors in the logs (see Inspect vaultca service logs) and that the status of the service is active (running) (see Check whether vaultca service is running).

Check whether vaultca service is running​

  1. SSH to the given machine.

  2. Run the following command:

    systemctl status vaultca.service

  3. The output will include the status of the service, which should be active (running).

Inspect vaultca service logs​

  1. SSH to the given machine.

  2. Run the following command:

    journalctl -u vaultca.service
    tip

    To see only the recent logs use:

    journalctl -u vaultca.service --since=<time-duration>

    For example, to check the logs of the last minute, run

    journalctl -u vaultca.service --since=1m
    tip

    The logs are also exposed to Loki and can be viewed with Grafana.

Manually renew CA certificates​

  1. SSH to the given machine.

  2. Run the following command. By default, the configuration file of the vaultca service is located at /etc/vaultca/config.toml. If the configuration file is located somewhere else, make sure to pass the correct path as input to the --config flag.

    vaultca renew --config=/etc/vaultca/config.toml
    note

    This will update the CA certificates for all the ISD-ASes configured on the host. To update the CA certificate for a specific ISD-AS, you can pass the --isd-as flag as an input.

    For example, to renew the CA certificate for ISD-AS 1-ff00:0:1, you can run:

    vaultca renew --config=/etc/vaultca/config.toml --isd-as=1-ff00:0:1

Inspect TRC​

  1. SSH to the machine that runs the Vault instance.
  2. Get the latest version of the TRC using the vault kv get command and following the official Vault documentation. The latest TRC is stored under the path ca/<isd-as>/kv/trcs/latest.
  3. Use the scion-pki trc inspect to inspect the TRC details.

Push new TRC​

To add a new TRC to the key/value engine, use the /ca/<isd-as>/kv/data/trcs/ISD<isd>-B<base>-S<serial> endpoint. The TRC must be PEM encoded.

note

Make sure that the TRC is PEM encoded. The raw ASN.1 DER encoding is not supported. You can format any TRC to PEM encoding with the following command::

scion-pki trc format <TRC-file>

For example, to upload the TRC ISD1-B1-S4, you could use the following curl command:

curl \
  --header "X-Vault-Token: $VAULT_TOKEN" \
  --request POST \
  --data '{"data": {"trc": "-----BEGIN TRC-----\nMIIJ..."}}' \
  https://<vault-addr>/v1/ca/1-ff00:0:110/kv/data/trcs/ISD1-B1-S4

If the uploaded TRC is the new latest TRC, you also need to upload it to /ca/<isd-as>/kv/data/trcs/latest. This could be done with the following curl command:

curl \
  --header "X-Vault-Token: $VAULT_TOKEN" \
  --request POST \
  --data '{"data": {"trc": "-----BEGIN TRC-----\nMIIJ..."}}' \
  https://<vault-addr>/v1/ca/1-ff00:0:110/kv/data/trcs/latest
note

For the SCION CP-PKI AS certificate to be verifiable, the Anapaya appliance should also be provisioned with this TRC. Furthermore, the CA certificate potentially has to be rotated if the issuing Root Certificate has changed in the TRC update.

Provision new root CA engine​

  1. SSH to the machine that runs the Vault instance.

  2. Ensure that the root PKI engine is enabled by following the official Vault documentation. Add the following fields in the JSON-formatted body of the request:

    • The path for the PKI engine is root/<ISD-AS>/pki.
    • The type should be marked as pki.
    • You can add an optional description such as root PKI for <ISD-AS>.

  3. Tune the root PKI engine with default and max lease TTL. Follow the official Vault documentation and configure the default_lease_ttl to be 264h and the max_lease_ttl to be 720h.

  4. Generate a new root key and certificate bundle.

  5. Configure root PKI with the root key and root certificate bundle. Follow the official Vault documentation and use the root/<ISD-AS>/pki/config/ca path. Include as pem_bundle the root key and certificate generated in the previous step.

Add an AS to the blocklist​

To add an AS to the blocklist, you need to create an entry in the key/value enginewith the /ca/<ca-isd-as>/kv/data/blocklist/<blocked-isd-as> endpoint. For example, to block ISD-AS 1-ff00:0:112 in the CA 1-ff00:0:110, you could use the following curl command:

curl \
  --header "X-Vault-Token: $VAULT_TOKEN" \
  --request POST \
  --data '{"data": {"blocked": true}}' \
  https://<vault-addr>/v1/ca/1-ff00:0:110/kv/data/blocklist/1-ff00:0:112
note

The "blocked": true is for informational purposes only. The existence of the entry blocks the 1-ff00:0:112 from getting a renewed SCION CP-PKI AS certificate. You are free to add any additional informational metadata.

Remove an AS from the blocklist​

To remove an AS from the blocklist, you need to delete the entry in the key/value engine with the /ca/<ca-isd-as>/kv/metadata/blocklist/<blocked-isd-as> endpoint. For example, to remove the ISD-AS 1-ff00:0:112 in the CA 1-ff00:0:110 from the blocklist, you could use the following curl command:

curl \
  --header "X-Vault-Token: $VAULT_TOKEN" \
  --request DELETE \
  https://<vault-addr>/v1/ca/1-ff00:0:110/kv/metadata/blocklist/1-ff00:0:112

Rolling CA Provision Strategy​

note

The rolling CA provision strategy is implemented by the vaultca service that is part of the Anapaya SCION CA product.

Vault is designed such that only one CA certificate is used per PKI engine. For example, it is not possible to query what CA certificate was used to issue an AS certificate via the /pki/cert endpoint. Together with the short lifetime of the CA certificates, this severely limits the observability and auditability of the system.

The remedy for this problem is a rolling CA certificate provisioning strategy. In this strategy, you do not reconfigure the PKI engine with the updated CA certificate and private key. Instead, the previous PKI engine is archived, and a new PKI engine with the new crypto material is created. This must be transparent to the CA adapter to simplify configuration and business logic. You can achieve this with a sequence of /sys/remount operations:

  1. Create the new PKI engine at ca/<isd-as>/pki-next and provision it with the fresh CA certificate and private key.
  2. Archive the current PKI engine by remounting ca/<isd-as>/pki to ca/<isd-as>/pki_<timestamp>, where the timestamp is the current date in yyyymmddHHMMSS format. E.g., ca/1-ff00:0:110/pki_20210624090000.
  3. Remount the new PKI engine from ca/<isd-as>/pki-next to ca/<isd-as>/pki.

This process can be scripted, and thus should cause a very minimal downtime. The advantage here is, that the CA adapter does not need to decide which PKI engine instance to talk to. It is always reachable under the same path at ca/<isd-as>/pki.

note

There is a practical limit on the number of mount points of ~14000. If you rotate the CA certificate every 14 days and run a single CA in Vault, you run out of mount points after more than 250 years. If you run more CAs in the same Vault cluster, this number decreases.

https://www.vaultproject.io/docs/internals/limits#mount-point-limits

The current amount of mounts can be monitored via the /sys/mounts endpoint. Before you exhaust the limits, you can drop old PKI engines that are no longer needed for auditing purposes.