Appliance Release v0.39
This page contains the release notes for the v0.39 Anapaya appliance software release. The appliance software release is applicable for the following Anapaya products:
- Anapaya CORE
- Anapaya EDGE
- Anapaya GATE
We recommend always upgrading to the latest available patch release. Please refer to Upgrade Notes (if any) of each release if there are any special steps to be taken when upgrading. For general information on how to upgrade your appliance, please refer to Software updates.
Known Issues
-
BGP LAN routes missing (affects v0.39.0 to v0.39.6)
Appliances that are connected to two BGP peers with the same BGP ASN, will sometimes lose routes in the kernel routing table from the BGP peers and therefore not export all routes to the SCION network. Upgrade to v0.39.7 or later to resolve this issue.
Quick fix and workaround
Quick fix
In case you detected that your appliance learns prefixes via a BGP session but does not propagate them via SCION, restart the BGP daemon such that the routing table is rebuilt:
appliance-cli post debug/services/frr/restart
Workaround
A configuration workaround without the need to install the new patch release is also available.
Refer to the advanced FRR configuration, select the template according to your appliance version and add the following line to the BGP configuration section after
ipv6 forwarding
.no zebra nexthop kernel enable
The zebra configuration section should look like this:
! Zebra configuration
!
ip forwarding
ipv6 forwarding
no zebra nexthop kernel enable
!Make sure to apply the configuration change and restart the BGP daemon afterwards:
appliance-cli put config/advanced/service-customization/frr/template < frr.conf.tmpl
appliance-cli post debug/services/frr/restart -
Disabling encryption can crash IP-in-SCION tunneling (affects v0.39.0 to v0.39.6)
When previously enabled security (encryption) is disabled in all domains, the IP-in-SCION tunneling component crashes which leads to service interruption. Upgrade to v0.39.7 or later to resolve this issue.
Quick fix
In case you disabled encryption on all domains and are experiencing service interruptions, please restart the affected services:
appliance-cli post debug/service-groups/data-plane/restart
appliance-cli post debug/services/gateway/restartIf this fails to resolve the issue, please reboot the appliance.
-
BGP external routes missing (affects v0.39.0 to v0.39.1)
Appliances that are configured with BGP will sometimes not export all routes learned via SCION to their BGP peers due to a race condition. FRR, the third-party BGP daemon used, will not recover unless the affected route is flushed. Upgrade to v0.39.2 or later, where we have optimized the route installation to work around this issue and have introduced metrics to monitor and alert if a route is not exported.
-
VRRP VRs go into INTF_DOWN state (affects v0.39.0)
VRRP VRs sometimes go into the INTF_DOWN state. This required a manual intervention to recover. Upgrade to v0.39.1 or later to resolve this issue.
-
Interface configuration fails if
gateway
field is used (affects v0.39.0 to v0.39.3)The appliance fails to configure interfaces, if the
gateway
field is used on an interface withVPP
driver variants. For the affected releases, please configure a0.0.0.0/0
and/or::/0
route instead of using thegateway
field. In a future release, thegateway
field will be deprecated. -
Legacy license tier takes precedence (unpatched)
In versions v0.38.0 and later, the legacy license tier incorrectly takes precedence over other license tiers. If both a legacy and a pro license are installed, the pro license won't be used. To fix this, list the installed licenses and remove the legacy one. (See License management for details.)
Upgrade Notes
After upgrading to v0.39, it is not possible to access secrets (passwords, forwarding key, etc.) that were contained in the appliance configuration in plain text prior to version v0.39.
Please make sure that you have stored a copy of the secrets in a secure location
prior to upgrading, if this is not already the case. When upgrading to v0.39,
the appliance will dump the secrets contained in the configuration to
/root/secrets_dump_v0_38.json
. This will allow you to retrieve the secrets and
store them in a secure location, provided you have root access to the appliance.
-
This release contains potentially breaking changes, please refer to Breaking Changes for more information.
-
This release requires the installation of the anapaya-system package v2.14.0 or later. The installer will automatically reject the installation if the required anapaya-system package is not available.
-
This software release also contains updated Grafana dashboards and alerting rules, please follow Recommended alerts and Recommended Grafana dashboards for more information how to update your monitoring setup.
Features
Appliance Secret Management
This release includes significant changes to the configuration interface due to a redesigned architecture for secret management. The API is designed so that secrets stored on the appliance cannot be revealed either intentionally or accidentally. As a result, configuration and observability data can be exchanged without the risk of exposure.
Starting with this release, secrets are decoupled from configuration. I.e., secrets are referenced in the appliance configuration rather than being included in plain text. Going forward, it will be safe to share the appliance configuration without any post-processing.
Because secrets are referenced in the appliance configuration, they need to be provisioned first. The appliance API is extended with suitable endpoints for secrets management.
Secret Identifier
Secrets are referenced in the configuration via a secret identifier:
- scheme:
<name>@<version>
- regex with permitted chars:
^[_a-zA-Z0-9:.-]+@[0-9]+$
The initial version must be 1
. Subsequent version must be incremented without
any gaps. I.e., if the current highest version is x
, the next version must be
x+1
. If you require a more elaborate versioning scheme, we recommend to encode
it in the name portion of the secret identifier.
Secrets Management
The appliance management API contains three new endpoints to manage secrets:
-
GET /v1/secrets
: List the existing secret identifiers.Example response
{
"secrets": [
{
"id": "forwarding-key_1-ff00:0:110@1",
},
{
"id": "loki-password@1",
}
{
"id": "loki-password@2",
}
]
} -
POST /v1/secrets
: Add a new secret.Example request body
{
"secret": {
"id": "bgp-password@1",
"value": "myBgpPassword"
}
} -
POST /v1/secrets/batch
: Add a batch of new secrets.Example request body
{
"secrets": [
{
"id": "bgp-password@1",
"value": "myBgpPassword"
},
{
"id": "forwarding-key_1-ff00:0:110@1",
"value": "Zm9yd2FyZGtleQo="
}
]
}
Deployment Workflow
So far, the appliance configuration contained plain text secrets directly. With the move to secret references, the appliances needs to be provisioned with secrets first, before they can be referenced in a configuration. This changes the deployment workflow slightly:
- Provision the secrets.
- Deploy the appliance configuration with secret references.
For example, to add a BGP password for an existing BGP neighbor, you can add the secret using:
# Interactive mode:
appliance-cli secrets add bgp-password@1
# Piping the secret from stdin:
echo myBgpPassword > secret.txt
cat secret.txt | appliance-cli secrets add bgp-password@1
# Using the management API endpoint:
echo '{"id": "bgp-password@1", "value": "myBgpPassword"}' > bgp-password.json
appliance-cli post secrets secret: @bgp-password.json
Update the appliance configuration to reference the secret:
appliance-cli edit config bgp.neighbors[0].auth_password_ref: bgp-password@1
The appliance configuration validation ensures that the referenced secret exists and has the expected format. For some secret references, a specific format of the referenced plain text secret is required. The API documentation of the secret reference field provides information on the expected format if necessary. For example, the forwarding key must be base64 encoded.
Configuration Migration
When upgrading to this release, the existing appliance configuration is automatically migrated. The appliance automatically chooses secret identifiers for the existing secrets in the appliance configuration.
For example, to find the generated secret identifier for the forwarding key you can use the following command:
appliance-cli get config -f body.config.scion.ases[0].forwarding_key_ref
If you desire a different naming scheme, you can re-provision the secrets with an identifier of your choice and update the references in a configuration update.
When upgrading to v0.39, the appliance will dump the secrets contained in the
configuration to /root/secrets_dump_v0_38.json
. This will allow you to
retrieve the secrets and store them in a secure location, provided you have root
access to the appliance.
If you rely on the the migration management API to prepare the configuration for the upgraded appliance, the automatic secrets migration will not be triggered. You need to manually provision the secrets using the new secrets management API. We recommend that you use the auto-migration when upgrading to v0.39.
RADIUS support
This release adds support for RADIUS authentication for the appliance. Together with PAM SSH connections can now be authenticated against a RADIUS server.
To enable RADIUS authentication for SSH connections, both the RADIUS server and PAM service need to be configured. The following example shows how to configure both the RADIUS server and the PAM service. Note that the users need to already exist on the appliance.
{
"config": {
"management": {
"ssh": {
"enable_password_login": true,
"radius": {
"servers": [
{
"description": "Radius server 1",
"secret_id_ref": "radius-secret@1",
"address": "10.0.0.1"
}
]
}
},
"pam": {
"services": [
{
"auth_modules": ["auth sufficient pam_radius_auth.so conf=/etc/pam_radius_auth.conf"],
"description": "SSH login customization",
"enabled": true,
"service": "sshd"
}
]
}
}
}
For more information how to configure RADIUS and PAM refer to the PAM configuration section.
Role-based access control for basic auth users
The appliance API now supports roles for basic auth users. The standard roles
"reader"
and "writer"
that already exit for OAuth2 users are now also
available in basic auth. Furthermore, a new role "observer"
has been added.
The "observer"
role has the same capabilities as the "reader"
role with the
addition of access on the POST /api/v1/debug/scion-tunneling/paths/search
and
POST /api/v1/tools/scion/*
endpoints. This allows observers to have more
network insights through additional API endpoints. This new role is available
for both basic auth and OAuth2 users. Going forward, the role is required for
basic auth users. Existing users are automatically migrated to have the
"writer"
role to keep the same set of capabilities.
Management API available via a UNIX socket
The appliance management API is now always exposed via a local UNIX socket. This
allows local privileged users to interact with the API without the need to
authenticate. This is useful as a way to always be able to connect to the
management API without any configuration or if other configured listeners are
not available. This local UNIX socket can be disabled by setting the
management.api.disable_local_unix_socket
configuration option to true
.
The appliance-cli
automatically uses the local UNIX socket if no other
appliance-cli context
is configured. When upgrading from an older version
where an appliance-cli context
is configured, you can either remove the
existing contexts (appliance-cli context delete
) or manually add a context to
use the local UNIX socket (appliance-cli context configure <name> --insecure
with base URI https+unix://var/run/caddy/api.sock
).
Breaking Changes
Secrets management
The secrets management has been completely overhauled. Secrets are now stored separately from the appliance configuration. For more information, refer to Secrets management.
SCION applications directly open an underlay UDP socket
SCION native applications now directly open a UDP socket on the operating system instead of going through the SCION dispatcher. To provide backwards compatibility, there is still a dispatcher component which dispatches traffic sent to port 30041 to the correct application.
This also means that SCION packets do not only use UDP port 30041 anymore, but
depend on the L4 port configured for the application. If you run multiple
appliances in a cluster setup and have a firewall between them, you need to
allow UDP traffic between the appliances on the port configured for the control
service (scion.ases[0].control.address
) and for the IP-in-SCION tunneling
control, data, and probe ports (scion-tunneling.endpoint.control_port
,
scion-tunneling.endpoint.data_port
, scion-tunneling.endpoint.probe_port
).
When upgrading to this release, be aware that SCION management API listeners will break as soon as you upgrade the anapaya-system package to the new v2.14.0 version. If you only have a SCION management listener configured, we recommend to also configure at least a localhost listener before upgrading the system package. If you do not have any SCION management listeners configured (the common case), you can safely upgrade to this release.
Finally, SCION management listeners on port 443 are no longer supported. The appliance will automatically migrate them to port 8443 on update. If this port is already used you will need to manually update the configuration before the upgrade.
Secret References
All fields in the appliance configuration that previously contained secrets have been replaced by fields that now refer to the secrets via the identifier.
The following fields have been changed:
-
old:
/bgp/neighbors/{neighbor}/auth_password
new:
/bgp/neighbors/{neighbor}/auth_password_ref
-
old:
/management/telemetry/logging/loki/basic_auth/password
new:
/management/telemetry/logging/loki/basic_auth/password_ref
-
old:
/management/api/oauth/identity_providers/client_secret
new:
/management/api/oauth/identity_providers/client_secret_ref
-
old:
/scion/ases/{as}/forwarding_key
new:
/scion/ases/{as}/forwarding_key_ref
-
old:
/scion/ases/{as}/ca_service/anapaya_vault/credentials/secret_id
new:
/scion/ases/{as}/ca_service/anapaya_vault/credentials/secret_id_ref
-
old:
/scion/ases/{as}/ca_service/external/shared_secret
new:
/scion/ases/{as}/ca_service/external/shared_secret_ref
Appliance UI
In an effort to focus our resources on the most useful features, we decided to remove the appliance UI. Very few users have been using it to configure their appliances. By now, the Anapaya Console offers a much better experience and we want to focus our efforts on that. Alternatively, you can still use the appliance CLI to interact with the appliance from the command line.
Metrics related to old flow exporting mechanism
A new flow exporting mechanism has been introduced with release v0.38.0 and has been the default since. The old flow exporting mechanism has been removed from the GATE appliance software.
As part of the removal the following configuration options have been removed:
management.telemetry.flow_metrics.cleanup_task_interval
as there is no explicit cleanup task anymore.management.telemetry.flow_metrics.max_active_flows
this is no longer configurable.
The following metrics have been removed, as they were only used by the old flow exporting mechanism:
gateway_flow_exporter_flows_exported
gateway_flow_exporter_cleanup_run_time
gateway_flow_exporter_lost
gateway_flow_exporter_export_run_time
gateway_flow_exporter_flows_limit
gateway_flow_exporter_flows_total
gateway_flow_exporter_last_cleanup_time
v0.39.7
Release date: 2025-09-12
Fixes
-
Fix a route table inconsistency that could lead to routes not being announced to IP-in-SCION tunneling peers. This could only happen if you have BGP configured and have multiple BGP peers with the same peer AS number.
-
When previously enabled encryption is disabled in all domains, the gateway no longer crashes.
-
Input filtering now considers the remote ISD-ASes of the domain instead of only the subset that is currently actively announcing the remote prefixes. Therefore, asymmetric traffic within a domain is forwarded correctly.
-
Fixes around IP-in-SCION tunneling metrics:
- Fix a bug that artificially reported discarded IP packets with reason
too_old
. - Fix a bug that increased the frames discarded with reason
too_old
even though no frames were actually discarded. - Fix the discarded IP packets metric (
gateway_ippkts_discarded_total
) wrongly reportingistream_counter
as discarded packets, as those packets are not discarded by the gateway. - The
gateway_ippkt_bytes_received_filtered_total
andgateway_ippkts_received_filtered_total
metrics now correctly report the packets filtered because of wrong combination of source and destination ISD-ASes. These packets are counted withreason
=bad_src_dst_ia
.
- Fix a bug that artificially reported discarded IP packets with reason
v0.39.6
Release date: 2025-09-01
Fixes
- Fix a crash of the IP-in-SCION tunneling component when receiving specifically crafted packets.
v0.39.5
Release date: 2025-05-19
Fixes for missing BGP prefixes
-
Outgoing BGP duplicate suppression is now disabled. When a BGP prefix experiences a rapid flap (i.e., a WITHDRAW followed quickly by a new ADVERTISE for the same prefix, typically within the Minimum Route Advertisement Interval (MRAI) interval), and the
suppress-duplicate
feature is enabled, the subsequent ADVERTISE message may be incorrectly suppressed. This occurs because the prefix is not removed from the Routing Information Base (RIB) until the WITHDRAW BGP UPDATE packet is actually sent to the peer. If the new ADVERTISE arrives for processing before this removal, thesuppress-duplicate
logic mistakenly identifies the new ADVERTISE as a duplicate of the previous (now withdrawn) advertisement still present in the RIB. This then leads to the BGP peer missing the prefix. This issue is now resolved by disabling thesuppress-duplicate
feature, which allows the new ADVERTISE to be sent immediately, regardless of the state of the previous advertisement. Disabling the the duplicate suppression has no negative impact, since the IP-in-SCION tunneling gateway already ensures that no duplicate prefixes are redistributed via BGP. We will follow up with a bug fix in the FRR project to make duplicate suppression work correctly in the future. -
The IP-in-SCION tunneling gateway does not immediately remove IP prefixes from a remote gateway anymore, when the prefix fetching fails. Instead, it will wait for up to 5s before removing IP prefixes that are not fetched successfully. This allows for a more graceful handling of temporary network issues, avoids unnecessary and large updates to the routing table and helps mitigating an issue where FRR would miss IP prefix updates if many of them happen in a short time span. Note that if a remote gateway explicitly retracts a prefix, it will be removed immediately without waiting for the TTL to expire. This will still guarantee immediate failover behavior in case the remote gateway loses internal connectivity to the prefix.
-
The remote gateway reachability monitoring does not consider a gateway unreachable if the underlying path used for reachability monitoring is dead. Now, a remote gateway can only be declared unreachable if it does not respond for at least 3 seconds over multiple paths. This solves an issue with IP prefixes being added and removed in short succession which FRR could sometimes miss.
-
The workaround for the missing BGP prefixes using
appliance-cli debug frr non-advertised-routes --fix
is now applied automatically without manual intervention. The appliance compares all the prefixes received from SCION peers with the prefixes announced to the BGP neighbor. If there is a difference, the appliance will trigger FRR to recalculate the BGP advertisement to the BGP neighbor (no traffic interruption). By default, this automated fix is disabled. It can be enabled on a per-BGP neighbor basis by settingbgp.neighbors.<neighbor>.automated_missing_prefixes_fix.enabled
totrue
. Furthermore, if some prefixes are expected to be missing, these prefixes can be configured inbgp.neighbors.<neighbor>.automated_missing_prefixes_fix.expected_missing_prefixes
.
Other Fixes
-
The IP-in-SCION tunneling gateway now correctly frees buffers on ingress stream handling which could trigger a buffer leak in rare conditions. Once the gateway was out of buffers, it needed to be reset to recover. This bug could be triggered for example, by a local misconfiguration of allowed interfaces.
-
The IP-in-SCION tunneling gateway now always correctly updates the metric concerning the expiration of the AS certificate being used for TLS handshakes. Previously, it was only updated when actual TLS handshakes were performed, which was not the case for example if no remote peers were configured. This could lead to the metric being stale and not reflecting the actual expiration time of the certificate.
Improvements
-
Core dumps are now periodically removed, retaining up to a configurable limit. By default, the latest 3 core dumps are kept. You can adjust this in the configuration under
system.resource.core-dump.keep
. -
Several log messages have been improved to occur less frequently in case of repeated errors. This reduces log spam and makes it easier to identify relevant issues.
-
The appliance now includes only the direct paths to its neighbors in the
appliance_cron_paths_to_neighbors
metric. In particular, this means that only paths with two hops and an interface that belongs to the appliance are counted. Previously, the metric would report all possible paths irrespective of their length or first hop interface. -
The
scion-pki certificate create
andscion-pki certificate sign
commands now have support for the--eku-any
flag. If this flag is set, the anyExtendedKeyUsage is set in the created certificate. This is especially useful for CP-PKI Root certificates as it allows the Root to also be used as a trust anchor during regular TLS certificate verification. Setting the extended key usage any is compatible with the current SCION CP-PKI specification.
v0.39.4
Release date: 2025-04-10
This release focuses on BGP integration fixes, including proper IPv6 support and resolving BGP BFD issues with Cisco devices.
This release changes the template for the FRR configuration. If you use template overrides you need to manually adapt the templates. Reach out to the Anapaya support to get the latest FRR template.
This release comes with new alert definitions. Install them in your alerting system, please follow Recommended alerts.
Fixes
-
Downgrade FRR to version 10.1.3. This fixes BGP BFP issues with Cisco devices. FRR 10.2.x and later versions are broken. We worked with the FRR team to fix them in a future version, until then we will use 10.1.3.
-
Fix Erroneous Alert: The
BGPUnexportedRoutes
alert would sometimes trigger incorrectly because it didn't differentiate between IPv4 and IPv6 metrics. This is now resolved. For compatibility with older versions (v0.39.2, v0.39.3), the old alert logic has been preserved under the nameBGPUnexportedRoutesLegacy
. -
Fix the configuration of IPv4 and IPv6 BGP sessions by separating route-maps and prefix-lists for each address family. This fixes an issue where IPv6 prefixes were previously unmatched.
-
The FRR exporter process is now correctly restarted when BFD is enabled or disabled and therefore no longer spams the logs with errors about not being able to connect to the BFD daemon, if BFD was disabled.
-
The
appliance-cli debug frr non-advertised-routes
command now correctly handles IPv6 neighbors.
v0.39.3
Release date: 2025-03-21
Improvements
- The debug scraper now reduces the attempts to fetch data from APIs that are not available, this can lead to a reduction of logs on the appliance-controller.
Fixes
-
Ensure that the network setup works correctly on appliances installed with the base image
base-image/sys_v2.15.0-scion_v0.39.2-1
. -
Upgrade to FRR 10.2.2, this fixes BGP BFD issues as described in the upstream FRR release notes
-
The
appliance-cli info
command now properly reports the remote interface ID of a SCION link. Previously, it was always 0. -
The appliance now correctly installs the default routes that are configured via the
gateway
field in theinterfaces
section of the appliance configuration on an interface withVPP
driver variants. -
Fix routes with multiple next-hops. In the case the interface of the first next hop has a carrier-down event the route now correctly fails over to the next next-hop, instead of failing.
v0.39.2
Release date: 2025-02-20
Improvements
-
Add a new BGPUnexportedRoutes alert that reports if there are IP-in-SCION tunneling routes, which are not exported to BGP neighbors along with a Runbook that explains how to resolve this issue. To use the new alert you will have to download the latest alert definitions
curl -O https://dl.cloudsmith.io/<access-token>/anapaya/stable/raw/names/anapaya-alerts-edge/versions/v0.39.2/anapaya-alerts-edge-v0.39.2.yml
and import them into your Prometheus instance.
Fixes
-
When calculating the number of buffers per NUMA for VPP the appliance controller now respects the number of NUMA nodes on the system. This ensures that the number of buffers per NUMA node is calculated correctly.
-
The appliance-cli is now able again to set service log levels and enable/disable interface dumps. This was broken in v0.39.0.
v0.39.1
Release date: 2025-02-12
Improvements
-
Write all levels, including the debug, to the FRR logs. This makes it easier to enable the debug logs when needed.
-
The IP-in-SCION tunneling gateway has improved logging of its monitoring activity. This will help with troubleshooting and monitoring the component's activity. To not spam the logs, most events are only logged if they constitute a change, i.e., if the same event happens multiple times in a row, then only a single log statement reporting this event will be present in the logs.
Fixes
-
If the username or the password in the basic auth settings for loki are unset, the configuration migration from v0.38 to v0.39 will drop the entire basic auth section for loki. This is done to not trigger validation errors in v0.39.
-
Setting MAC addresses on interfaces with Linux driver now works as expected. Previously, the MAC setting would sometimes not apply, furthermore, interfaces would not be correctly configured after being down.
-
The configuration migration now drops invalid features in the experiments section. Previously, the migration failed because there is a new validation that checks for validity of the features.
-
Reduce Linux route flaps when routing to remote IP-in-SCION endpoints changes.
-
Prevent VRRP VRs going into INTF_DOWN state which required the VR to be restarted or setting the VR interface admin down then up again. The VRRP VRs are now stopped on admin and carrier state down and started again on admin and carrier up.
v0.39.0
Release date: 2025-01-27
Improvements
-
Introduce an explicit
allow_hostname_change
query parameter to thePUT /config
API endpoint. By default, the appliance API now disallows changing the hostname, except when it is still unset. If the hostname is already set, the API will return a validation error. This is a safety measure to prevent accidental deployment of a configuration meant for a different appliance. If you want to change the hostname after it has been set, you need to set theallow_hostname_change
query parameter totrue
. -
The Appliance API is extended with a new
POST config/diff
endpoint which reports the diff between the current active configuration on the appliance, and the provided configuration. The diff is reported in a patch like structure to be easily understandable to operators. We recommend that you use this endpoint before applying a configuration change to sanity check the effects of your change. The endpoint also reports an E-Tag in the response headers, which you can use to ensure that in the subsequent configuration push, the appliance configuration has not changed on the device in the meantime. -
The
GET api/v1/health
now supports negative queries to filter out health checks. E.g., you can now passstatus=-passing
as a query parameter to filter out all health checks with the statuspassing
. -
The
api/v1/debug/bgp/config
endpoint now reports the BGP timers in seconds rather than milliseconds. -
FRR has been updated to version to v10.2.1. This addresses CVE-2024-55553
Fixes
-
Metrics are now always exposed, regardless of whether the
management.telemetry.address
is configured or not. Metrics have always also been exposed on the management API address, but this was only working if themanagement.telemetry.address
was configured - now they will also be exposed on the management API address even if no telemetry address is configured. -
The appliance-cli now properly handles
--query
and--header
values that contain a,
. Theappliance-cli get health --query status=degraded,failing
now works as intended. -
The
appliance-cli info
command now deals better with incomplete state when displaying CP-PKI information. Previously, the command relied on health checks to show TRC and certificate information. With this change, the/api/v1/config
,/api/v1/cppki/trcs
, and/api/v1/cppki/certificates
endpoints are queried for information, instead of only relying on health checks. -
Network interfaces with a configured IPv6 address can now be removed, without the need to deconfigure the IPv6 address first.