All notable changes to this project will be documented in this file.
The format is based on Keep a Changelog, and this project adheres to Semantic Versioning.
nebula ctl <command> provides access to all of the ssh debug server's commands.
It is built into both the nebula and nebula-service binaries, and a bare argument that is not ctl,
whether first or after the daemon's flags, is reported as a mistyped subcommand rather than ignored, which
used to start a second daemon. Access is via a unix domain socket everywhere except windows, which uses a
named pipe, and is limited to the user nebula runs as: the socket is 0600 and the daemon checks the peer's
credentials on every connection, so nebula ctl is run as that user, or with sudo when nebula runs as root.
The socket is /var/run/nebula/ctl.sock unless local_api.control.socket names another absolute path,
which a daemon that does not run as root, or a second daemon on one host, does. The directory holding the
socket must be owned by nebula's user or by root and be writable by its owner alone; one that is not is
refused, with the owner or mode named, since whoever could write it could put their own socket at the path. nebula ctl talks to the
socket named with -socket, or the one named in the config given with -config or found next to the
executable, else the default; both flags may come before or after the command, and
nebula -config <path> ctl <command> carries the daemon's config over. It never searches for a socket. On
windows the pipe is created under
\\.\pipe\ProtectedPrefix\Administrators\, where only administrators and SYSTEM may create names, so an
ordinary user cannot put something there ahead of the daemon; a daemon that is not an administrator says so
and names the fix. nebula ctl reads the owner of the pipe it connected to before sending anything and
refuses one that is not its own account, SYSTEM, Administrators or a service account, which is what
protects an override outside that namespace; the daemon refuses to serve a name another process already
holds and logs that as an error. local_api.control.group names a group that shares the socket, the way
dockerd -G does: the socket is created 0660 and owned by the group on unix, and the group is granted
access to the pipe on windows, so its members run nebula ctl as themselves; the daemon then leaves the
access check to the kernel. local_api.control.enabled: false turns the socket off, and a reload that
changes it removes the socket or brings it back.nebula ctl now takes -json and -pretty, rendered by one
shared implementation per command. nebula ctl prints human readable output by default; print-tunnel,
print-relays and query-lighthouse have no human form yet and ask for -json or -pretty rather than
inventing one. The ssh server's output with no flags is unchanged. Previously -json was ignored by some
commands, rejected by others, and taken as an address by query-lighthouse; print-cert -json -raw now
returns {"pem": ...} where -json used to silently win. A flag may come before or after an operand, so
print-tunnel 10.0.0.1 -json asks for JSON rather than dropping the flag, and -- ends the flags as it
always has. A negative number is a flag, as everywhere else in nebula; an operand that looks like one goes
after --.identity -address <vpn addr>[:port] prints the certificate identity of the host owning the address when a
tunnel to it is up, and whoami prints this node's own: name, vpn addresses, networks, unsafe networks, groups,
fingerprint, issuer, validity window and cert version. The port is ignored so a connection's remote address
can be passed through as is; identity without an address is refused rather than answered with this node, so
a program that failed to pass one cannot authorize against the daemon's own identity. Both are available over
the ssh debug server and nebula ctl, and are what the new opt-in info API serves, and all it serves:
local_api.info names a unix socket (a named pipe on windows), and GET /v1/identity?address=<addr> and
GET /v1/whoami on it answer with the same JSON, 404 when no tunnel to the address is up, so a program on the
host can resolve the source of a connection arriving over the overlay to an identity without a client library.
Who may use it is local_api.info.group, a group the socket is handed to at 0660 on unix and granted access to
the pipe on windows, with socket_mode to widen it further on unix; without a group only the user nebula runs
as can. The section is reloadable.start-cpu-profile, stop-cpu-profile, save-heap-profile, save-mutex-profile, and
mutex-profile-fraction commands, and sshd.sandbox_dir, which is now ignored if set.A peer whose certificate carries an unsafe network overlapping the overlay could send packets as any node in it: the firewall accepted any source inside the unsafe network. A source address inside this node's own networks must now be the sending peer's own; through an unsafe network it is dropped, counted as a dropped remote address.
Signing a v2 certificate with an empty name, a name longer than 253 bytes, or an empty group now fails. These certificates were signed successfully but could not be loaded.
create-tunnel -address <ip:port> panicked on a nil RemoteList and, over the ssh debug server, took nebula
down with it. A hostinfo returned by StartHandshake had no RemoteList until the handshake loop's first
pass attached one, and create-tunnel -address set the remote before that. The list is now attached when
the handshake starts, so a pending hostinfo never exists without one.
See the v1.11.1 milestone for a complete list of changes.
proto: any rule allows them. If you carry one of these protocols over the
overlay, confirm a proto: any rule covers it before upgrading, it may have been passing only through this
bypass. (#1840)github.com/cyberdelia/go-metrics-graphite, which has been unmaintained for over ten
years, by inlining the small amount of code Nebula used. (#1832)nebula-cert ca from running out of memory on 32bit systems when generating encrypted private keys. (#1834)ErrDumpInterrupted when listing tun addresses on Linux, so a transient interrupted netlink dump
no longer aborts startup. (#1835)See the v1.11.0 milestone for a complete list of changes.
slog. Log output changes: levels are upper case
(level=INFO), trace prints as level=DEBUG-4, timestamps are always RFC3339Nano and logging.timestamp_format
is ignored, and some messages were reworded. Review any log parsing before upgrading. This is also an API break
for embedders, as constructors now take a *slog.Logger. (#1672, #1734, #1621)firewall.inbound_action and firewall.outbound_action (used to set reject vs. drop policy) were each being
applied to the opposite direction, that is now corrected. This only affects how blocked packets are answered, not
which packets the firewall allows or denies. If you set either of these you are getting the behavior of the other
one today and likely want to swap them before upgrading. (#1798)tun.windows_bypass_wdf and listen.windows_bypass_wdf to false to leave WDF in charge. (#1710)private network category instead of whatever Windows decided,
which is usually Public. This makes the host firewall less restrictive on the overlay. Set
tun.network_category to unset to keep the old behavior. (#1710)sshd.sandbox_dir, which defaults to
$TMP/nebula-debug. Relative paths resolve inside it and absolute paths outside it are rejected, so anything
scripting start-cpu-profile, save-heap-profile, or save-mutex-profile with a path elsewhere needs the
directory set. The directory is not created for you. (#1622)- in nebula-cert to read from stdin or write to stdout. (#1714)config.yml and config.yaml in service and command line modes. (#1717)listen.rebind_on_network_change (default true, not reloadable). (#1816)NODATA instead of NXDOMAIN from the DNS server for a name that exists but has no record of the
requested type, so clients that query AAAA first (busybox/Alpine) fall through to A. (#1668)Control is now safe to stop and wait on from any lifecycle state, and a new Control.Wait blocks until nebula
has fully stopped and returns the first fatal reader error. Failed starts release the udp sockets and tun fd
instead of leaking them. (#1794)remote_allow_list IPv4 rules to 4-in-6 mapped addresses. (#1786)ed25519 key lengths. (#1601, #1756)closetunnel packets. (#1638)Disestablished, which silently dropped every send until dead tunnel detection forced a re-handshake. (#1805)use_system_route_table that was introduced in v1.10.1. (#1580)See the v1.10.1 milestone for a complete list of changes.
recv_error packets which defaults to always. (#1569)See the v1.10.0 milestone for a complete list of changes.
unsafe_routes. (#1332)pkcs11 tag (#1153, #1482)default_local_cidr_any now defaults to false, meaning that any firewall rule
intended to target an unsafe_routes entry must explicitly declare it via the
local_cidr field. This is almost always the intended behavior. This flag is
deprecated and will be removed in a future release. (#1373)rountines is > 1 on systems that don't support more than 1 routine. (#1531)any that negates a more restrictive filter. (#1513)nebula as a library with userspace networking to configure the logger and build version. (#1239)yaml.v3. (#1148, #1371, #1438, #1478)remote_allow_list in hole punch response. (#1186)tun.use_system_route_table is true and a route lacks a destination. (#1437)tun.use_system_route_table: true could result in heavy CPU utilization when many thousands of routes
are present. (#1326)static_host_map
which could cause rapid re-handshaking with an incorrect remote. (#1259)recv_error messages when a packet is received outside the allowable counter window. (#1459)tunnels.drop_inactive. See example config for more details. (#1413)default_local_cidr_any that defaults to
true to match previous behavior, but will default to false in the next
release (1.10). When set to false, local_cidr is matched correctly for
firewall rules on hosts acting as unsafe routers, and should be set for any
firewall rules you want to allow unsafe route hosts to access. See the issue
and example config for more details. (#1071, #1099)Nebula now has an official Docker image nebulaoss/nebula that is
distroless and contains just the nebula and nebula-cert binaries. You
can find it here: https://hub.docker.com/r/nebulaoss/nebula (#1037)
Experimental binaries for loong64 are now provided. (#1003)
Added example service script for OpenRC. (#711)
The SSH daemon now supports inlined host keys. (#1054)
The SSH daemon now supports certificates with sshd.trusted_cas. (#1098)
Config setting tun.unsafe_routes is now reloadable. (#1083)
Small documentation and internal improvements. (#1065, #1067, #1069, #1108, #1109, #1111, #1135)
Various dependency updates. (#1139, #1138, #1134, #1133, #1126, #1123, #1110, #1094, #1092, #1087, #1086, #1085, #1072, #1063, #1059, #1055, #1053, #1047, #1046, #1034, #1022)
Support for the deprecated local_range option has been removed. Please
change to preferred_ranges (which is also now reloadable). (#1043)
We are now building with go1.22, which means that for Windows you need at least Windows 10 or Windows Server 2016. This is because support for earlier versions was removed in Go 1.21. See https://go.dev/doc/go1.21#windows (#981)
Removed vagrant example, as it was unmaintained. (#1129)
Removed Fedora and Arch nebula.service files, as they are maintained in the upstream repos. (#1128, #1132)
Remove the TCP round trip tracking metrics, as they never had correct data and were an experiment to begin with. (#1114)
Fixed a potential deadlock introduced in 1.8.1. (#1112)
Fixed support for Linux when IPv6 has been disabled at the OS level. (#787)
DNS will return NXDOMAIN now when there are no results. (#845)
Allow :: in lighthouse.dns.host. (#1115)
Capitalization of NotAfter fixed in DNS TXT response. (#1127)
Don't log invalid certificates. It is untrusted data and can cause a large volume of logs. (#1116)
golang.org/x/crypto, which includes a fix for CVE-2023-48795. (#1048)Fix a deadlock introduced in v1.8.0 that could occur during handshakes. (#1044)
Fix mobile builds. (#1035)
Linux: Notify systemd of service readiness. This should resolve timing issues
with services that depend on Nebula being active. For an example of how to
enable this, see: examples/service_scripts/nebula.service. (#929)
Windows: Use Registered IO (RIO) when possible. Testing on a Windows 11 machine shows ~50x improvement in throughput. (#905)
FreeBSD: Add support for naming tun devices. (#903)
pki.disconnect_invalid will now default to true. This means that once a
certificate expires, the tunnel will be disconnected. If you use SIGHUP to
reload certificates without restarting Nebula, you should ensure all of your
clients are on 1.7.0 or newer before you enable this feature. (#859)
Limit how often a busy tunnel can requery the lighthouse. The new config
option timers.requery_wait_duration defaults to 60s. (#940)
The internal structures for hostmaps were refactored to reduce memory usage and the potential for subtle bugs. (#843, #938, #953, #954, #955)
Lots of dependency updates.
Windows: Retry wintun device creation if it fails the first time. (#985)
Fix issues with firewall reject packets that could cause panics. (#957)
Fix relay migration during re-handshakes. (#964)
Various other refactors and fixes. (#935, #952, #972, #961, #996, #1002, #987, #1004, #1030, #1032, ...)
static_host_map config was changed. (#886)static_host_map DNS lookup queries being
treated as IPv6 addresses. (#877)nebula-cert ca now supports encrypting the CA's private key with a
passphrase. Pass -encrypt in order to be prompted for a passphrase.
Encryption is performed using AES-256-GCM and Argon2id for KDF. KDF
parameters default to RFC recommendations, but can be overridden via CLI
flags -argon-memory, -argon-parallelism, and -argon-iterations. (#386)
Support for curve P256 and BoringCrypto has been added. See README section "Curve P256 and BoringCrypto" for more details. (#865, #861, #769, #856, #803)
New firewall rule local_cidr. This could be used to filter destinations
when using unsafe_routes. (#507)
Add unsafe_route option install. This controls whether the route is
installed in the systems routing table. (#831)
Add tun.use_system_route_table option. Set to true to manage unsafe routes
directly on the system route table with gateway routes instead of in Nebula
configuration files. This is only supported on Linux. (#839)
The metric certificate.ttl_seconds is now exposed via stats. (#782)
Add punchy.respond_delay option. This allows you to change the delay
before attempting punchy.respond. Default is 5 seconds. (#721)
Added SSH commands to allow the capture of a mutex profile. (#737)
You can now set lighthouse.calculated_remotes to make it possible to do
handshakes without a lighthouse in certain configurations. (#759)
The firewall can be configured to send REJECT replies instead of the default DROP behavior. (#738)
For macOS, an example launchd configuration file is now provided. (#762)
Lighthouses and other static_host_map entries that use DNS names will now
be automatically refreshed to detect when the IP address changes. (#796)
Lighthouses send ACK replies back to clients so that they do not fall into connection testing as often by clients. (#851, #408)
Allow the listen.host option to contain a hostname. (#825)
When Nebula switches to a new certificate (such as via SIGHUP), we now
rehandshake with all existing tunnels. This allows firewall groups to be
updated and pki.disconnect_invalid to know about the new certificate
expiration time. (#838, #857, #842, #840, #835, #828, #820, #807)
Always disconnect blocklisted hosts, even if pki.disconnect_invalid is
not set. (#858)
Dependencies updated and go1.20 required. (#780, #824, #855, #854)
Fix possible race condition with relays. (#827)
FreeBSD: Fix connection to the localhost's own Nebula IP. (#808)
Normalize and document some common log field values. (#837, #811)
Fix crash if you set unlucky values for the firewall timeout configuration options. (#802)
Make DNS queries case insensitive. (#793)
Update example systemd configurations to want nss-lookup. (#791)
Errors with SSH commands now go to the SSH tunnel instead of stderr. (#757)
Fix a hang when shutting down Android. (#772)
Refuse to process underlay packets received from overlay IPs. This prevents confusion on hosts that have unsafe routes configured. (#741)
The ssh reload command did not work on Windows, since it relied on sending
a SIGHUP signal internally. This has been fixed. (#725)
A regression in v1.5.2 that broke unsafe routes on Mobile clients has been fixed. (#729)
Experimental: nebula clients can be configured to act as relays for other nebula clients. Primarily useful when stubborn NATs make a direct tunnel impossible. (#678)
Configuration option to report manually specified ip:ports to lighthouses. (#650)
Windows arm64 build. (#638)
punchy and most lighthouse config options now support hot reloading. (#649)
Build against go 1.18. (#656)
Promoted routines config from experimental to supported feature. (#702)
Dependencies updated. (#664)
Packets destined for the same host that sent it will be returned on MacOS. This matches the default behavior of other operating systems. (#501)
unsafe_route configuration will no longer crash on Windows. (#648)
A few panics that were introduced in 1.5.x. (#657, #658, #675)
listen.send_recv_error to control the conditions in which
recv_error messages are sent. Sending these messages can expose the fact
that Nebula is running on a host, but it speeds up re-handshaking. (#670)x509 config stanza support has been removed. (#685)No longer fatals if expired CA certificates are present in pki.ca, as long as 1 valid CA is present. (#599)
nebula-cert will now enforce ipv4 addresses. (#604)
Warn on macOS if an unsafe route cannot be created due to a collision with an existing route. (#610)
Warn if you set a route MTU on platforms where we don't support it. (#611)
Rare race condition when tearing down a tunnel due to recv_error and sending packets on another thread. (#590)
Bug in routes and unsafe_routes handling that was introduced in 1.5.0. (#595)
-test mode no longer results in a crash. (#602)
x509.ca config alias for pki.ca. (#604)golang.org/x/crypto to address an issue which allowed unauthenticated clients to cause a panic in SSH
servers. (#603)(This release was skipped due to discovering #610 and #611 after the tag was created.)
SSH print-cert has a new -raw flag to get the PEM representation of a certificate. (#483)
New build architecture: Linux riscv64. (#542)
New experimental config option remote_allow_ranges. (#540)
New config option pki.disconnect_invalid that will tear down tunnels when they become invalid (through expiry or
removal of root trust). Default is false. Note, this will not currently recognize if a remote has changed
certificates since the last handshake. (#370)
New config option unsafe_routes.<route>.metric will set a metric for a specific unsafe route. It's useful if you have
more than one identical route and want to prefer one against the other. (#353)
Build against go 1.17. (#553)
Build with CGO_ENABLED=0 set, to create more portable binaries. This could
have an effect on DNS resolution if you rely on anything non-standard. (#421)
Windows now uses the wintun driver which does not require installation. This driver
is a large improvement over the TAP driver that was used in previous versions. If you had a previous version
of nebula running, you will want to disable the tap driver in Control Panel, or uninstall the tap0901 driver
before running this version. (#289)
Darwin binaries are now universal (works on both amd64 and arm64), signed, and shipped in a notarized zip file.
nebula-darwin.zip will be the only darwin release artifact. (#571)
Darwin uses syscalls and AF_ROUTE to configure the routing table, instead of
using /sbin/route. Setting tun.dev is now allowed on Darwin as well, it
must be in the format utun[0-9]+ or it will be ignored. (#163)
preferred_ranges option has been supported as a replacement for
local_range since v1.0.0. It has now been documented and local_range
has been officially deprecated. (#541)Valid recv_error packets were incorrectly marked as "spoofing" and ignored. (#482)
SSH server handles single exec requests correctly. (#483)
Signing a certificate with nebula-cert sign now verifies that the supplied
ca-key matches the ca-crt. (#503)
If preferred_ranges (or the deprecated local_range) is configured, we
will immediately switch to a preferred remote address after the reception of
a handshake packet (instead of waiting until 1,000 packets have been sent).
(#532)
A race condition when punchy.respond is enabled and ensures the correct
vpn ip is sent a punch back response in highly queried node. (#566)
Fix a rare crash during handshake due to a race condition. (#535)
Ability to output qr code images in print, ca, and sign modes for nebula-cert.
This is useful when configuring mobile clients. (#297)
Experimental: Nebula can now do work on more than 2 cpu cores in send and receive paths via
the new routines config option. (#382, #391, #395)
ICMP ping requests can be responded to when the tun.disabled is true.
This is useful so that you can "ping" a lighthouse running in this mode. (#342)
Run smoke tests via make smoke-docker. (#287)
More reported stats, udp memory use on linux, build version (when using Prometheus), firewall, handshake, and cached packet stats. (#390, #405, #450, #453)
IPv6 support for the underlay network. (#369)
Darwin will now log stdout/stderr to a file when using -service mode. (#303)
Example systemd unit file now better arranged startup order when using sshd
and other fixes. (#317, #412, #438)
Reduced memory utilization/garbage collection. (#320, #323, #340)
Reduced CPU utilization. (#329)
Build against go 1.16. (#381)
Refactored handshakes to improve performance and correctness. (#401, #402, #404, #416, #451)
Lighthouse performance and correctness improvements. (#406, #418, #429, #433, #437, #442, #449)
Better ordered startup to enable sshd, stats, and dns subsystems to listen on
the nebula interface. (#375)
No longer report handshake packets as lost in stats. (#331)
Orphaned pending hostmap entries are cleaned up. (#344)
Refuse to run a lighthouse on an ephemeral port. (#399)
Reloading via ssh command avoids a panic. (#447)
Shutdown is now performed in a cleaner way. (#448)
Logs will now find their way to Windows event viewer when running under -service mode
in Windows. (#443)
You can emit statistics about non-message packets by setting the option
stats.message_metrics. You can similarly emit detailed statistics about
lighthouse packets by setting the option stats.lighthouse_metrics. See
the example config for more details. (#230)
We now support freebsd/amd64. This is experimental, please give us feedback. (#103)
We now release a binary for linux/mips-softfloat which has also been
stripped to reduce filesize and hopefully have a better chance on running on
small mips devices. (#231)
You can set tun.disabled to true to run a standalone lighthouse without a
tun device (and thus, without root). (#269)
You can set logging.disable_timestamp to remove timestamps from log lines,
which is useful when output is redirected to a logging system that already
adds timestamps. (#288)
Handshakes should now trigger faster, as we try to be proactive with sending them instead of waiting for the next timer tick in most cases. (#246, #265)
Previously, we would drop the conntrack table whenever firewall rules were changed during a SIGHUP. Now, we will maintain the table and just validate that an entry still matches with the new rule set. (#233)
Debug logs for firewall drops now include the reason. (#220, #239)
Logs for handshakes now include the fingerprint of the remote host. (#262)
Config item pki.blacklist is now pki.blocklist. (#272)
Better support for older Linux kernels. We now only set SO_REUSEPORT if
tun.routines is greater than 1 (default is 1). We also only use the
recvmmsg syscall if listen.batch is greater than 1 (default is 64).
(#275)
It is possible to run Nebula as a library inside of another process now. Note that this is still experimental and the internal APIs around this might change in minor version releases. (#279)
pki.blacklist is deprecated in favor of pki.blocklist with the same
functionality. Existing configs will continue to load for this release to
allow for migrations. (#272)advmss is now set correctly for each route table entry when tun.routes
is configured to have some routes with higher MTU. (#245)
Packets that arrive on the tun device with an unroutable destination IP are
now dropped correctly, instead of wasting time making queries to the
lighthouses for IP 0.0.0.0 (#267)
Add logging.timestamp_format config option. The primary purpose of this
change is to allow logging timestamps with millisecond precision. (#187)
Support unsafe_routes on Windows. (#184)
Add lighthouse.remote_allow_list to filter which subnets we will use to
handshake with other hosts. See the example config for more details. (#217)
Add lighthouse.local_allow_list to filter which local IP addresses and/or
interfaces we advertise to the lighthouses. See the example config for more
details. (#217)
Wireshark dissector plugin. Add this file in dist/wireshark to your
Wireshark plugins folder to see Nebula packet headers decoded. (#216)
systemd unit for Arch, so it can be built entirely from this repo. (#216)
See deprecated, this also adds a new punchy.delay option that defaults to 1s.
Validate all lighthouse.hosts and static_host_map VPN IPs are in the
subnet defined in our cert. Exit with a fatal error if they are not in our
subnet, as this is an invalid configuration (we will not have the proper
routes set up to communicate with these hosts). (#170)
Use absolute paths to system binaries on macOS and Windows. (#191)
Add configuration options for handshakes. This includes options to tweak
try_interval, retries and wait_rotation. See example config for
descriptions. (#179)
Allow -config file to not end in .yaml or yml. Useful when using
-test and automated tools like Ansible that create temporary files without
suffixes. (#189)
The config test mode, -test, is now more thorough and catches more parsing
issues. (#177)
Various documentation and example fixes. (#196)
Dependencies updated. (#188)
punchy, punch_back configuration options have been collapsed under the
now top level punchy config directive. (#210)punchy.punch - This is the old punchy option. Should we perform NAT hole
punching (default false)?
punchy.respond - This is the old punch_back option. Should we respond to
hole punching by hole punching back (default false)?
Reduce memory allocations when not using unsafe_routes. (#198)
Ignore packets from self to self. (#192)
MTU fixed for unsafe_routes. (#209)
nebula
-service. If you are building from source, use make service to build this feature.mips, mips64, 386 and ppc64le processors on Linux.lighthouse.dns.host
and lighthouse.dns.port.unsafe_routes section to your
config to allow hosts to act as gateways to other subnets. Read the example
config for more details. This is supported on Linux and macOS.0, never update the lighthouse (mobile
optimization).group: ["one-group"], this will
now be accepted, with a warning to use group: "one-group" instead.listen.host configuration option was previously ignored (the bind host
was always 0.0.0.0). This option will now be honored.ca_sha and ca_name firewall rule options should now work correctly.