multiport: bring docs and comments in line with the lane design

examples/config.yml still described lanes as extra tunnels with their own
handshakes, built lazily per routine. Rewrite it for lanes as sessions derived
from the base tunnel, and say that multiport.lanes only bounds sending.

In design/multiport.md, correct the partner-lane pairing (our lane s pairs
with the peer's lane s+portOffset, which laneBias lines up), the asymmetric
host example, the retry backoff, which reader owns what, the full-path
diagram, the demand-after-reset story and the metric descriptions.

Drop comments left over from earlier iterations: base traffic on "socket 0",
lanes picked by routine, lane handshakes, and multiport defaulting on.

control.go

@@ -212,7 +212,7 @@ func (c *Control) RebindUDPServer() {
 		return
 	}
 
-	// Every socket needs rebinding, not just the base: with multiport each one is bound to its own lane port, and
+	// Every socket needs rebinding, not just the base: with multiport the sockets are spread over the lane ports, and
 	// even without it the surplus SO_REUSEPORT sockets stay pinned to the interface we came up on otherwise.
 	//
 	// A failure here means we are likely still pinned to the interface we came up on, so the rest of this is
@@ -420,7 +420,7 @@ func copyHostInfo(h *HostInfo, preferredRanges []netip.Prefix) ControlHostInfo {
 
 // copyLanes snapshots the sendable multiport lanes of a tunnel, or nil when it
 // has none. txAddr is the lane's gate as well as its destination, so a nil load
-// is exactly "this lane is down and its routine is riding the base tunnel".
+// is exactly "this lane is down and its flows are riding the base tunnel".
 func copyLanes(h *HostInfo) []ControlLane {
 	ls := h.lanes
 	if ls == nil || ls.txLanes < 2 {

design/multiport.md

@@ -108,10 +108,11 @@ egressSock(q)  = laneSock(q, 0)                          // base traffic
 
 Two properties fall out of this, and the rest of the design depends on both:
 
-1. **Every routine owns exactly one socket.** `listenIn` blocks in `recvmmsg`
-   and everything downstream of it -- the batcher, the conntrack cache, the
-   `txQueue` -- is single-owner and lock-free. More sockets than routines would
-   need epoll or locks.
+1. **Every routine owns exactly one socket.** Routine `i`'s UDP reader
+   (`listenOut`) blocks in `recvmmsg` on socket `i` and owns its batcher; its tun
+   reader (`listenIn`) owns its `txQueue` and conntrack cache. All of it is
+   single-owner and lock-free. More sockets than routines would need epoll or
+   locks.
 2. **Every routine has a sibling socket at the same group position on every
    port.** So socket selection is a pure function of `(queue, lane)` with no
    borrowing and no shared state, and each port's traffic spreads across its
@@ -145,8 +146,9 @@ txLanes   = min(myLanes, peerPortCount, sessions) // we may only SEND on ports t
 ```
 
 Sizing RX by the peer's count and TX by our own is what lets asymmetric hosts
-work: a 4-port laptop talking to a 32-port server sends on 4 lanes and receives
-on 32.
+work: a 4-port laptop talking to a 32-port server sends on 4 lanes, and keeps
+room for 32 lane sessions so it can receive on anything the server advertised it
+might send on. The server, bounded by the laptop's 4 ports, sends on 4.
 
 ### Port pairing
 
@@ -160,15 +162,19 @@ The rotation has to cancel, though, or the two directions of one flow would take
 unrelated 4-tuples and neither side's traffic would arrive through the conntrack
 or NAT entry the other's probe opened. So both sides hash the *same* sorted pair
 and the higher-addressed side **negates** the result. When the port counts match,
-the two rotations cancel exactly: our lane `s`'s 4-tuple is the reverse of the
-peer's lane `s`. `laneBias` does the matching rotation on the flow hash for the
-same reason.
+the two rotations cancel exactly on ports: our lane `s` runs from our port
+`base+s` to the peer's port `base+s+portOffset`, and the peer's lane
+`s+portOffset` runs back the other way, so the two are each other's reverse
+4-tuple. The lane *indices* don't cancel -- our lane `s` partners the peer's lane
+`s+portOffset` (mod the port count) -- and `laneBias` makes up for that: it
+rotates the flow hash by the same amount, so both directions of a flow pick
+partner lanes.
 
 That pairing only exists when our lane indices map one-to-one onto the peer's
 ports, which is exactly the `txLanes == peerPortCount` test `newLaneSet` applies
 before setting `laneBias`. If we send on 4 lanes and the peer bound 8 ports, four
 of its ports have no lane of ours pointing at them, and no choice of rotation can
-make our lane `s` and its lane `s` be each other's reverse. So in that case
+make each of our lanes the reverse of one of its lanes. So in that case
 `laneBias` stays 0 and each side hashes the flow to a lane on its own. The flow
 still works and is still spread; the two directions simply take two unrelated
 4-tuples instead of one 4-tuple and its exact reverse, and each direction depends
@@ -211,9 +217,9 @@ keepalive cannot keep an otherwise idle tunnel from being dropped by
 
 | timer | value |
 |---|---|
-| probe timeout | 2s (shorter than the 5s tick on purpose) |
+| probe timeout | 2s (shorter than the default 5s tick on purpose) |
 | keepalive | 30s |
-| retry backoff | 5s, doubling to 60s |
+| retry backoff | 10s, doubling to 60s |
 | max failure count | 8 |
 
 Traffic on a lane is not evidence the lane works -- that is the whole reason
@@ -296,7 +302,8 @@ nothing at all on one that doesn't.
      v
   tun queue q  ->  routine q
      |
-     |  s = laneFlowHash(flow) % txLanes      lane s arrives on port base+s
+     |  s = (laneFlowHash(flow) + laneBias)   lane s arrives on the peer's
+     |      % txLanes                         port for lane s
      v                                           |
   lane s session                                 |  SO_REUSEPORT hash of the
      |                                           |  4-tuple picks one socket
@@ -407,7 +414,7 @@ configuration:
 
 With a dynamic `listen.port: 0`, the first socket binds dynamically and the
 range is claimed above it; a partially-occupied range re-rolls with a fresh
-dynamic port up to 6 times.
+dynamic port, for up to 6 attempts in all.
 
 `laneSock` collapses to the identity when multiport is off, so the send path is
 unchanged: a routine writes to its own socket and no lane batches are built at
@@ -440,8 +447,9 @@ decided per packet with no state to unwind:
   all when there is no direct path and rebuilds when one returns;
 - the peer **roamed** -- a new NAT mapping has no derivable relationship to the
   old lane ports, so every lane is torn down and re-probed from a clean backoff.
-  Standing demand is deliberately kept across a reset, so the lanes that were
-  actually carrying data come back first.
+  Demand is kept across a reset, and the first packet of a flow that hashes onto
+  a lane that is now down raises it again, so lanes still carrying flows are
+  re-probed on the next tick.
 
 ### Always on the base tunnel
 
@@ -513,8 +521,9 @@ way to tell "no lanes negotiated" (key absent) from "lanes up but traffic on one
 Metrics, registered only when multiport is running so they don't sit at zero on
 nodes without it:
 
-- `multiport.lanes.up` -- lanes currently carrying traffic
-- `multiport.lanes.tunnels` -- tunnels with any lanes
+- `multiport.lanes.up` -- lanes currently proven usable for sending
+- `multiport.lanes.tunnels` -- tunnels with a lane set, including ones with no
+  lane up and receive-only ones
 
 Both are counted by walking the hostmap rather than kept at promotion/demotion,
 because a counter would drift upward forever: a tunnel torn down with its lanes

examples/config.yml

@@ -190,23 +190,24 @@ listen:
 # socket sharing listen.port, multiport.ports consecutive ports are bound and
 # each gets its own group of `routines` sockets sharing it via SO_REUSEPORT, so
 # no port (the base port above all, which carries every handshake and every
-# vanilla peer) depends on a single core. One extra tunnel ("lane") per port is
-# negotiated with capable peers: lane i handshakes from local port listen.port+i
-# to the peer's advertised base+((i + pair_offset) mod peer_ports), where
-# pair_offset is a per-pair hash that spreads many small peers across a big
-# peer's whole port range.
-# Each lane is a full Noise session with its own
-# keys, nonce counter and replay window, so flows taking different paths
-# never fight over shared replay state.
+# vanilla peer) depends on a single core. Each port carries one lane: lane i
+# sends from local port listen.port+i to the peer's advertised
+# base+((i + pair_offset) mod peer_ports), where pair_offset is a per-pair hash
+# that spreads many small peers across a big peer's whole port range. Lane 0 is
+# the base tunnel itself.
+# A lane is not a separate tunnel and has no handshake of its own. Both sides
+# derive its keys from the base tunnel's, so it comes and goes with the base
+# tunnel, but it has its own nonce counter and replay window, so flows taking
+# different paths never fight over shared replay state. Each inside flow is
+# hashed onto one lane for its whole life, so multiport adds no reordering.
 #
 # Peers negotiate lanes in the handshake; vanilla peers get a single normal
 # tunnel. All control traffic (handshakes, lighthouse, punching, relays) and
-# the data fallback stay on the base tunnel/port. Lanes are built lazily: a
-# lane is only established once a routine actually has traffic for that peer
-# and no lane to carry it, so a peer you exchange a trickle with costs one
-# tunnel regardless of how many routines are configured. Established lanes
-# are kept alive with their own keepalives, and traffic falls back to the base
-# tunnel while a lane is down or not yet up.
+# the data fallback stay on the base tunnel/port. A lane only carries data once
+# a probe on it has been answered, and is re-probed every 30s while it is up;
+# traffic falls back to the base tunnel while a lane is down or not yet up.
+# Lanes are only probed on tunnels with traffic, so an idle tunnel costs
+# nothing extra.
 #
 # Requirements: multiport.ports > 1, Linux, and the port range
 # [listen.port, listen.port+multiport.ports-1] reachable through firewalls on
@@ -221,10 +222,11 @@ listen:
   # each port costs a full set of `routines` threads and sockets. Capped at 256
   # (the lane header limit). 4-8 is plenty to escape a single ECMP path.
   #ports: 0
-  # How many lanes to run, counting the base tunnel as lane 0. 0 (default)
-  # means one per bound port. Lowering this sends on a subset of the range,
-  # which bounds how many extra tunnels each peer pair maintains (useful on a
-  # big server with many peers); the ports are bound and read either way.
+  # How many lanes to send on, counting the base tunnel as lane 0. 0 (default)
+  # means one per bound port. Lowering this sends on a subset of the range
+  # (useful on a big server with many peers, to bound how many lane probes and
+  # sessions each peer costs). It only limits sending: the ports are bound and
+  # read either way, and we still receive on every lane a peer sends on.
   #lanes: 0
 
 punchy:

handshake_manager.go

@@ -1148,7 +1148,8 @@ func (hm *HandshakeManager) sendHandshakeResponse(via ViaSender, msg []byte, hos
 		fields := append(logFields, "from", via)
 		// Reply from the socket the handshake arrived on so the initiator sees
 		// the source port it targeted. Identical to f.outside under vanilla
-		// config (all writers share one port); required for multiport lanes.
+		// config (all writers share one port); under multiport it keeps the reply on
+		// the base port, from whichever socket of its group the handshake reached.
 		err := f.writers[via.SockIdx].WriteTo(msg, via.UdpAddr)
 		if err != nil {
 			f.l.Error("Failed to send handshake message", append(fields, "error", err)...)

inside.go

@@ -156,14 +156,14 @@ func (f *Interface) sendInsideEncrypt(hostinfo *HostInfo, ci *ConnectionState, l
 //
 // When this flow has a usable multiport lane to this peer, the direct path swaps
 // to that lane's session and socket below. Relay and base traffic stays on
-// tx.base (socket 0).
+// tx.base, this routine's socket on the base port.
 func (f *Interface) sendInsideMessage(hostinfo *HostInfo, pkt tio.Packet, fwPacket *firewall.Packet, nb []byte, tx *txQueue) {
 	ci := hostinfo.ConnectionState
 	if ci.eKey == nil {
 		return
 	}
 
-	// Base and relay traffic stays on socket 0; the direct path may swap to tx.lane below.
+	// Base and relay traffic stays on the base port; the direct path may swap to tx.lane below.
 	sendBatch := tx.base
 
 	// One traffic-out mark covers every segment of the superpacket; doing it

interface.go

@@ -45,7 +45,7 @@ type InterfaceConfig struct {
 	routines           int
 	// Multiport means the sockets are spread over a range of ports
 	// (listen.port+slot) rather than all sharing listen.port, and that lane
-	// tunnels are negotiated with capable peers.
+	// sessions are negotiated with capable peers.
 	Multiport bool
 	// RoutinesPerPort is how many sockets share each port under multiport, and so
 	// the stride between port slots in writers: writers[s*RoutinesPerPort+r] is
@@ -294,9 +294,8 @@ func (f *Interface) activate() error {
 		"fips140Enforced", fips140.Enforced(),
 	)
 
-	// Under multiport each socket has exactly one reader on its own port, so
-	// the shared-port multi-reader capability is irrelevant (and main.go
-	// already hard-errored on unsupported platforms).
+	// Under multiport, main.go already probed this capability and turned
+	// multiport off where it is missing, so there is nothing to fall back from.
 	if f.routines > 1 && !f.multiport && !f.outside.SupportsMultipleReaders() {
 		f.routines = 1
 		f.l.Warn("multiple udp readers are not supported on this platform, falling back to a single routine")

lanes.go

@@ -41,12 +41,14 @@ import (
 // stays down until a probe on it is acked, and falls back to the base tunnel the
 // moment it stops being acked.
 //
-// Both directions are pay-per-use. A lane session is derived on the first packet
-// that needs it, because how many lanes exist is partly the peer's call: it
-// advertises how many it sends on, and we have to be able to receive all of
-// them. Deriving them all up front would let a peer advertising the maximum cost
-// us a replay window and two cipher states per lane, per tunnel, for lanes it
-// may never send on.
+// Receive is pay-per-use. A lane we only receive on gets its session on the
+// first packet that arrives on it, because how many lanes exist is partly the
+// peer's call: it advertises how many it sends on, and we have to be able to
+// receive all of them. Deriving them all up front would let a peer advertising
+// the maximum cost us a replay window and two cipher states per lane, per
+// tunnel, for lanes it may never send on. A lane we send on gets its session
+// from its first probe, and every such lane is probed on the tunnel's first
+// tick with traffic.
 
 const (
 	// laneKeyInfo is the HKDF label prefix for lane key expansion. Changing it
@@ -113,7 +115,7 @@ type laneSet struct {
 	// txLanes is how many lanes we may send on — our lane count clamped to the
 	// ports the peer bound — and so the modulus a flow's hash is reduced by.
 	// Lanes from txLanes up can only receive, which is how a peer with more
-	// routines than us still spreads its own traffic.
+	// ports than us still spreads its own traffic.
 	// Immutable, and the length of every TX-side slice here.
 	txLanes int
 
@@ -299,14 +301,15 @@ func (f *Interface) emitLaneStats(up, tunnels metrics.Gauge) {
 }
 
 // lanePortOffset returns the rotation applied to this pair's lane target ports,
-// in [0, peerPortCount). Without it every low-routine peer would aim its few
+// in [0, peerPortCount). Without it every peer with few ports would aim its few
 // lanes at a big peer's first few ports, concentrating the big peer's receive
 // work on a couple of sockets; the hash spreads pairs across the whole range.
 //
 // Both sides hash the same sorted vpn-address pair and the higher address
 // negates the result, so when port counts match the two sides' rotations
-// cancel: our lane s's 4-tuple stays the reverse of the peer's lane s, and each
-// side's probe opens the conntrack entry the other's arrives through. (The one
+// cancel on ports: our lane s's 4-tuple is the reverse of the peer's lane
+// s+offset, and each side's probe opens the conntrack entry the other's arrives
+// through. The lane indices still differ by offset; laneBias lines those up. (The one
 // lane a nonzero rotation lands on the peer's base port has no partner lane, so
 // behind a port-restricted NAT it is the one lane that may never come up.)
 func lanePortOffset(myAddr, peerAddr netip.Addr, peerPortCount uint16) uint16 {
@@ -638,10 +641,11 @@ func (f *Interface) probeLanes(hostinfo *HostInfo, now time.Time, nb, out []byte
 // for a probe sent before the reset match the first probe sent after it, and
 // promote the lane to a target nothing has proven.
 //
-// Demand is deliberately left standing: it records that a routine has real
-// traffic for this peer, which a roam or a relay detour does not change. Keeping
-// it re-probes the lanes that were actually carrying data as soon as a path
-// exists again, while a lane whose routine has gone quiet stays down.
+// Demand is deliberately left standing: a roam or a relay detour doesn't change
+// what the data plane wants. Lanes that were up had their demand spent when they
+// were probed, but the first packet of a flow that hashes onto one after the
+// reset raises it again, so lanes still carrying flows are re-probed on the next
+// tick, while a lane nothing hashes onto stays down.
 func (ls *laneSet) resetLocked() {
 	for s := 1; s < len(ls.probe); s++ {
 		ls.txAddr[s].Store(nil)

main.go

@@ -144,7 +144,7 @@ func Main(c *config.C, configTest bool, buildVersion string, l *slog.Logger, dev
 	// Multiport lanes: bind a range of consecutive UDP ports (listen.port+p)
 	// instead of SO_REUSEPORT-sharing one, and derive one extra session per lane
 	// with capable peers, so a tunnel's inside flows spread over several underlay
-	// 5-tuples instead of one. Defaults on, degrading gracefully when
+	// 5-tuples instead of one. Off unless multiport.ports > 1, degrading gracefully when
 	// preconditions aren't met — managed deployments (dnclient) can't be
 	// hard-errored on config they don't control.
 	//

udp/udp_linux_writebatch.go

@@ -22,10 +22,11 @@ import (
 // scratch WriteBatch packs mmsghdr entries into, plus the GSO capability
 // state probed at socket creation.
 //
-// One socket can have several senders: under multiport every routine writes its
-// base-session traffic to socket 0, and any routine may write to any lane
-// socket, since a packet's lane comes from its flow rather than from the routine
-// that read it. The scratch is per socket, not per sender, so mu serializes
+// One socket can have several senders: under multiport a routine writes lane s
+// traffic to the socket at its own group position on lane s's port, so every
+// socket is shared by the routines at that group position, one per port, since
+// a packet's lane comes from its flow rather than from the routine that read
+// it. The scratch is per socket, not per sender, so mu serializes
 // packing and draining. It is taken once per flush — a syscall's worth of work —
 // and only contends when two routines flush the same socket at the same instant.
 //