I run my Apple builds on a small fleet of Mac runners wired into GitLab CI. Most of the pipeline is unremarkable — build, test, archive, upload — and it had been humming along for months. Then our App Store packaging job started failing intermittently, and the failure made no sense: the build succeeded, the archive succeeded, the .ipa was exported, the dSYMs were zipped... and then the job died uploading the artifacts, with a terse:

ERROR: Uploading artifacts as "archive" to coordinator... error
  ... write tcp 100.85.x.x->...:443: write: broken pipe
WARNING: Retrying...
(×3)
FATAL: invalid argument
ERROR: Job failed: exit status 1

What follows is the whole diagnostic journey, wrong turns included, because the wrong turns are the interesting part. The short version: a "network" problem that wasn't a network problem, hiding behind a "slow upload" that wasn't slow, ultimately caused by a macOS kernel bug that detonates after exactly 49.7 days of uptime.

"It's the network, surely"

The first instinct with a broken pipe on a big upload is that something is too slow or too flaky. Our artifact is, admittedly, very large — the packaging job uploads the entire output/export/ directory: the .ipa, the dSYMs, and the build logs, about 935 MB all told. So the working theory was: large upload + some timeout = death.

Rather than theorize, I pulled the data. For every recent run of the packaging job we (Claude and I) extracted which host it ran on, how long the artifact upload took, and whether it passed:

Two things jumped out. First, every failure died at almost exactly 60 seconds. Second, one host — europa — failed every single time, while others almost never did. A 60-second wall plus a host that always trips it sure smells like a timeout that europa is a hair too slow to beat.

Runner or server?

Before chasing "slow," I wanted to know who was enforcing the 60 seconds — the GitLab runner or the server. The error semantics answer this cleanly:

• write: broken pipe is EPIPE — the remote closed the socket while the runner was still writing the body. A runner-side timeout looks different (context deadline exceeded, Client.Timeout).
• The runner retries three times — it wants to keep going, so it isn't the one giving up.
• Each of the three attempts independently got ~61 seconds before being cut, and each carried a fresh GitLab correlation_id — so the requests reached the server stack before being severed.
• The edge is Server: nginx (GitLab 19.0.2-ee), and ~60 s is exactly nginx's default client_body_timeout / proxy_read_timeout.

So: a server-side timeout, not the runner. Good to know — but it only told us who pulled the trigger, not why the upload sat there long enough to get shot.

Slow, or stalled?

Here's where the "it's just slow" theory fell apart. Do the arithmetic: 935 MB is about 7.85 gigabits. On a nominally 1 Gbps link that should take ~8 seconds, and the bandwidth-delay product at 8 ms RTT is only ~1 MB, so a single TCP stream can trivially fill the pipe. Yet our successful uploads were taking ~50 seconds — roughly 8× slower than line rate — and the failures wedged at 60.

That's not "a bit slow." Something was stalling. And there's a tidy logical proof: nginx's 60s timeouts are inter-read timeouts — they fire on a 60-second gap, not on total duration. A steadily-slow-but- progressing stream never trips them (indeed one upload ran 79.8s and succeeded). For a connection to die at the 60s wall, it has to go silent for 60 seconds. It was stalling, not crawling.

A packet capture of a stalled upload confirmed the shape of it — the server endlessly re-acknowledging the same byte while the connection made zero forward progress, and then, 60 seconds later, a reset:

... dagobah.https > europa: Flags [.], ack 2114283, ... sack 1 {2115707:2144187}
... (a flood of duplicate ACKs, all stuck at ack 2114283, in ~1 ms)
(60 seconds of silence)
... dagobah.https > europa: Flags [R], seq 1082354262

One segment was missing (the gap from 2114283 to the SACK left edge 2115707 is exactly one 1424-byte segment), the server SACK'd everything after it, and then... nothing. For a full minute. Then RST.

A red herring named MTU

The hosts reach the server over a Tailscale/WireGuard path (those 100.x CGNAT addresses), and the tunnel interface has an MTU of 1280. Aha — a classic WireGuard MTU black hole: full-size segments silently dropped, PMTU discovery defeated by filtered ICMP, bulk transfers wedging while pings sail through. It fit the "one segment keeps vanishing" story beautifully.

It was wrong, and the host operator (rightly) pushed back. If europa's egress interface is MTU 1280, its TCP stack natively clamps its own segments to fit — it can't emit anything larger, full stop. A path-MTU sweep proved the path was clean:

% ping -D -s 1252 192.0.2.10      # 1280-byte packet, DF set
1260 bytes from 192.0.2.10: ... 0.0% packet loss
% ping -D -s 1253 192.0.2.10
ping: sendto: Message too long     # local interface refuses >1280

1280-byte packets reach the server with zero loss; anything bigger is rejected locally by the interface, not black-holed downstream. PMTU discovery is moot when the interface already constrains you. The MTU theory was dead, and we were back to: why does one segment vanish and never come back?

The smoking gun

So we captured the inner TCP on the tunnel interface during three failed attempts — headers only, big buffer, clean (zero kernel drops) — and pulled it apart with tshark. The picture was unambiguous, and it was not what any of the network theories predicted:

• europa transmitted the missing segment exactly once (seq 1949707, 1228 bytes), contiguously in its stream. No gap on its side.
• The server received everything around it and SACK'd the one-segment hole [1949707, 1950935), flooding ~30 duplicate ACKs that named the missing bytes unmistakably. SACK was negotiated on both SYNs.
• europa then sent nothing at all for 60 seconds. Zero retransmissions in the entire capture. Not a fast-retransmit (30 duplicate ACKs is ten times the threshold), not even a single RTO retransmission.
• The server gave up and RST at +60.2 s. All three attempts identical.

This was the moment the whole thing turned over. A single dropped packet is normal — TCP expects to retransmit it in milliseconds. The fault wasn't the drop. The fault was that europa's TCP never retransmitted it. Neither loss-detection mechanism fired: not the SACK/dup-ACK fast path, not the retransmission timeout. The sender just... stopped.

One more breadcrumb, which we'd glossed over earlier: netstat on europa reported 23,076 open TCP sockets. That is an absurd number for a mostly-idle build box, and it would turn out to be the same bug waving a second flag.

A reader's tip, and the penny drops

At this point some more searching on the internet located Photon's "We Found a Ticking Time Bomb in macOS TCP Networking" — and everything snapped into place.

The bug: macOS tracks a millisecond clock for the TCP stack in a 32-bit counter, tcp_now. After 2³² ms — 49 days, 17 hours, 2 minutes, 47 seconds — it overflows, and a botched monotonicity check in calculate_tcp_clock() freezes it permanently at its pre-overflow value. The headline symptom in that writeup is that TIME_WAIT connections stop expiring (because the garbage collector runs off the frozen clock), sockets pile up, and eventually new connections fail.

But a frozen TCP clock breaks every timestamp-driven mechanism in the stack, and that's exactly our failure:

• RTO never fires. The retransmission timeout expires when the clock advances past a deadline. If tcp_now never advances, the timer never expires, and the sender never retransmits. That's our 60 seconds of silence.
• RACK never fires either. macOS's modern loss detection is time-since-sent based. With a frozen clock, a segment's age computes as nonsense, so RACK never marks the lost segment for retransmission — even with 30 SACK dup-ACKs screaming about it.
• The 23,076 sockets are the writeup's other symptom, directly: TIME_WAIT entries that never expire because tcp_gc() reads the same dead clock.

So europa isn't dropping more packets than its peers, and its network is fine. The single lost segment is ordinary background loss that every host hits now and then. The difference is that a healthy host retransmits it in a millisecond and you never notice — while europa, with a frozen clock, cannot retransmit at all, so every routine packet drop is fatal. That's why it was 0-for-7 while clean hosts passed, and why the failures landed mid-upload rather than at connection setup: the clock was frozen (timers dead) but it hadn't quite exhausted ephemeral ports yet.

The confirmation took ten seconds: europa had been up 98 days — nearly two full overflow periods past the 49.7-day fuse — running macOS 26.3.1.

The fix (such as it is)

There is no Apple patch as of this writing; the only remedy is a reboot, which resets tcp_now and buys you another ~49 days. So:

1. Reboot the affected hosts. Force-rebooting the fleet cleared it immediately.
2. Monitor uptime and reboot before ~49 days. A scheduled reboot at, say, 40 days, or an alert on kern.boottime, eliminates the entire class of failure until Apple ships a fix. Mac minis and Mac Pros doing CI/CD are precisely the machines that stay up long enough to hit this; your laptop reboots for updates and never sees it.
3. Stop shipping a 935 MB artifact anyway. This was a clearly-unnecessary, near-gigabyte upload eout. Trimming it to the few hundred bytes a downstream job actually needs makes the pipeline robust regardless of TCP clocks. Defense in depth, now that we know the depth.

Lessons for the next mystery

A few things this taught (or re-taught) me:

• Sanity-check against physics early. "935 MB should take 8 seconds, not 50" reframed the whole investigation from "tune the timeout" to "something is stalling." I should have done that arithmetic on day one.
• Separate proven from inferred, out loud. Half of the theories (sluggish uploads, broken cable, MTU black hole) were plausible and wrong. The packet capture was the arbiter every time, and the cleanest progress came from asking "what would the packets look like if this were true?" and then going to look.
• The capture doesn't lie, but you have to read all of it. "Zero retransmissions" was the single most important fact in the whole affair, and it was sitting in a one-line tshark summary. The 23,076 sockets were in plain sight even earlier, if I'd looked then.
• Sometimes the network problem is an OS bug, and sometimes the OS bug is a calendar problem. A 32-bit millisecond counter is a ~49.7-day fuse, and a Mac that's "rock solid, hasn't been touched in months" is the worst-case input.

If you run long-lived Macs for anything — CI, monitoring, a home server — go check your uptime. If it's north of 49 days and your networking has gotten weird, you don't have a network problem. You have a clock that stopped.