A 4am debugging story: when the packet went the wrong way

Some outages are complicated. This one was simple enough to be embarrassing, and it's the ones like this that teach you how to debug properly.

The symptom

One service — and only one — became unreachable from outside the network. Everything else on the same host was fine. curl from the host itself worked. From anywhere else, nothing. No timeout, no reset, just silence.

The wrong assumptions

My first instinct was the firewall. I'd touched it earlier in the week. I checked the rules — and they were right. The port was open, the zone made sense, the source restriction was sane. I spent twenty minutes re-reading the same rule set and finding nothing.

Then I checked the service. It was listening. On the right interface. Binding to 0.0.0.0, even.

The actual cause

It was a routing table subtlety. The packet from outside arrived, the host accepted it, and the reply was being sent out the wrong interface. The routing table had two almost-identical default routes with different metrics, and the answer took the loop that led to nowhere.

# the thing that finally told the truth
ip route get 8.8.8.8

One ip route get showed the reply path clearly. For every other service the asymmetric route didn't matter; this one protocol — the one with the firewall that checks source addresses and drops anything unexpected — noticed instantly.

The lesson I keep re-learning

When local access works and remote access fails with silence, stop editing configs and start tracing the packet's actual path. The symptom was a firewall behavior; the cause was a route. Debugging is remembering that your assumptions are the bug, most of the time.