Networking Troubleshooting — Working Outward From the Instance

Work outward, and stop at the first layer that fails

“The instance cannot reach the internet” is five different problems wearing the same sentence. The only reliable approach is to test each layer in order and stop at the first one that fails, because every layer above a broken one will also fail and tell you nothing new.

  1. Does the port exist and is it ACTIVE?          Neutron record
  2. Does the instance have an address?              DHCP
  3. Can it reach its own gateway?                   local bridge + L2
  4. Can it reach outside the tenant network?        L3 agent / router
  5. Can something reach it from outside?            floating IP + security groups

Layer 1 was the previous lesson. This one is 2 through 5.

The discipline that matters: do not start at layer 5. Almost everyone does, because “cannot ping the floating IP” is how the problem is reported, and a failure there is consistent with everything below it being broken too.

Test from inside the network, not from your laptop

The single most useful technique in Neutron troubleshooting is testing from a network namespace on the network node, which puts you inside the tenant network rather than outside it.

$ ip netns listqrouter-9f2e1c4a-… (id: 2)qdhcp-3a7b5d81-… (id: 1)# Ping the instance from inside its own network$ ip netns exec qdhcp-3a7b5d81-… ping -c2 192.168.10.14

This distinguishes two very different situations that look identical from outside:

• Reachable from the namespace, not from outside — the instance and its local networking are fine. The problem is routing, floating IP, or security groups.
• Not reachable from the namespace either — the problem is below routing: the port, the local bridge, or the instance itself.

That single test eliminates roughly half the search space and takes seconds.

Security groups fail in a distinctive way

Security groups are stateful and default-deny inbound. Two behaviours follow, and both are diagnostic:

Outbound works, inbound does not. Because the rules are stateful, replies to connections the instance initiated are allowed automatically. An instance that can reach the internet but cannot be reached is the classic security group signature — and it is almost never a routing problem, because routing is not directional in that way.

Ping fails while SSH works, or the reverse. ICMP and TCP are separate rules. A group with TCP 22 and no ICMP produces an instance that is reachable and unpingable, which reads as “half broken” and is entirely expected.

openstack security group rule list $(openstack server show <id> \
  -f value -c security_groups)

Routing and floating IPs

A floating IP is a NAT rule inside the router namespace. Three things must all be true, and each fails independently:

1. The floating IP is associated with a port. An allocated-but-unassociated floating IP is a very common cause of “the address does not work” — it belongs to the project and points at nothing.
2. The router has an external gateway, and the instance’s subnet is attached to that router.
3. The L3 agent hosting the router is alive.

$ openstack floating ip list –port <port-id>$ openstack router show <router> -c external_gateway_info$ openstack network agent list –agent-type l3# And from inside the router itself$ ip netns exec qrouter-<id> iptables -t nat -L -n | grep 203.0.113.42

The last command is the one that settles arguments. If the NAT rule is present in the router namespace, Neutron has done its job and the problem is upstream of OpenStack — a physical network, a firewall, or routing outside the cloud.

MTU: the failure that looks like nothing

Overlay networks add encapsulation overhead. When the instance MTU is not reduced to account for it, small packets succeed and large ones are silently dropped.

The symptom is deeply misleading: ping works, SSH connects and then hangs at the banner, HTTP returns headers and stalls on the body. Everything looks almost fine, which sends people searching for an application problem.

# From the instance: find the largest packet that survives the path
ping -M do -s 1472 <gateway>

If small pings succeed and this fails, it is MTU, and no amount of security group or routing investigation will help.

When to reach for a packet capture

A capture is the right tool only after the layer tests have narrowed the question to a specific segment. Placed correctly it settles the question in seconds; placed hopefully it produces a large file and no insight.

The useful placements, in order of how often they answer:

• The tap interface on the compute host — does the instance’s traffic leave at all?
• Inside the router namespace — does traffic arrive at the router, and does it leave translated?
• The physical uplink — does encapsulated traffic reach the wire?

# On the compute host, the instance's tap device
tcpdump -ni tap<port-id-prefix> icmp

If traffic is present at the tap and absent at the router, the problem is between them and you have halved the remaining space again.

The method, in order

1. Port ACTIVE and bound? If not, stop — this is the previous lesson’s problem.
2. Does the instance have its address? No address means DHCP, not routing.
3. Reachable from the qdhcp- namespace? This splits local from routed problems.
4. Outbound works, inbound does not? Security groups, almost always.
5. Floating IP associated, router has a gateway, L3 agent alive?
6. NAT rule present in the router namespace? If yes, the problem is outside OpenStack.
7. Works small, fails large? MTU.
8. Still unresolved? Capture at the tap, then at the router.

Stopping at the first failure is the whole method. Each step is cheap; running them out of order is what makes network incidents long.

Exercise

Take a working instance with a floating IP and break it three separate times, restoring between each:

1. Remove the inbound rule from its security group
2. Disassociate the floating IP
3. Stop the L3 agent

For each, run the method from step 1 and record which step first identified the fault. All three present as “cannot reach the instance”; the point of the exercise is to feel how differently they behave once you are testing in layers rather than from the outside.