At the end of Part 3 I had a working fabric, with the v4 prefix in the global table, failover tested, packets arriving over both ISPs.

And one lingering design flaw that I knew about and had been ignoring: my house was originating 100% of the announcements.

What actually happens when the house goes dark

Let’s follow the chain: The ERs originate the /24 into iBGP, the BRs learn it from them and pass it to their upstreams. The announcement exists because home says it exists.

Cut the power at home and here’s the sequence:

  1. Tunnels stop passing traffic
  2. BFD notices in under a second, OSPF drops the adjacencies
  3. The BRs lose their iBGP sessions to the ERs
  4. The /24 is gone from both BRs (because they’re coming in via iBGP)
  5. Both BRs send a withdrawal to their upstreams
  6. That withdrawal propagates to every DFZ router on the planet in minutes

If you’re wondering what “DFZ” means, it’s the Default-Free Zone, or in other words, the routers on the internet where there is NO default route, because they have specific routes to everyone else. This is also known as the Internet Routing Table or Global Routing Table. THIS is what I was really after all along.

Now, when my prefix no longer exists on the DFZ, traffic is effectively blackholed. Then power comes back and all of that runs again in reverse. Every router that carries a full table does work for something that was only intended to be transitive.

For a single outage, who cares. Global reconvergence is a couple of minutes and nobody notices. The problem is the flappy case, which is the normal case with residential-grade connections over DOCSIS and cell. A 5G link having a bad afternoon, a Comcast modem doing its third death rattle of the year, me unplugging the wrong thing while cable managing at 11am RIGHT BEFORE BED like an idiot, like I always tend to do. Each one of those is an announce/withdraw cycle in the DFZ.

Do that enough times and you run into route flap damping, where upstreams start penalizing a prefix that won’t sit still and suppress it for a while. Damping is less common than it was in the 2000s and the parameters got a lot saner because we have more router oomph, but it absolutely still exists out there, and I’d rather not find out. Which means the failure mode is: my house comes back up, my announcement is fine, and some networks still can’t reach me for the next hour because I spent the afternoon flapping like a hummingbird.

So what’s the fix? The announcement needs to stop depending on the least reliable part of the system, let’s go and fix that eh?

Originate at the border instead

The fix is conceptually trivial: each BR originates the /24 on its own, unconditionally, forever. It doesn’t ask home for permission. It doesn’t care if the tunnels are up. As long as the BR is running and its eBGP session is up, the prefix is announced.

The world now sees a stable /24 with my ASN as origin, from two upstreams, permanently. What’s happening behind that announcement is my problem, not the DFZ’s.

The anchor route

Here’s where RouterOS gets specific about it.

In v7 you can originate a prefix by pointing the BGP connection at an address list:

/ip/firewall/address-list/add list=bgp-originate address=203.0.113.0/24
/routing/bgp/connection/set [find name=vultr] output.network=bgp-originate

But there’s a sneaky condition: RouterOS will only advertise the network if there is an active route for it in the routing table. No route, no advertisement, which is sane behavior. It stops you from announcing space you have no idea what to do with. We’re ignoring default-originate here on purpose, we’re not sending a default to our peer, that’s just not how this works…

Normally that route came from home over iBGP, but that’s exactly the dependency we’re trying to kill. So instead, each BR gets its own static route for the whole /24, going nowhere:

/ip/route/add dst-address=203.0.113.0/24 type=blackhole distance=250 \
    comment="anchor - keeps the aggregate alive for origination"

That’s the anchor route. Its entire job is to exist locally, like a prefix gravity pull. It gives the router something to point at so the network statement fires, and it never goes away because it isn’t attached to any interface or protocol.

I chose the name because it’s basically what it does: the announcement is anchored to the border router instead of drifting around depending on what’s happening miles away in my basement.

The distance number is the whole trick

A static route in RouterOS defaults to distance 1. iBGP-learned routes land at 200. So if you add that anchor at the default distance, it wins against every real path you have with the same or less-specific prefix. Traffic arrives at the BR, matches a blackhole with distance 1, and gets dropped, while a perfectly good tunnel to home sits there unused.

You’d have built a router that announces your prefix to the world and then deletes everything sent to it. Very stable! Completely useless.

However, at distance 250 the anchor loses to literally everything: OSPF at 110, iBGP at 200, static routes you add later. It only becomes active when there is genuinely nothing else, and “nothing else” is exactly the state where dropping is the correct answer anyway.

flowchart TB
    A["Packet arrives at BR<br/>destined for my /24"] --> B{"More specific route<br/>from home via iBGP?"}
    B -->|Yes| C["Down the tunnel<br/>business as usual"]
    B -->|No| D{"Path to the other BR?"}
    D -->|Yes| E["Over to the other border<br/>it has a way home"]
    D -->|No| F["Anchor route, distance 250<br/>drop it"]

Anchor-guard, or: keep it on the box

The anchor is a blackhole for my own address space. That is a genuinely dangerous object to have lying around, and it must never leave the border router it lives on.. So there are two non-negotiable guardrails:

Don’t redistribute static into anything. Not OSPF, not iBGP. The anchor is not a route anybody else needs to know about, it’s just meant as a gravity pull for the prefix in the DFZ. If you need static redistribution for something else later, filter this prefix out explicitly rather than relying on the fact that you happened not to enable it.

Filter the eBGP session outbound to exactly the prefix you’re announcing. The only thing my upstreams should ever hear from me is the /24, at length 24 (and eventually v6 blocks, not there yet). This is basically how upstream providers also keep announcements in check, so it’s best practice to do it anyway:

/routing/filter/rule/add chain=ebgp-out \
    rule="if (dst==203.0.113.0/24 && dst-len==24) { accept }; reject;"

Default-reject, one explicit permit. This is the filter that stops you from being the guy who leaked a full table to his transit provider at 2am (even thought my upstreams just won’t take my announcements, but still). It also lines up with the ROA from Part 2, where max-length is 24 and nothing more specific should ever validate.

So what does home advertise now?

If the BRs own the aggregate, the ERs stop originating it entirely. What they advertise into iBGP is more-specifics: the subnets actually in use at home, the ones with real hosts behind them.

The BRs end up with a routing table that looks like:

  • more-specific subnets, learned via iBGP, pointing down the tunnels
  • the /24 anchor at distance 250, doing nothing

And there’s a side effect I didn’t plan for, and now really like. Remember the scan noise from Part 2, the constant background hum of people probing every address in a freshly allocated block? A big chunk of that is aimed at addresses I’m not using.

With the anchor in place, all that shit dies at the border instead of clobbering up my home connections. It never gets encrypted, never goes down a tunnel, never touches my 5G backup if we’re there. Traffic to space I actually use gets a more-specific and comes home. Everything else hits the anchor and stops.

I built it for announcement stability and got a free border scrubber!

The tunnel I didn’t think I needed

Now a partial failure that the old design handled by accident and the new one doesn’t.

Say Vultr has a bad day and BR1 loses its tunnels home, but my house is completely fine and BR2 is happily passing traffic. Old design: BR1 loses the /24, withdraws from Vultr and all traffic shifts to HostHatch 100%.

But with the new design: BR1 keeps advertising (because that’s the entire point of the anchor), traffic arrives at BR1, finds no more-specifics, hits the anchor and blackholes. Meanwhile BR2 could have delivered every one of those packets.

The symptom of this is very hard to pin down initially, because some sites work and some don’t. It’s almost like an MTU issue except it’s at the border. Pings “reach” out, but they may or may not come back to me depending on the AS_PATH.

A fix is a tunnel I didn’t have in the Part 3 topology: BR1 talks directly to BR2. Ninth WireGuard interface, provider to provider, no home involvement. It carries OSPF and iBGP like every other link, with a cost high enough that it’s never preferred on a normal day. This is basically an emergency-only backhaul.

Now BR1 losing home isn’t blackholing traffic anymore, because it still has an iBGP session with BR2, still learns the more-specifics, and forwards across to the other border. The path is stupid and inefficient but it beats dropping the packets!

flowchart TB
    subgraph P1[Vultr]
        BR1[br1]
    end
    subgraph P2[HostHatch]
        BR2[br2]
    end
    subgraph HOME[Home]
        ER1[er1]
        ER2[er2]
    end

    BR1 -.high cost, backup only.- BR2
    BR1 -.- ER1
    BR1 -.- ER2
    BR2 -.- ER1
    BR2 -.- ER2

Only when the whole house is unreachable do both BRs fall through to the anchor and start dropping. Which is correct, because at that point there is genuinely nowhere to deliver anything to. The prefix is still announced, I’m not flapping routes, but packets die at the border.

BGP is no longer my smoke alarm

Under the old design a broken fabric was extremely visible. With the prefix gone from the global table, I’d have my external lambda checks losing their shit and pinging me via Pushover every 5 minutes (which would in turn make me lose my shit)

But now? The prefix is always up. It will be up while every tunnel is down and regardless of my house getting its third lightning hit (that’s a different story). From the outside my network looks perfect right up until you try to send it a packet.

  • iBGP session state on each BR, specifically whether it has any more-specifics from home at all
  • OSPF neighbor count per router, since “3 of 4 adjacencies” is a warning and I want to know before it’s 0 of 4
  • WireGuard last-handshake age per tunnel, which is useless for failover but great for telling me a specific path has been dead for six hours
  • Whether the anchor route is the active route for the /24 on a BR, which is a perfect one-bit alarm. Anchor active means that border can’t deliver anything. It should NEVER be active, and if it is then something is wrong.

Testing it, which is the fun part

The test is anticlimactic in the best way. Kill everything at home, then go look at the DFZ from outside.

Nothing happens. Prefix still there, same origin, same two upstreams, no churn. Packets sent to it get dropped at the border instead of delivered, but no other network on Earth has to do any work about it, and there’s nothing to damp because nothing changed. Things seem “normal” by design now.

Bring the edge routers back and traffic just resumes, because the /24 announcement never blipped. The “fix” is past the BRs, so as fast as OSPF and iBGP reconverge, we’re good.

That’s the actual goal here, not that nothing breaks, because stuff breaks constantly, but when it breaks, it stays MY problem instead of becoming a tiny bit of everyone’s problem.

What’s still ugly

Two things I’ve now mentioned twice and keep deferring, so let’s actually do them.

I’ve got nine tunnels, four routers, a ton of WireGuard keypairs, and a set of OSPF costs and BGP filters that all have to agree with each other or the whole thing quietly misbehaves. For a while I had been maintaining this by hand. Adding the BR-to-BR tunnel meant touching two routers and getting four things exactly right, and I got one of them wrong on the first pass which made the whole thing go poof

That does not scale past about here, and it violates the Automated requirement I put in Part 2 and have been cheerfully ignoring for four posts. I ultimately wanted this thing to be treated as a Terraform or CloudFormation deployment, make a change, push it out, and wait. Ideally we’d get the config “template” as part of a CI/CD pipeline, and then we’d not only have a reproducible environment but an easy way to automatically roll back if things get ugly.

Next up, Part 5: The intro to my topology.yml, generating every router’s config from one file, and then pushing them out with Ansible, so we’ll build a custom Rube Goldberg’s thing that reads a desired state file and achieves it, NetDevOps-style. Things break? roll back quickly and easily since it’s all stateless anyway. Imagine that!