At the end of Part 1 I had an ASN, a /24, and absolutely no way to use either one, which made me happy and sad at the same time.

A prefix is only useful in the global Internet when some other network accepts it from you and passes it on, but until that happens you just own a number. Comcast over cable was never going to be that provider, so I needed to find someone else who would run BGP with a guy whose infra lives next to the laundry sink and sump pump.

Core solution requirements

Before going down into the solutioning rabbit-hole, I wrote down what “done” should look like:

Frugal

A main motive for doing this is to pay the same or less as I do with Comcast’s static block. Any solutions MUST be as frugal as possible. 2x DIA circuits would be nice, but come on…

Resilient

This means multihoming from N+1 upstream providers, redundant routers at home over multiple WANs. Everything in the architecture MUST have redundancy.

Stateless

Let routers route, a failure upstream shouldn’t kill my flows. Stateful firewalling and NAT belong elsewhere down the stack.

Automated

The whole setup MUST be as automated and scripted as possible. Ephemeral from day 1, generating configurations via script, provisioning routers via Ansible. If I want a new router, I just add specs to a YAML and get a new config file in minutes. If I want to add BFD (more on that later) I just update the generator for that and shoot out the configs to the entire stack.

This list basically drove every decision that follows.

So what are my options?

There’s providers called Tunnel Brokers that will announce your prefix and give you a tunnel (GRE, WireGuard, IPsec…). They were a bit pricey but it’d be an easy setup, just not as flexible as running my own.

There’s also the option of a Colo and renting out a 1U somewhere. Also works, but now I own hardware in a building, it’s a single upstream and doesn’t fully solve last-mile delivery to my basement. Maybe this is a potential future initiative, but not now.

Then finally there’s providers hosting VPSs who let you bring your own IP space and run BGP with them. bgp.services has a large list of such providers, so I just needed to find the two closest to me so the latency hit would be minimal.

So I settled with the VPS route as ideal route. It’s kind of wild that it exists, that providers will spin you up a normal virtual machine and then establish an eBGP session with it, accept your prefix, and announce it to the world. I went with Vultr and HostHatch, with Vultr having a datacenter in the town next door, and HostHatch at Equinix Chicago. Two providers, two ASNs, two completely separate transit networks, both about the price of a streaming subscription. Also, bonus, Vultr can even send you the FULL internet routing table!

BGP on a VPS

Things they will ask you for before they’ll accept a single route:

  • An LOA (Letter of Authorization) saying you’re allowed to announce the space. It’s basically you authorizing yourself, which feels ridiculous the first time.
  • A ROA so RPKI validators say your origin is legit. Created in ARIN Online, free and basically a requirement in 2026 for proper route security.
  • An IRR route object. ARIN’s IRR handles this, and their auto-manager will keep it in sync with your ROA.

Then you wait ~24 hours for upstream filters to refresh, because your provider’s provider is also filtering. Once peered and announcing my prefix, I saw some propagations within an hour, but it’s basically safe to say you’ll be waiting overnight.

Oh, one thing that cost me an Saturday evening: both providers happen to establish the session against the SAME link-local-ish address but with slightly different peer ASNs. When setting up HostHatch, I copied the working config from Vultr, changed what looked like it needed changing, and got a session stuck in Active with nothing in the log to explain it. It was the peer ASN. It is always something like the peer ASN (like it’s always DNS eh?).

Let’s think about the architecture

Going back to the requirements, our design needs resiliency throughout. We’ll also need a way to connect to the VPSs that are announcing the prefix and make it “seem” like we’re there, so tunneling is really the best (maybe only?) option.

So the basic architecture is:

  • Two border routers (BRs) as VPSs, one in Vultr, one in HostHatch, each holding an eBGP session with its upstream.
  • Two edge routers (ERs) at home connected to the Border Routers via tunnels to form a routing fabric.
  • Anything that wants to use an address from the prefix must peer with at least one, but ideally both ERs.

By definition, Border peers with public networks, Edge peers with internal networks.

We end up with a two-tier WAN with a full mesh overlay.

flowchart TB
    subgraph P1[Vultr]
        BR1[br1]
    end
    subgraph P2[HostHatch]
        BR2[br2]
    end
 
    subgraph HOME[Home]
        ER1[er1]
        ER2[er2]
    end
      
    BR1 -.- ER1
    BR1 -.- ER2
    BR2 -.- ER1
    BR2 -.- ER2

Now let’s think about the routers

I needed something that:

  • runs on a generic KVM VPS
  • speaks BGP, OSPF and WireGuard natively
  • is the same OS at home and in the cloud
  • is frugal

For years I’ve been using MikroTik devices and also their software-only versions called Cloud Hosted Router (CHR). They make fantastic fully-featured routers, are pretty cheap and can be sideloaded into a regular Linux KVM VM.

A P1 license runs about $45 one time for 1Gbps of throughput per interface, which is plenty for now. The P10 gets you 10Gbps per interface, but it’s $95. The free tier is capped at 1Mbps, which is useful for building the whole thing out and testing before you pay anything.

Is RouterOS the most polished BGP implementation on the planet? Nope!. Its v7 routing stack has opinions, some of the documentation is iffy at best, and they can be a bit buggy at the most fundamental levels between releases. But… since I’m very comfortable with them, it does everything on that list, it runs identically in both places, and I can rebuild a router by pasting a text file at it, it’s good enough.

You want to roll with VyOS or custom build some Linux stuff? Sure, why not, it’s all possible.

Getting the traffic home

Tunnels!

GRE was my first instinct, since it’s the simplest thing that could work and MikroTik does it well. Killed immediately: my 5G backup link is CGNAT’d, and GRE cannot traverse NAT. Even ignoring that, it’s plaintext across the public Internet, carrying every packet destined for my house. Encryption is basically free these days with hardware accelerated AES.

IPsec works and is well understood, and I’ve deployed plenty of it professionally. It’s also more moving parts than I want in the one component whose failure takes everything down. Phase 1’s, Phase 2’s… IKEv2… too complex for a simplified overlay.

WireGuard ended up on top, and not just because it’s the latest and greatest…

  • It’s UDP, so my 5G CGNAT is fine as long as the inside end initiates and keeps the conversation alive. My edge routers always dial out, the border routers never dial in, and a keepalive holds the pinhole open.
  • It’s encrypted by default with no negotiation surface to get wrong. P1/P2 mismatches not a thing…
  • The config is small enough to generate, which matters a lot later when there are dozens of keys.

Now, two things I learned the hard way and will spend real time on in Part 3:

MTU is kind of important. A tunnel over a 5G link that hands you less than 1500 bytes, and you get an evening of banging your head against the desk. Pings work, SSH mostly works, and then specific sites hang forever (but mysteriously not Youtube?). Every tunnel in the fabric has to use the same MTU, otherwise a flow that fails over from the main WAN to 5G mid-stream just quietly dies.

WireGuard interfaces never go down. A WireGuard interface is a virtual thing that exists whether or not the far end is alive. Kill the border router and your local interface stays cheerfully running. So I couldn’t use interface state for failover, which means I needed something else. Currently, A routing protocol is the only liveness detector in the entire network. More on that next, and it’ll explain most of the design choices in the next few posts.

Routing protocols

  • eBGP to each upstream using my public ASN as the local AS. This is the only place the real outside world exists for my prefixes.
  • OSPF + BFD between all my routers, over the tunnels. It carries loopbacks and tunnel links, detects dead tunnels via hellos, and expresses which home WAN I prefer as a simple cost metric. THIS rides over WireGuard as the failure detection protocol.
  • iBGP between all my routers, over their loopbacks. This carries my actual prefix around.
  • eBGP with Private ASNs from the edges into the internal network, which came from another need for redundancy and getting away from using VRRP on the ERs.

Sounds complex, but it’s rather simple, and all with a purpose. WAN preference lives in OSPF cost, so BGP never knows or cares which physical link a tunnel rides. Promoting the 5G link to primary if needed is a simple change, and could add a third WAN link easily, even ECMP over two WANs.

Let routers route

Once I started advertising the prefix for the first time, and after propagation, I noticed how much background “noise” there was. Scanners, legitimate traffic for the previous owner of the block, script kiddies, all kinds of stuff hitting all IPs in the range.

These connection attempts started filling up my conntrack state table pretty quickly, so I went back to the core requirement of the architecture being stateless

The border routers protect themselves and forward everything else statelessly, so connection tracking is on only for traffic addressed to the router itself, and that’s very scoped.

All the stateful work, NAT, the actual security policy, lives on a pair of pfSense firewalls, one hop behind the edge routers, so my prefixes are routed, cleanly, to the firewalls, and they are the boundary.

Routers route, firewalls firewall, and when something breaks I know which box to log into (hopefully).

Where I F’d up (spoiler)

I’ll spoil one thing before you think it, made me go back and rebuild the architecture once I started testing failure modes.

In the first version my home routers originated the /24. The Edge layer would send the prefix, and BRs would just redistribute, because why wouldn’t they? I’m trying to emulate direct BGP peers at home, the space is mine, my house is the network, so the announcement should come from the source of truth. I guess it made sense?

Turns out it was flawed thinking, becasue with an extended power outage or if I unplug the wrong interfaces, my prefix gets withdrawn from the global routing table from both upstreams. It’s not a huge deal because global reconvergence happens quickly (about 2-5 minutes), but it’s loud, it’s rude to everyone else’s routers, and it’s completely preventable while still keeping ownership of the advertisement.

Moving origination out to the border routers fixes that, and also creates a whole new set of problems that need solving, which is why it will its own post.

Target architecture

Before we roll into Part 3, the target architecture so far looks like this:

flowchart TD
    subgraph PROV["VPS + eBGP Providers"]
        subgraph P1[Vultr]
            P1BGP([eBGP Peer])
            BR1[br1<br/>MikroTik CHR]
        end
        subgraph P2[HostHatch]
            P2BGP([eBGP Peer])
            BR2[br2<br/>MikroTik CHR]
        end
    end

    subgraph HOME[Home]
        ER1[er1<br/>MikroTik CHR]
        ER2[er2<br/>MikroTik CHR]
        FW@{ shape: div-rect, label: "pfSense CARP" }
        HOMENET@{ shape: procs, label: "Home Networks" }
    end
    P1BGP ---|eBGP| BR1
    P2BGP ---|eBGP| BR2
    
    BR1 -.Wireguard<br/>OSPF + BFD<br/>iBGP.- ER1
    BR1 -.Wireguard<br/>OSPF + BFD<br/>iBGP.- ER2
    BR2 -.Wireguard<br/>OSPF + BFD<br/>iBGP.- ER1
    BR2 -.Wireguard<br/>OSPF + BFD<br/>iBGP.- ER2

    ER1 ---|eBGP| FW
    ER2 ---|eBGP| FW
    FW === HOMENET

    style PROV fill:none,stroke:dotted

Next up, Part 3: the build