A Technical Pro’s Home Network

I’m a career operations guy with some background in security and network engineering. This means my home network is something some people might call “over-engineered” (or even “completely overkill” if they were feeling particularly ungenerous). That said, I often work from my home office, so my network is more important to me than many home networks might be. I’ve also never had a single service outage or security compromise. If any of this sounds interesting, please read on.

Project Requirements

Since it has as much to do with my livelihood as personal use, my home networking project had some key requirements:

  1. The end result should be secure.
  2. It should be able to survive local power outages and carrier failures reasonably well.
  3. Throughput is important, but not so much as latency… Ever tried using SSH over a satellite link? Not pretty.
  4. The cost to operate should be relatively low. This is a home office after all, not a colocation.

Hardware

In order to keep costs low and landfills somewhat less full, for my firewall I used cast-off server hardware a previous employer was going to e-waste. The specs:

  • 2x 600MHz P3 server
  • 384MB RDRAM (RIMM 2400)
  • 6x PCI Intel Pro/100 NICs

Geriatric by today’s standards, but quite enough to filter packets and perform simple routing. For context, consider that a typical home internet gateway appliance will usually have an embedded RISC CPU running at a few hundred MHz and be equipped with only 8-64MB RAM. On the business side, the popular Cisco ASA 5505 shipped with only a single AMD Geode @500MHz and 256MB of 400MHz DDR.

To handle sporadic power hiccups, I’ve got a Tripp Lite OMNIVS1500 UPS backing the network equipment. It does correct some line problems which would otherwise be invisible, and of course protects against sags and spikes which could cause an outage or damage equipment.

 Connectivity

Internet connectivity options in my area are ironically quite limited, especially considering there are half a dozen datacenters only miles away from my home. The best I could do within my budget was a 50/15Mb primary service through Comcast Cable, with AT&T 12/1Mb U-verse ADSL to function as my secondary link.

The idea here is that the primary carrier will be the workhorse link, preferred for most types of traffic. If that link should fail for any reason, the secondary link is diverse in that it’s a different technology from a different carrier. The chances that they will both fail simultaneously are relatively slim.

Topology

The shape of the network is driven by my particular situation with my available resources in mind. I’m a fan of segmentation, which is really just an extension of the principle of least privilege as applied to networking. Thus the firewall ends up doing some internal routing and security segmentation duties which would normally be delegated to core routing infrastructure in a large organization. But since this is just a humble home network, my firewall will not be a substantial bottleneck for any traffic which will need to traverse it.

To illustrate, here’s a simple topology map:

My Network Topology

 Software

I’m a big fan of open source software, especially GNU/Linux. That saves on licensing costs, makes dealing with malware a non-issue, and affords me a larger degree of flexibility over traditional, proprietary solutions.

My servers are currently running Debian 6.x “Squeeze” (old-stable), while my desktops run Debian 7.x “Wheezy” (stable). I like the balance Debian strikes between freshness and stability, meaning it doesn’t feel stale (like Red Hat), but it doesn’t feel flaky (e.g. Ubuntu, Fedora) and has a very reasonable life cycle.

There’s also a couple of inexpensive, low-power Intel Atom-based set tops, one in the living room and one in the master bedroom, which run XBMC as a user interface. This allows for smooth and seamless playback of HD content over the network. The biggest advantage over something more conventional, such as AppleTV, is that I am not limited to playing a few “blessed” formats, but can handle even the most obscure media types. It’s also running a full-blown operating system, which means they can be used for surfing using standard browsers, playing emulators, running Steam in big picture mode, etc.

Finally, the firewall runs pfSense, the FreeBSD-based routing & security platform which forked from m0n0wall some years back. Unlike m0n0wall, pfSense utilizes OpenBSD’s PF rather than IPFW, supports multiple WAN interfaces in various configurations, advanced traffic shaping, OSI layer-7 filters, and more.

pfSense

pfSense 2.1 Dashboard

The open-source pfSense firewall is performing many tasks on my network, from the mundane to the fairly advanced:

  • NAT/PAT
  • NTP, DHCP, DNS forwarding & caching, etc.
  • Zone-based, stateful firewalling
  • Policy-based routing across multiple WAN links
  • WAN link health checking and failover
  • In-depth traffic shaping and QoS (hierarchical fair-service curve)
  • Per-connection traffic limiters to stay under the high latency thresholds for each link
  • Detailed traffic and health graphing (RRD)

Currently, I’ve utilized policy-based routing so that wireless clients will prefer the ADSL link, while hard-wired clients route through the cable link. If one of the two WAN links should fail (high latency or packet loss), all traffic will automatically swing to the healthy link. Because it is especially sensitive to latency and bandwidth, as long as the cable link is up, VOIP calls via wireless are routed through Comcast.

The Good

pfSense is free and open-source, generally reliable, has lots of advanced and useful features, optional professional support, and requires little in the way of resources. High-availability is available via pfsync / CARP, so if zero downtime is required, multiple pfSense firewalls may be clustered easily. As for administration, its web-based user interface is responsive and intuitive for both novice users and seasoned pros. For most use cases pfSense is more than adequate.

The Bad

Occasionally minor services, NTP for example, inexplicably crash or refuse to start. Historical data is often missing or damaged, leaving gaping holes in your RRD graphs.

pfSense Quality Graph

Sometimes CPU load climbs to unusually high levels with little rhyme or reason. In my case, the latter has historically been associated with the “check_reload_status” service.

pfSense CPU spikes

Those experienced with various firewall platforms may miss having a powerful CLI, as pfSense lacks one. While there is a developer console available via root shell on the system which allows you to interact with the firewall, this is definitely not designed nor suitable for normal usage. If you need to add a lot of rules, prepare to do a lot of clicking in the web UI.

The Ugly

Since PF is single-threaded, pfSense does not scale linearly on multi-core architectures. If PF has maxed out a single core, you’ve essentially hit the performance ceiling of pfSense. While the FreeBSD system beneath pfSense can utilize multiple cores, they will only be used for secondary services. For this reason, there’s little advantage to running pfSense on a CPU with more than two cores. Additionally FreeBSD’s gmirror facility for software RAIDs sometimes eats itself for no good reason, leaving you with a degraded array.

Probably the most annoying issue I’ve run into with pfSense is trying to migrate configurations between different hardware. Getting it working out of the box requires an exhaustive find and replace exercise beforehand, swapping out the MAC and driver entries for every interface of the old hardware to that of the new. Otherwise, the old interfaces (and all associated firewall rules, policies, etc) are buried in the config, and new interfaces relating to the new hardware are added instead.

Future Plans

On the software front, I’d like to migrate from pfSense to Vyatta in the near future. pfSense is a decent platform, but its limitations have left a bad taste in my mouth over the years. I’ve been working heavily with Vyatta on the professional side, which is an open-source, Linux-based routing and security platform that was recently acquired by Brocade. It scales much, much better than pfSense, has superior hardware support, boasts many advanced features, is rock-solid, and has a really nice CLI (fairly similar to Junos). While the latter might sound unappealing for those new to firewalls, it makes writing advanced rulesets quickly a lot more convenient. And once you’re used to how a firewall functions and why, the interface just doesn’t matter that much… as long as it doesn’t get in the way.

In terms of hardware, it’s time to think about retiring that old P3 server which is currently functioning as my firewall. Something like a small, inexpensive, and power-efficient Supermicro SuperServer 5017A-EF coupled with an Intel i350 would make a fantastic firewall (or, even better, the 5018A-FTN4 when they come down in price). Plus, an Atom-based server would be a bit easier on my electricity bill. Add Vyatta and it’d be able to handle millions of simultaneous connections without breaking a sweat.

If you want to try Vyatta for yourself, ISOs can be found here. [Edit] Obsolete. See my later post: Leaving pfSense for VyOS (formerly Vyatta) 

Feedback

Questions? Comments? Want to see more detail on a particular topic I’ve mentioned here? Let me know in the comments.

10 thoughts on “A Technical Pro’s Home Network

  1. Pingback: Leaving pfSense for VyOS (formerly Vyatta) | DotBalm.org

  2. This was a great write up I am also a fan of pfsense. I use it as a border firewall and it performs inter vlan routing. However, I plan to segment out my home network more so I will be moving all lan routing duties to vyos.

  3. This is a great lecture, right now am implementing a similar network with pfsense virtualized. only that i would not have redundancy in wan because of the market in my country $50 for a 5mbps 1/3.

    Greatings from El salvador pal.

  4. hoping you can help me …. I have a cable modem/wireless router from Comcast. I work from home & need to restrict bandwidth for my kids (home on summer vacation) – in order for me to get full speeds for work. Netflix & Xbox are taking priority over my work traffic. Any topology you could recommend?

    • Topologically, I’d downgrade the home gateway CPE to act as a bridge, and place a “real” firewall behind it, having that act as your gateway. That could be something like an UBNT EdgeRouter, small server hardware running VyOS or pfSense, or even some little Asus gateway running Tomato, DD-WRT, etc. If you run a firewall without wireless capabilities, you’d want your WAP acting as a bridge behind the firewall, like so:

      Internet → modem (bridged) → firewall → WAP → [clients]

      Once the topology is in place, the important thing is traffic shaping / QoS. You want to be able to guarantee bandwidth and low latency to the services that relate to your work. How you do this depends heavily on which FW solution you go with, but even just shaping the WAN connection based on your actual UP/DOWN numbers will make a big difference, even with the default shaping behavior on most routers. It will give your router the capability to properly budget traffic evenly to all clients, thus reducing dropped packets and max latency substantially.

  5. I enjoyed reading your article. I am in a similar situation as you. I work from home and have been a Network Administrator / Engineer since around +/- 1990. I have been using Vyatta and then VyOS since around Vyatta 6.0 and have really liked it. Lately I’m working on moving to an ALL IPv6 Network except for the edge where I have a NAT64 router running Debian and my VyOS with all or mostly IPv6 interfaces. I’ve been using Hurricane Electric IPv6 tunnel in “production” since 2010 and it has worked perfectly and given me IPv6 access to the Internet. Otherwise I would be going crazy wanting to learn / try IPv6 without access to the IPv6 Internet. I have a business account with my ISP and multiple IPv4 addresses as well as an IPv6 /48 tunnel from HE. My ISP has *FINALLY* turned on IPv6 for residential accounts this last 6 months and I’m anticipating them turning on IPv6 for their business customers sometime in the next 6 months. Once that happens I will finish migrating to all IPv6 internally and use NAT64 translation on the edge. The big problem I’ve been having with VyOS lately is that it doesn’t do NAT64 which is why I set up the Debian NAT64 using TAYGA. VyOS also doesn’t have the latest version of OpenVPN, but I can work around that by translating IPv6 -> IPv4 through the VPN tunnel and then back again on the other side. I was contemplating changing over to PfSense but I think I will stick with VyOS as VyOS has ZONE-BASED firewalls and PfSense doesn’t. I will probably head over to the PfSense forum and post a feature request for NAT64 and ZONE-BASED firewalls.

  6. Me too – very nicely pitched description
    One question I wonder to ask:
    – you use pfsense
    and I use zeroshell
    do you have any observations comparing zeroshell to pfsense ?
    I am mostly satisfied with how it works,
    but I am always on the lookout for order of magnitude improvements
    in my preferences and toolkits.
    thanks for your time

    • I haven’t used pfSense in years I’m afraid, so I can’t be too much help there. For my part I’ve been primarily using VyOS or EdgeOS (Ubiquiti’s embedded respin of Vyatta; runs on EdgeRouter products). Aside from the latest stable version of VyOS being a bit old (though new versions of VyOS should be released in the near future, from what I’m told), I’ve been quite happy with both platforms.

Leave a Reply

Your email address will not be published. Required fields are marked *