I’m a career operations guy with some background in security and network engineering. This means my home network is something some people might call “over-engineered” (or even “completely overkill” if they were feeling particularly ungenerous). That said, I often work from my home office, so my network is more important to me than many home networks might be. I’ve also never had a single service outage or security compromise. If any of this sounds interesting, please read on.
Since it has as much to do with my livelihood as personal use, my home networking project had some key requirements:
- The end result should be secure.
- It should be able to survive local power outages and carrier failures reasonably well.
- Throughput is important, but not so much as latency… Ever tried using SSH over a satellite link? Not pretty.
- The cost to operate should be relatively low. This is a home office after all, not a colocation.
In order to keep costs low and landfills somewhat less full, for my firewall I used cast-off server hardware a previous employer was going to e-waste. The specs:
- 2x 600MHz P3 server
- 384MB RDRAM (RIMM 2400)
- 6x PCI Intel Pro/100 NICs
Geriatric by today’s standards, but quite enough to filter packets and perform simple routing. For context, consider that a typical home internet gateway appliance will usually have an embedded RISC CPU running at a few hundred MHz and be equipped with only 8-64MB RAM. On the business side, the popular Cisco ASA 5505 shipped with only a single AMD Geode @500MHz and 256MB of 400MHz DDR.
To handle sporadic power hiccups, I’ve got a Tripp Lite OMNIVS1500 UPS backing the network equipment. It does correct some line problems which would otherwise be invisible, and of course protects against sags and spikes which could cause an outage or damage equipment.
Internet connectivity options in my area are ironically quite limited, especially considering there are half a dozen datacenters only miles away from my home. The best I could do within my budget was a 50/15Mb primary service through Comcast Cable, with AT&T 12/1Mb U-verse ADSL to function as my secondary link.
The idea here is that the primary carrier will be the workhorse link, preferred for most types of traffic. If that link should fail for any reason, the secondary link is diverse in that it’s a different technology from a different carrier. The chances that they will both fail simultaneously are relatively slim.
The shape of the network is driven by my particular situation with my available resources in mind. I’m a fan of segmentation, which is really just an extension of the principle of least privilege as applied to networking. Thus the firewall ends up doing some internal routing and security segmentation duties which would normally be delegated to core routing infrastructure in a large organization. But since this is just a humble home network, my firewall will not be a substantial bottleneck for any traffic which will need to traverse it.
To illustrate, here’s a simple topology map:
I’m a big fan of open source software, especially GNU/Linux. That saves on licensing costs, makes dealing with malware a non-issue, and affords me a larger degree of flexibility over traditional, proprietary solutions.
My servers are currently running Debian 6.x “Squeeze” (old-stable), while my desktops run Debian 7.x “Wheezy” (stable). I like the balance Debian strikes between freshness and stability, meaning it doesn’t feel stale (like Red Hat), but it doesn’t feel flaky (e.g. Ubuntu, Fedora) and has a very reasonable life cycle.
There’s also a couple of inexpensive, low-power Intel Atom-based set tops, one in the living room and one in the master bedroom, which run XBMC as a user interface. This allows for smooth and seamless playback of HD content over the network. The biggest advantage over something more conventional, such as AppleTV, is that I am not limited to playing a few “blessed” formats, but can handle even the most obscure media types. It’s also running a full-blown operating system, which means they can be used for surfing using standard browsers, playing emulators, running Steam in big picture mode, etc.
Finally, the firewall runs pfSense, the FreeBSD-based routing & security platform which forked from m0n0wall some years back. Unlike m0n0wall, pfSense utilizes OpenBSD’s PF rather than IPFW, supports multiple WAN interfaces in various configurations, advanced traffic shaping, OSI layer-7 filters, and more.
The open-source pfSense firewall is performing many tasks on my network, from the mundane to the fairly advanced:
- NTP, DHCP, DNS forwarding & caching, etc.
- Zone-based, stateful firewalling
- Policy-based routing across multiple WAN links
- WAN link health checking and failover
- In-depth traffic shaping and QoS (hierarchical fair-service curve)
- Per-connection traffic limiters to stay under the high latency thresholds for each link
- Detailed traffic and health graphing (RRD)
Currently, I’ve utilized policy-based routing so that wireless clients will prefer the ADSL link, while hard-wired clients route through the cable link. If one of the two WAN links should fail (high latency or packet loss), all traffic will automatically swing to the healthy link. Because it is especially sensitive to latency and bandwidth, as long as the cable link is up, VOIP calls via wireless are routed through Comcast.
pfSense is free and open-source, generally reliable, has lots of advanced and useful features, optional professional support, and requires little in the way of resources. High-availability is available via pfsync / CARP, so if zero downtime is required, multiple pfSense firewalls may be clustered easily. As for administration, its web-based user interface is responsive and intuitive for both novice users and seasoned pros. For most use cases pfSense is more than adequate.
Occasionally minor services, NTP for example, inexplicably crash or refuse to start. Historical data is often missing or damaged, leaving gaping holes in your RRD graphs.
Sometimes CPU load climbs to unusually high levels with little rhyme or reason. In my case, the latter has historically been associated with the “check_reload_status” service.
Those experienced with various firewall platforms may miss having a powerful CLI, as pfSense lacks one. While there is a developer console available via root shell on the system which allows you to interact with the firewall, this is definitely not designed nor suitable for normal usage. If you need to add a lot of rules, prepare to do a lot of clicking in the web UI.
Since PF is single-threaded, pfSense does not scale linearly on multi-core architectures. If PF has maxed out a single core, you’ve essentially hit the performance ceiling of pfSense. While the FreeBSD system beneath pfSense can utilize multiple cores, they will only be used for secondary services. For this reason, there’s little advantage to running pfSense on a CPU with more than two cores. Additionally FreeBSD’s gmirror facility for software RAIDs sometimes eats itself for no good reason, leaving you with a degraded array.
Probably the most annoying issue I’ve run into with pfSense is trying to migrate configurations between different hardware. Getting it working out of the box requires an exhaustive find and replace exercise beforehand, swapping out the MAC and driver entries for every interface of the old hardware to that of the new. Otherwise, the old interfaces (and all associated firewall rules, policies, etc) are buried in the config, and new interfaces relating to the new hardware are added instead.
On the software front, I’d like to migrate from pfSense to Vyatta in the near future. pfSense is a decent platform, but its limitations have left a bad taste in my mouth over the years. I’ve been working heavily with Vyatta on the professional side, which is an open-source, Linux-based routing and security platform that was recently acquired by Brocade. It scales much, much better than pfSense, has superior hardware support, boasts many advanced features, is rock-solid, and has a really nice CLI (fairly similar to Junos). While the latter might sound unappealing for those new to firewalls, it makes writing advanced rulesets quickly a lot more convenient. And once you’re used to how a firewall functions and why, the interface just doesn’t matter that much… as long as it doesn’t get in the way.
In terms of hardware, it’s time to think about retiring that old P3 server which is currently functioning as my firewall. Something like a small, inexpensive, and power-efficient Supermicro SuperServer 5017A-EF coupled with an Intel i350 would make a fantastic firewall (or, even better, the 5018A-FTN4 when they come down in price). Plus, an Atom-based server would be a bit easier on my electricity bill. Add Vyatta and it’d be able to handle millions of simultaneous connections without breaking a sweat.
Questions? Comments? Want to see more detail on a particular topic I’ve mentioned here? Let me know in the comments.