Leaving pfSense for VyOS (formerly Vyatta)

A few months back I wrote a bit about my unusual home network topology and, in particular, how I’d been planning to modernize it. Though it had worked pretty well for years already, the aim then was to improve it further by moving the firewall to newer, more power-efficient hardware and from pfSense to Vyatta, my favorite network operating system. Well, that’s essentially what happened, but with a slight detour.

In fact, I did migrate to a new Atom D525-based Supermicro X7SPA-HF + 4-port I350, and successfully ditched pfSense in favor of Vyatta 6.6R1. At least, for a short while. But after a couple of days, before I was even finished writing my new policies, I wound up abandoning Vyatta. You see, much like Fredo Corleone in The Godfather Part II, Vyatta broke my heart, and is now dead to me. As it turned out, the community builds of Vyatta hadn’t been updated at all since the company’s acquisition by Brocade. Thankfully, there was a light at the end of the tunnel.

VyOS

Introducing VyOS

VyOS is the new community fork of Vyatta, the open source routing and security platform based on Linux. Since there doesn’t seem to be any interest on the part of Brocade in maintaining Vyatta’s open source codebase or its community any longer, VyOS has stepped in to pick up the slack.

Some key points:

  • The project seems quite active and there have been several releases already.
  • New features have been added and longstanding bugs fixed since splitting from Vyatta.
  • Features which were previously paid-only in Vyatta have been added back into VyOS including the webUI. [Edit: apparently this is just a stub for possible future use.]
  • There seems to be some direct collaboration from the EdgeOS folks at Ubiquiti, and possibly even some of the Vyatta folks who stayed at Brocade. Therefore VyOS may end up becoming the natural upstream for both projects, though only time will tell.
  • Vyatta started out with Quagga for advanced routing, then moved to a proprietary solution (inspiring much controversy in the process). VyOS has reinstated Quagga.
  • As of this writing, VyOS is backwards compatible with configs from Vyatta… though that may not be the case forever. Thus Vyatta users currently have a clean upgrade path to VyOS.

 The Migration Process

In the future it’s possible that the addition of new features in VyOS will break backwards compatibility. But currently, upgrading from Vyatta to VyOS is dead simple. First you add the VyOS maintainers GPG key:

curl http://vyos.net/so3group_maintainers.key | sudo apt-key add -

Then you can pull down the new system image. For example, if you wanted to add VyOS 1.0.2:

add system image http://mirror.tuxhelp.org/vyos/iso/release/1.0.2/vyos-1.0.2-amd64.iso

Afterwards, you can reboot to your new VyOS system. Easy peasy.

Feature Parity with pfSense

Since I was originally coming to Vyatta VyOS from the FreeBSD-based pfSense, it was important that some of the more advanced features I was using continued to function as expected. I’m happy to report that VyOS covered all the bases nicely.

VyOS Screenshot

I’m using WAN load balancing for all outbound connections across two distinct and asymmetric Internet providers. I never had any problems with WLB in pfSense, and it works just as well in VyOS. With this facility, I was easily able to achieve the following:

  • Force wired network traffic out through the faster of the two Internet connections.
  • Conversely, set the wireless network to prefer the ADSL link.
  • If one of the two should fail, all traffic swings to the healthy link automatically.

There are of course other must-have features like traffic shaping, which keep latency tight even when the WAN pipe in question is totally saturated. And a decent QoS policy makes sure latency-sensitive apps get the bandwidth they need when they need it.

Performance

The new VyOS rig performs much better than my old pfSense box. Granted, this due in no small part to newer, faster hardware.  But based on the performance bottlenecks inherent in the PF firewall stack, I’d still expect VyOS to outperform pfSense quite handily on the same hardware.

With six gigabit ethernet interfaces, I am able to achieve a total aggregate throughput of 12Gb/s — even with a thorough, zone-based firewall policy in place between each interface. Latency is exceptional too, as traversal of the firewall adds only ~0.08ms on average, even on a low-power Atom CPU that’s now several generations old.

VyOS (and Linux’s firewall connection tracking facility in general) is also much more efficient in terms of memory consumption. While PF consumes about 1k of non-swappable kernel RAM for every state in the table, Netfilter requires less than a third of that at a meager ~300 bytes. This means VyOS can handle roughly 3x the simultaneous firewall states in the same memory footprint as pfSense.

Issues

While I really didn’t have any problems per se with my transition, there were a few minor things in VyOS I felt could be better handled. For instance, I’ll acknowledge that the combination of WAN load-balancing and DHCP on the same interface might be an unusual scenario. But unfortunately, that’s exactly what I have to work with. The WLB function in VyOS lacks the capability of monitoring the dynamic gateway for health checks as a variable. Thus the monitor target for the mandatory gateway health check must be updated manually if it changes when the lease is renewed.

In my case both carriers give leases on the order of six months to a year, so the impact of this shortcoming is negligible. Still, I’ve filed a bug (well, more of a feature request really), so we’ll see what comes of it.

Closing Thoughts

While pfSense did an admirable job of protecting and segmenting my network for several years, the migration to VyOS definitely felt like an upgrade. For the cost of an entry-level server (that’s less than the price of an ASA 5505 of yesteryear, which could only firewall 70mb/s at peak), I have wire-speed filtering across six gigabit ethernet interfaces. And being a bit of a network guy anyway, I actually prefer the straightforward CLI interface to the web interface of pfSense — though in fairness, I do find the pfSense web UI more appealing than many of its proprietary competitors.

If you’re interested in a similar solution but you’d prefer a nicely bundled turnkey product (and the accompanying commercial support), I’d strongly recommend looking into Ubituiti’s EdgeMax line of routers. The EdgeMax series runs EdgeOS, which is Ubiquiti’s fork of Vyatta. While it lacks some of the nice features which made it into Vyatta after the EdgeOS fork, such as global state policies, it does have a pleasant looking web UI which makes creating basic policies simple for novice users.

At the low end Ubiquiti offers the SOHO style wall-mount EdgeRouter Lite for under $100 USD, which can filter at wire speed across its three gigabit ethernet ports and handle an equally impressive 1 million packets per second. Or, if you’d rather go rackmount, for around $500 USD their high-end EdgeRouter Pro unit can pass several million PPS, filter 8Gb/s, and sports two gigabit RJ45/SFP combo ports should you require fiber.

Want to try VyOS for yourself? You can get the images here.

VyOS.net

37 thoughts on “Leaving pfSense for VyOS (formerly Vyatta)

    • @Joe Kohler: No, though it’s possible. EdgeRouter is not an x86 platform, it’s MIPS-based, you’d have to cross-compile. There’s also some Octeon optimizations happening on the little EdgeRouter to help its PPS and crypto throughput, resulting in at least one proprietary module:

      ubnt_platform: module license 'Proprietary' taints kernel.

      I’m not sure exactly how its images are loaded either, though I don’t think there’s anything that would lock out 3rd party images. Could be an interesting project.

  1. Current EdgeOS (VC6.3 derivative, v1.4.1 now iirc) on Ubiquiti EdgeRouter has some changes in wan-load-balancing configuration. I can’t recall why or whether it has more features.

    • WAN load balancing seems roughly equivalent to the current Vyatta/VyOS builds from what I’ve seen, albeit slightly less flexible, although it does seem better equipped to deal with health checks on dynamic gateways.

      • This is actually false. When you don’t configure your targets it will use the DHCP provided GW as testing-target. It’s an hidden feature in the codez.

    • You just need to create the node in the config:
      set service https
      You can also specify whether or not it redirects 80 -> 443, as well as which address(es) it will listen on. That said, I never use the WebUI, as I much prefer the shell. Even the fancy Ubiquiti web GUI is pretty limited in its usefulness.

      • This is what you get if you go to the web address of your vyos router:

        This is a VyOS router.

        There is no GUI currently. There may be in the future, or maybe not.

        • Thanks, I’ll make an addendum to the article. I’d poked around with the Vyatta GUI back in the day, but it’s just a tree-based version of exactly what you already have in the CLI. Didn’t really see the point. Even working with UBNT routers, which do have a pleasant looking web UI, I’ve never actually used it to configure anything.

  2. hello i have problem with my vyatta system i cant run update on it,
    W: Failed to fetch Release.gpg Could not connect to packages.vyatta.com:80 (144.49.164.21). – connect (110: Connection timed out)

        • Yes, exactly. Vyatta is now a commercial-only product by Brocade, intended for cloud usage only. VyOS is an open source fork of Vyatta which can even import your old Vyatta configuration. There are instructions for migrating from Vyatta to VyOS right in this very article, near the top. It’s a bit outdated though; you’ll want the latest version of VyOS:

          Add the VyOS maintainers GPG key so you can trust the integrity of the new image:

          curl http://vyos.net/so3group_maintainers.key | sudo apt-key add -

          Then you can pull down the new system image. For example, if you wanted to add VyOS 1.0.4 [AMD64]:

          add system image http://mirror.tuxhelp.org/vyos/iso/release/1.0.4/vyos-1.0.4-amd64.iso

          Afterwards, you can reboot to your new VyOS system. That’s it. You can see which images are available here.

          • Meh, everybody says or does something stupid from time to time. Hopefully when that person happens to be me, someone will extend me the same courtesy.

  3. VyOS < 1.05 is susceptible to the 'shellshock' vulnerability via its DHCP client, so make sure you run the latest available version if you choose to run VyOS with any dynamic interfaces.

  4. Pingback: Vyatta | Pearltrees

  5. “With six gigabit ethernet interfaces, I am able to achieve a total aggregate throughput of 12Gb/s ”

    Is this a typo or is there some voodoo or magic that I am not aware of?

    • It’s voodoo called “full duplex”. 😉 Each interface can do 1Gb/s in each direction. 2Gb/s x 6 interfaces for a total of 12Gbs.

      • However total through put depending on CPU able to handle interrupts/traffic, multi-cores can share load, but lot always depending how the affinity is configured for interfaces. not exact but CPU speed times cores would be theoretical max throughput could process packets ? so in that case a four core 3gHz CPU could push 12Gbs but reality with over head would not reach that?

  6. I’ve worked on the WLB instance in VyOS last year and I think the bug you mentioned isn’t present actually. When you set up your nexthop to DHCP the WLB deamon will internally use the DHCP gateway, however it was a little to difficult to easily wire this variable back into the show wlb command. You can see this behaviour in line 45 in https://github.com/vyos/vyatta-wanloadbalance/blob/lithium/src/lbtest_icmp.cc actually. It’s one of those easter eggs. I tested this myself and validated the findings.

    • I’ve noticed that too, though I’d completely forgotten about that old bug. The biggest issue today is that if I don’t set static "interface-route" entries for both interfaces with "next-hop-interface" specified, bad things can still happen. Namely, in case of one of two interfaces failing, the heartbeat packets could potentially leave out of the healthy gateway and prevent the WLB from automatically failing over the dead or flapping interface. I guess that means it’s time to close out the old bug and file a new one; thanks for pointing that out. 🙂

  7. That’s not really a bug but sort of how routing works. In pfsense tradition it’s normal to setup routes for your health targets. Since target lies in non-connected subnet range, routing applies, but yes … it’s on my todo list somewhere. It wouldn’t be so hard I think to implement it. Maybe first update the health-target help text and fix the show wlb command accordingly.

    • The problem is that, at least with DHCP+WLB, the L2 link is not preferred over a route. Route weighting and PBR shouldn’t strictly be necessary for a network you’re directly connected to, since it should always have a higher default weight than an L3 net.

  8. did/could you post a scrubbed version of your config file anywhere? i’d love to see the technique you are using to push wired and wireless traffic out separate connections.

  9. I just migrated the other way around, the vyos backend drives me crazy and the web ui of pfsense is just so nice for the daily simple stuff, and i can always drop down to CLI whenever needed.

    • One of the main reasons I migrated from pfSense is that the configuration seemed very brittle. This was further reinforced recently, as I had to do an upgrade on a pfSense router for a non-profit which I’d originally setup for them years ago. The last two upgrades completely broke their configuration, so I wound up having to rebuild their (rather complex) configurations both times. At this point, I’ve lost confidence in the ability for pfSense to seamlessly upgrade. I’m kicking myself for not having gone with Vyatta [VyOS] early on, since updates and restoring configurations on replacement hardware has always been painless.

  10. wios@DGYQ-WIAC1000# run show dhcp server leases

    IP address Hardware address Lease expiration Pool Client Name
    ———- —————- —————- —- ———–
    172.18.1.137 00:34:cb:90:c1:a8 2015/08/13 11:55:43 e5_v204
    172.18.1.134 00:34:cb:90:c5:9d 2015/08/13 11:55:45 e5_v204
    wios@DGYQ-WIAC1000# date
    Wed Aug 12 23:05:33 CST 2015

    hi , Time does not match! why?

    • Make sure you’re not blocking NTP into the router, that you have working NTP servers configured, and that your timezone is correct.

      # show system time-zone
      time-zone America/Los_Angeles

      # show system ntp
      server 0.north-america.pool.ntp.org {
      }
      server 0.us.pool.ntp.org {
      }
      server 1.north-america.pool.ntp.org {
      }
      server 1.us.pool.ntp.org {
      }
      server 2.north-america.pool.ntp.org {
      }
      server 2.us.pool.ntp.org {
      }
      server 3.north-america.pool.ntp.org {
      }
      server 3.us.pool.ntp.org {
      }

  11. Hi, using VyOs as had to work on a project linking a Sophos UTM Firewall to a Vyatta Brocade (IPSec) , which led me to spend time on the VyOs as a result afterwards…
    Excellent results… So when Sophos bought Astaro, they closed off upgrades on the hardware… there are some excellent second hand bargains on the hardware look at AG220 , 320 appliances via online auction sites, they are x86 run VyOs spot on and later ones are 8 port gigabit units

Leave a Reply

Your email address will not be published. Required fields are marked *