My x86 router journey

varkey

Well-Known Member
Messages
2,330
Location
Bangalore | Ernakulam
ISP
Excitel | BSNL
I had recently acquired a mini pc with 4 Intel NICs to be used as an x86 router / firewall. More details around the process of acquiring it is in another thread - Important - All international "Gift" packages now liable for customs duty @ 41.2% | Online Shopping

So I got the mSATA SSD module (and a keyboard :p) and I was trying to set up things over the last 1-2 days. At first I tried OPNsense, but to be honest, I didn't find it whole thing as refined or polished as pfSense. I couldn't even get my BSNL and Kerala Vision connections up for some reason. Maybe it is bit of unique or non-supported, I do not know, but the same config works on first try in OpenWRT without issues.

The Kerala Vision and BSNL connections come over the same fiber, goes into my Huawei ONT in transparent bridge mode (well, it doesn't have any other mode ;)) and then into the router, on the same port. Then VLANs are setup for each (1830 for BSNL Voice, 702 for BSNL Internet and 140 for Kerala Vision). On OPNsense, the the voice interface came up fine cause it was just a static IP config and didn't need mac spoofing. However for BSNL and Kerala Vision, as they do mac binding, I needed to spoof the mac.

But this doesn't appear to be supported natively OPNsense, and my PPPoE interfaces never even came up. Maybe once or twice, the BSNL interface came up but got disconnected in a short time. The logs show PPPoE timeout and retrying. 🤷‍♂️
I tried what I can and gave up.

Now on pfSense, the UI is more clean and polished. The layout also seemed better than OPNsense. I did the same config on pfSense and here they were pretty direct and mac spoofing on VLAN interfaces is not supported and must be set on the parent interface (this is the feature request Feature #1337: VLANs with different MAC address than parent interface - pfSense - pfSense bugtracker). But the issue is, I need two different macs when talking to Kerala Vision and BSNL (as that's what is originally registered on their end and I didn't want to get it reset).

I anyway I found a work-around in pfSense forum (MAC address spoofing on VLAN's and impressions from a second-try user) where you need to create bridge interface with just the VLAN interface as a member, and on the bridge interface you can spoof the mac. So BSNL Internet was setup on the regular VLAN interface and Kerala Vision on the bridge interface.

Finally, I was able to get my WAN interfaces up, now the next problem. The unbound DNS resolver kept on getting restarted. Log entries like below were being logged repeatedly. I searched online but couldn't find a solution that worked for me. Every 4 seconds or so the resolver was getting restarted which breaks everything as DNS fails.

Code:
Mar 22 13:51:25 mercury unbound: [54761:0] info: start of service (unbound 1.9.1).
Anyway I found the problem, it was the DHCP client for the BSNL IPv6 which was constantly getting a no addresses response. And for each such response, unbound gets restarted, I have no clue why 🤷‍♂️ Mostly related to this bug Bug #5413: Incorrect Handling of Unbound Resolver [service restarts, cache loss, DNS service interruption] - pfSense - pfSense bugtracker where unbound is restarted for each change rather than using unbound-control for such cases.

Code:
Mar 22 12:55:33 mercury dhcp6c[97095]: status code for NA-0: no addresses
Mar 22 12:55:41 mercury dhcp6c[97095]: status code for NA-0: no addresses
Mar 22 12:55:47 mercury dhcp6c[97095]: status code for NA-0: no addresses
Mar 22 12:55:54 mercury dhcp6c[97095]: status code for NA-0: no addresses
Mar 22 12:56:00 mercury dhcp6c[97095]: status code for NA-0: no addresses
Mar 22 12:56:06 mercury dhcp6c[97095]: status code for NA-0: no addresses
Mar 22 12:56:13 mercury dhcp6c[97095]: status code for NA-0: no addresses
Mar 22 12:56:19 mercury dhcp6c[97095]: status code for NA-0: no addresses
Mar 22 12:56:25 mercury dhcp6c[97095]: status code for NA-0: no addresses
Setting this to true fixed the problem, as it is probably BSNL not responding.

pfsense-ipv6-dhcp.PNG

So far things are working, I am still playing with how the fail over and load balancing should work. Overall, OpenWRT seems more mature for my use cases? I mean so many issues in getting something like this working which I could get it running in OpenWRT with no such challenges.

dashboard-pfsense-edited.png


@C3PO @vishalrao @vignesh_venkatesan @achaudhary997 @JB700
 

C3PO

Member
Messages
276
Location
Pune
ISP
You Broadband
@varkey pfSense takes a bit of time and commitment to resolve. It took me many days before I could get it just right. Afterwards it was a piece of cake. Just remember to always take a config backup (diagnostics -->Backup/Restore) before you make changes so you can always roll them back.

I have kept the DNS forwarder service off and resolver on.
 

varkey

Well-Known Member
Messages
2,330
Location
Bangalore | Ernakulam
ISP
Excitel | BSNL
Yep, definitely looks like it. I had to reset to defaults and start from scratch quite a few times. 😅😅

Thanks, good point regarding keeping backups. I'll probably setup a daily backup to a remote location I think I did see some options for that.

Yep, same here, forwarder is off, with unbound active as the resolver although forwarding is enabled there. I liked the fact that DNS over TLS is supported out of the box. Right now I've pointed it to Google DNS.
 


JB700

Active Member
Messages
606
Location
Delhi
ISP
Tripleplay
If you want to setup DNS Ad Blocking, AdGuard Home has a FreeBSD Version AdguardTeam/AdGuardHome , I ran it on a PFSense VM and it ran flawlessly. Imo Adguard Home is better for DNS Blocking than PFBlockerNG (better UI) but PFBlockerNG can also block IPs as well so..
 


varkey

Well-Known Member
Messages
2,330
Location
Bangalore | Ernakulam
ISP
Excitel | BSNL
It's sad that there is no support for wireguard on pfSense, even my tiny Mi Router 3G does great with wireguard


I'm wondering if I should just install OpenWRT on my mini pc. 🤪
 

C3PO

Member
Messages
276
Location
Pune
ISP
You Broadband
@varkey dont worry it will be included in pfSense in due course. WG is still in beta and pfSense is used in larger organisations that value stability over any new feature. Also, WG has to be implemented in the underlying OS i.e. freeBSD. See this thread: Installing WireGuard VPN and the last post from Netgate in particular.
 

varkey

Well-Known Member
Messages
2,330
Location
Bangalore | Ernakulam
ISP
Excitel | BSNL
Yep I had seen that thread, but now it is the mainline linux kernel as well. It's high time pfSense added it. OPNsense has wireguard support though.

Meanwhile IPv4 over a IPv6 wireguard tunnel on OpenWRT ;)



 

C3PO

Member
Messages
276
Location
Pune
ISP
You Broadband
@varkey I dont think we will see WG till freeBSD updates the kernel and it is relatively stable. From my years of experience with pfsense believe me I think they are a bit conservative, as they ought to be, for a router/firewall firmware that if less than stable will cause a lot of grief to users especially in the small office and enterprise category. If you want to try their betas there is an option go to Syesm -->Update and select betas or RCs from the drop down list.
 
Top