I had recently acquired a mini pc with 4 intel NICs to be used as an x86 router / firewall. More details around the process of acquiring it is in another thread - Important - All international "Gift" packages now liable for customs duty @ 41.2% | Online Shopping
So I got the mSATA SSD module (and a keyboard 😛) and I was trying to set up things over the last 1-2 days. At first I tried OPNsense, but to be honest, I didn't find it whole thing as refined or polished as pfSense. I couldn't even get my BSNL and Kerala Vision connections up for some reason. Maybe it is bit of unique or non-supported, I do not know, but the same config works on first try in OpenWRT without issues.
The Kerala Vision and BSNL connections come over the same fiber, goes into my Huawei ONT in transparent bridge mode (well, it doesn't have any other mode 😉) and then into the router, on the same port. Then VLANs are setup for each (1830 for BSNL Voice, 702 for BSNL Internet and 140 for Kerala Vision). On OPNsense, the the voice interface came up fine cause it was just a static IP config and didn't need mac spoofing. However for BSNL and Kerala Vision, as they do mac binding, I needed to spoof the mac.
But this doesn't appear to be supported natively OPNsense, and my PPPoE interfaces never even came up. Maybe once or twice, the BSNL interface came up but got disconnected in a short time. The logs show PPPoE timeout and retrying. 🤷♂️
I tried what I can and gave up.
Now on pfSense, the UI is more clean and polished. The layout also seemed better than OPNsense. I did the same config on pfSense and here they were pretty direct and mac spoofing on VLAN interfaces is not supported and must be set on the parent interface (this is the feature request Feature #1337: VLANs with different MAC address than parent interface - pfSense - pfSense bugtracker). But the issue is, I need two different macs when talking to Kerala Vision and BSNL (as that's what is originally registered on their end and I didn't want to get it reset).
I anyway I found a work-around in pfSense forum (MAC address spoofing on VLAN's and impressions from a second-try user) where you need to create bridge interface with just the VLAN interface as a member, and on the bridge interface you can spoof the mac. So BSNL Internet was setup on the regular VLAN interface and Kerala Vision on the bridge interface.
Finally, I was able to get my WAN interfaces up, now the next problem. The unbound DNS resolver kept on getting restarted. Log entries like below were being logged repeatedly. I searched online but couldn't find a solution that worked for me. Every 4 seconds or so the resolver was getting restarted which breaks everything as DNS fails.
Anyway I found the problem, it was the DHCP client for the BSNL IPv6 which was constantly getting a
Setting this to true fixed the problem, as it is probably BSNL not responding.
So far things are working, I am still playing with how the fail over and load balancing should work. Overall, OpenWRT seems more mature for my use cases? I mean so many issues in getting something like this working which I could get it running in OpenWRT with no such challenges.
@C3PO @vishalrao @vignesh_venkatesan @achaudhary997 @JB700
So I got the mSATA SSD module (and a keyboard 😛) and I was trying to set up things over the last 1-2 days. At first I tried OPNsense, but to be honest, I didn't find it whole thing as refined or polished as pfSense. I couldn't even get my BSNL and Kerala Vision connections up for some reason. Maybe it is bit of unique or non-supported, I do not know, but the same config works on first try in OpenWRT without issues.
The Kerala Vision and BSNL connections come over the same fiber, goes into my Huawei ONT in transparent bridge mode (well, it doesn't have any other mode 😉) and then into the router, on the same port. Then VLANs are setup for each (1830 for BSNL Voice, 702 for BSNL Internet and 140 for Kerala Vision). On OPNsense, the the voice interface came up fine cause it was just a static IP config and didn't need mac spoofing. However for BSNL and Kerala Vision, as they do mac binding, I needed to spoof the mac.
But this doesn't appear to be supported natively OPNsense, and my PPPoE interfaces never even came up. Maybe once or twice, the BSNL interface came up but got disconnected in a short time. The logs show PPPoE timeout and retrying. 🤷♂️
I tried what I can and gave up.
Now on pfSense, the UI is more clean and polished. The layout also seemed better than OPNsense. I did the same config on pfSense and here they were pretty direct and mac spoofing on VLAN interfaces is not supported and must be set on the parent interface (this is the feature request Feature #1337: VLANs with different MAC address than parent interface - pfSense - pfSense bugtracker). But the issue is, I need two different macs when talking to Kerala Vision and BSNL (as that's what is originally registered on their end and I didn't want to get it reset).
I anyway I found a work-around in pfSense forum (MAC address spoofing on VLAN's and impressions from a second-try user) where you need to create bridge interface with just the VLAN interface as a member, and on the bridge interface you can spoof the mac. So BSNL Internet was setup on the regular VLAN interface and Kerala Vision on the bridge interface.
Finally, I was able to get my WAN interfaces up, now the next problem. The unbound DNS resolver kept on getting restarted. Log entries like below were being logged repeatedly. I searched online but couldn't find a solution that worked for me. Every 4 seconds or so the resolver was getting restarted which breaks everything as DNS fails.
Code:
Mar 22 13:51:25 mercury unbound: [54761:0] info: start of service (unbound 1.9.1).Anyway I found the problem, it was the DHCP client for the BSNL IPv6 which was constantly getting a
no addresses response. And for each such response, unbound gets restarted, I have no clue why 🤷♂️ Mostly related to this bug Bug #5413: Incorrect Handling of Unbound Resolver [service restarts, cache loss, DNS service interruption] - pfSense - pfSense bugtracker where unbound is restarted for each change rather than using unbound-control for such cases.
Code:
Mar 22 12:55:33 mercury dhcp6c[97095]: status code for NA-0: no addresses
Mar 22 12:55:41 mercury dhcp6c[97095]: status code for NA-0: no addresses
Mar 22 12:55:47 mercury dhcp6c[97095]: status code for NA-0: no addresses
Mar 22 12:55:54 mercury dhcp6c[97095]: status code for NA-0: no addresses
Mar 22 12:56:00 mercury dhcp6c[97095]: status code for NA-0: no addresses
Mar 22 12:56:06 mercury dhcp6c[97095]: status code for NA-0: no addresses
Mar 22 12:56:13 mercury dhcp6c[97095]: status code for NA-0: no addresses
Mar 22 12:56:19 mercury dhcp6c[97095]: status code for NA-0: no addresses
Mar 22 12:56:25 mercury dhcp6c[97095]: status code for NA-0: no addressesSetting this to true fixed the problem, as it is probably BSNL not responding.
So far things are working, I am still playing with how the fail over and load balancing should work. Overall, OpenWRT seems more mature for my use cases? I mean so many issues in getting something like this working which I could get it running in OpenWRT with no such challenges.
@C3PO @vishalrao @vignesh_venkatesan @achaudhary997 @JB700
