[Technical] Problems with second network

  • Hey ho

    It’s been a while since I last contacted you, I set up a small home lab about half a year ago, it’s structured as follows:

    Unfortunately, I no longer have access to all http/https services in the opnsense subnet every day, the opnsense appears as online in the router, but access to my homelab is no longer possible, direct IP access usually works completely, only http/https doesn’t work at all .

    A Swisscom router restart makes all problems disappear. (Router otherwise works without problems for Swisscom subnet)

    I’m not really sure what details to add here at the moment, so I’ll leave it alone.

    Thank you for any help solving this problem!

    Show original language (German)

    @Neliommiosch84

    According to two other threads of yours, you have been experimentally working on several special solutions in your network area over the last few months.

    If anyone wants to join in the discussion on the new questions, I think it’s relevant to read through the previous threads beforehand, so I’ll link them here:

    [https://community.swisscom.ch/t5/sicherheit-im-Internet/DMZ-Firewall-und-WLAN/td-p/791751](https://community.swisscom.ch/t5/sicherheit-im- Internet/DMZ-Firewall-and-WLAN/td-p/791751)

    and also:

    [https://community.swisscom.ch/t5/Internet-general/DNS-Server-aus-anderem-Subnetz/m-p/815635#M71908](https://community.swisscom.ch/t5/Internet-general/ DNS-server-from-another-subnet/m-p/815635#M71908)

    I don’t feel able to do this myself at the moment, as I have no idea whether you are working with strictly separate networks or with a hybrid network including a static route, what kind of DNS construct is currently active, and in which network There are actually the clients that are currently causing problems 🙂

    Show original language (German)

    Hobby-Nerd ohne wirtschaftliche Abhängigkeiten zur Swisscom

    Hello @Werner

    Thank you for the link, that would have been a good idea!

    I decided on the 2 separate networks, which can be reached via a static route, so I suspect the hybrid variant.

    DNS is both separated.
    One DNS server for the Swisscom clients, two DNS (pihole/adghome) via OPNsense for the servers in the “server network”.

    I have the feeling that DNS will hardly be a problem here, since it is handled by a separate device within the Swisscom Ibox subnet, and therefore my PC can resolve DNS at any time while the OPNsense Net is “down”.

    I also have less of a problem with the reverse proxy, as in the end a Swisscom router reboot was always necessary to solve the problem.

    The entire Opnsense network is actually causing problems, but I increasingly have the feeling that it is not the OPNSense as always suspected, but the Swisscom IBox that is the problem, since it is responsible for the static route.

    What I’m also wondering is that, theoretically, there are currently several 10.X cause something to overflow?

    I’ve heard of some kind of Nat Table overflow or something like that.

    Show original language (German)

    @Neliommiosch84

    As already mentioned earlier, my own approach would be to avoid “mishmash” in networks and the resulting unnecessarily increasing complexity as early as possible by design.

    So my solution would still be:

    - all clients (except perhaps the blue TV boxes) directly into the network behind the pfSense

    - Delete static route between IB and cascaded network for clear network separation and increased security

    This means that you would no longer have any possible side effects of the combined operation, because DHCP and DNS would then be centralized for everything on the pfSense.

    By the way, I’m happy to let @r00t judge your current problems with the hybrid approach, because unlike me, he actually has a pfSense network in operation at the moment.

    Show original language (German)

    Hobby-Nerd ohne wirtschaftliche Abhängigkeiten zur Swisscom


    @Neliommiosch84 wrote:

    […] Unfortunately, I no longer have access to all http/https services in the opnsense subnet every day, the opnsense appears as online in the router, but access to my homelab is no longer possible, direct IP access usually works completely, only http/https doesn’t work at all.

    What exactly do you mean by “direct IP access works most of the time”? Ping? And from where does http and https not work; from externally (Internet) or from the home network?

    Another question about your server network: how big is it? /24 or /16 or something in between?

    I briefly read through the other two (long) threads and summarized the most important things from my point of view (correct me if I misunderstood something):

    - On the opnsense (or is it a pfsense according to some of your posts?!?) NAT is disabled

    - The opnsense is configured as a DMZ host on the Internet box

    - A static route for the server network via opnsense is set up on the Internet box

    - The DHCP server in the home network is the IB, whereby you let the clients use their own DNS resolver in the server network (or does it run in the client network?) using option 6

    - The DNS server is located in your server network, with opnsense (pfsense?) configured as a DNS proxy

    - Clients (LAN/WLAN) all run directly on the IB and not in the server network, so there is no longer an IGMP proxy on the opnsense

    It’s permissible to remark that you’ve already put together a pretty complex construct of potential sources of error…

    Show original language (German)

    Have you tried turning it off and on again?

    Hey @PowerMac

    Thank you for the answer!

    - Currently I’m always talking about access from the home network, the network directly behind the Swisscom IBox where all clients live.
    The http/https access also refers to this, as an example, I have jellyfin on ip xyz:8096 and via reverse proxy on jellyfin.mydomain.com. I can then access the services directly via IP, but not with the domain .
    - It’s an opnsense, used to be a pfsense, but I was no longer so comfortable with the current netgate situation.
    - I have currently switched off DMZ because the homelab can currently only be accessed from outside via VPN.
    - The Internet box has two static routes, one is the server network -> a subnet for all physical hosts, and one into the VM/container network, a subnet for all services, containers, VM’s and LXC’s.
    - DHCP is the Ibox and in the server network the OPNsense.
    - Theoretically, I could remove the LAN cable from the server network and have my existing, completely “normal” Swisscom home network.
    - Exactly the clients in my Swisscom home network are referred to a RaspberryPi with Pihole via Option6.
    - The servers have all registered OPNsense as DNS servers, which forwards the requests to a Pihole/AdguardHome that are running on the servers.
    - Exactly correct, IGMP Proxy is not on the OPNsense, because the OPNSense really only has servers behind it.

    Thank you very much for your help!

    And sorry for the few details at the beginning!

    Show original language (German)

    @“x”#234740I would love to implement your approach, which I completely understand, but unfortunately it’s not possible at all because the servers are about 50m away from the IBox and the main connection is relatively difficult to relocate, I’ve already done that often tried to suffice.

    The IBOX is currently on the Swisscom main connection, but since there is only limited space available there, it is not really possible to connect the opnsense there.

    Show original language (German)

    @Neliommiosch84

    Why would you have to place the OpenSense appliance directly next to the Internet box for a truly separate network?

    Where it is now, it is probably connected to a LAN cable and this can be up to 100 meters long.

    Do you also have an additional physical LAN cabling problem?

    Show original language (German)

    Hobby-Nerd ohne wirtschaftliche Abhängigkeiten zur Swisscom

    @“x”#234740No, since I currently have my servers including Opnsense in the basement and can only run a single LAN cable there (due to lack of space in the pipes in the wall), I could not connect the clients of the home network behind the OPNsense , so my clients have to stay behind the IBox in the home network, and a switch at the top is not possible because it requires 2 cables (one down, one up).

    Sorry, but I’ve already discussed this topic too many times, here on Reddit, Opnsense forums, etc.
    IT IS NOT POSSIBLE OTHERWISE!

    I would therefore like to try to solve the other problems of the current situation.

    Thank you anyway for the solution to changing the network structure.

    Show original language (German)

    @Neliommiosch84

    And as a general addendum on the topic of the number of possible clients directly in the network of an Internet box:

    In the past, there have been repeated user reports about problems with Internet boxes when dealing with a very large number of clients in the home network.

    It is somewhat unclear where the specific size limits are, but based on various practical experiences it has become clear that anything with more than 60-80 clients can be a problem repeatedly.

    By the way, the maximum limits in the WLAN area are clearer, because depending on the WLAN drivers used, they are 32 or 64 WLAN clients per WLAN band, depending on the specific model.

    Long story short, Internet boxes are purely SOHO devices for run-of-the-mill users, and usage that deviates greatly from the usual user profile always leads into unexplored territory, as it lies outside of any of the commonly used test scenarios.

    Show original language (German)

    Hobby-Nerd ohne wirtschaftliche Abhängigkeiten zur Swisscom


    @Neliommiosch84 wrote:

    […] I would therefore like to try to solve the other problems of the current situation.


    The only thing I understand from your initial post is the following

    • Sporadically problems occur with access to http and https services in your server network that is disconnected via opnsense
    • The error condition disappears for approx. one day after rebooting the Internet box

    In order to solve a problem, it is always helpful to first understand it as precisely as possible 😀

    And in order to achieve exactly this, you should leave the error state as it is next time and then test what still works and what doesn’t. So for example:

    • Does ping go from the user network to the Internet?
    • Does ping go from the user network to the internal server network?
    • Does ping go from the internal server network to the user network?
    • Does ping go from the internal server network to the Internet?
    • Is DNS possible in the user network (if necessary, don’t use your own DNS server, but rather the IB’s)?
    • Does DNS work in the internal server network?
    • Does http/https access come from the Internet?
    • Does http/https access come from the user network?
    • Does http/https access come from a client within the internal server network?
    • Does it really only affect http/https services, or others too?
    • etc. etc.

    So next time, don’t restart the IB straight away, but rather systematically test one thing at a time and narrow down the error. Divide and conquer.

    Show original language (German)

    Have you tried turning it off and on again?

    14 days later

    Small update for this thread:

    After a long time of back and forth, it turned out that there was an error in the driver of the Intel i225 + i226 network modules. This had already been fixed upstream and was completely resolved with a BIOS update in July.

    https://forum.opnsense.org/index.php?topic=42368.msg210625#msg210625 -> https://forum.opnsense.org/index.php?topic=42240.0 -> https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=279245

    Huge thanks to the OPNsense community and of course everyone for the input!

    Show original language (German)