cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 

⁉️ Issues with IPv6 access to swisscom.ch website

lucaberta
Level 2
21 of 40

Thanks @SkyBeam for your additional considerations.

 

Yes, the website IPv6 test is a whole different issue compared to what I have mentioned vis-à-vis the swisscom.ch website.

 

It might very well be that some ICMPv6 messages are dropped elsewhere. Surely NOT on my Zyxel box, as I have created explicit rules on the different interfaces to allow such ICMPv6 packets on the local LAN. Rules on the VSDL interface would not matter as the frames would still be encapsulated in the 6RD tunnel.

 

Like you say, the issues are different and could very well be both coming from the Swisscom IPv6 network, or maybe from the firewalls in front on the web servers, at least for what concerns the issue with MTU.

 

Meanwhile, I have manually configured to MTU to 1420 on my macOS machine, and I can now access the IPv6 Swisscom website fine.

 

I will experiment with other settings too. I was not even aware that the MTU setting could be manually configured on macOS via the GUI, I have done from the CLI first, as an experiment!

 

image.png

 

This is the easiest way for me to fix things, as the radvd configurations is fixed on my Zyxel router. I should be running an OPNsense or pfSense box someday, I just don't have time to configure it properly right now!

 

And yes, I have done a few tests myself and most likely the tighter configs for ICMPv6 only seem to affect the swisscom.ch website, so maybe it's a local configuration on that load balancing cluster.

 

Ciao, Luca

lucaberta
Level 2
22 of 40

Thanks @LeylaG!

 

I have involved a longtime friend of mine who has been at Cisco for more than twice the years I have been in that company (which is still a good 11 years for me!) and is an IPv6 expert, and he believes that the discussion with @SkyBeam  is correct.

 

These are the remarks from my friend Éric, who clearly has read the thread... 👍🏻

 

Interesting and educated thread BTW. I suspect that it is a MTU issue 😞 either inbound to the server or outbound from the server. You can put your local MTU (on your device) to 1280 and see what happens... Of course, if this is inbound to you, then you would need to set your TCP MSS to 1240 or so...

 

ANYWAY, the issue should be solved by Swisscom, I can forward your email to the Swiss IPv6 Council and/or some Swisscom engineers (but unsure of the result of course)

Let's wait for an analysis from the higher level support engineers, and since this is a very easy to repeat issue, only if we don't get the appropriate level of response, we might escalate things outside of the “normal“ chain of command...😜

 

Meanwhile, I can access the IPv6 version of the website after having manually lowered the MTU of my Mac's interface to 1420.

 

Bye, Luca

 

lucaberta
Level 2
23 of 40

@lucaberta wrote:

 

Meanwhile, I have manually configured to MTU to 1420 on my macOS machine, and I can now access the IPv6 Swisscom website fine.

 

I will experiment with other settings too.

just tried with MTU of 1472, and it worked too. Anything above it doesn't work, like it was mentioned before.

 

L

IPv6@swisscom
Level 2
24 of 40

Let me give you some more details on this problem. It actually is a consequence of two mistakes.

 

First, let me explain why it works for 99.9% of all cases:

 

The scenario is shown below. We have a situation with a link with a smaller MTU than normal, like the mentioned 6rd or HE tunnel. The Router properly advertises this small MTU to the end system, which takes it into account to determine the Maximum Segment Size of the TCP session with the web server, making sure the packets never exceed the small link MTU. As the TCP client communicates the MSS to the server, it will also make sure its packets are not too large.

happy.png

 

Now, for a problem to occur, two things must happen:

  1. The Router has to incorrectly advertise a too large MTU to the client
  2. The Firewall in front of the web server has to drop inbound Packet Too Big messages

As the advertised MTU is too large, the TCP client selects an MSS that will result in packets that are too large to be transported across the small MTU link. The router will drop the packet and inform the client about the MTU that it should use. The MSS is not adjusted, however. The packet that was dropped is simply fragmented into two packets, which can be transported across the small MTU link and are reassembled by the server. The web server will reply with a packet that is also too big. Again, the dropping router will generate a Packet Too Big message. However, this message is dropped by the firewall, so the web server is not informed about it and cannot react with fragmentation.

unhappy.png

 

That's why it worked when @lucaberta changed the link MTU directly on the end system.

 

I've asked the firewall in front of the www.swisscom.com web server to be corrected. Rest assured that the IPv6 backbone of Swisscom does not filter ICMPv6 Packet Too Big messages. Eric would be telling me off if we did that, and rightfully so 🙂

SkyBeam
Level 2
25 of 40

Many thanks for the reply and the analysis. This sounds like the issue is going to be corrected soon.

 

However the explanation also includes the statement that the link which cannot carry more than 1472 Bytes is on the Client Router side. I would have to test again, but I am quite sure I was able to send 1480 Byte sized packages over the 6rd link (tunnel from my internal router directly to 6rd.swisscom.ch gateway).

If what you wrote in step 3 is true then this would lower efficiency because the client would try to send packages with MSS of 1480 and will have to re-send it in 2 packages as the MTU is 1472. But I did not see this happening. My 1480 Bytes long packages with don't fragment flag set did actually pass my router and the 6rd gateway (again: to be re-validated).

 

Your explanation however makes perfect sense assuming the MTU 1472 problem is entirely on 6rd tunnel side. But all the world seems to use MTU of 1480 for 6rd tunnels so all 3rd party routers will assume this MTU by default - some of them even not allowing to change it.

As mentioned by  @lucaberta he cannot change the advertised MTU on LAN side.

So your scenario explains why keeping MTU 1480 on client/LAN side will still trigger the failure even though the MTU on the 6rd tunnel is lowered. Reason is simply that the TCP MSS is kept and the response never gets delivered and the webserver is also not informed that the package got dropped on  its way back.

 

So actually if the firewall issue on Swisscom web server side is resolved, then the response packets still get dropped, the server is informed about this and will fragment its 1480 bytes long response into 2 packets, each smaller than 1472 bytes.

 

Actually this is indeed inefficient as the information is split into 2 packets with additional overhead. In addition it requires the server to re-send the response (as well as it requires the request to be re-sent fragmented as well).

 

 

 

In addition we got feedback from @c.jaquier that the issue also appears on an IPv6 tunnel from hurricane electric (HE). Assuming this tunnel can transport packets >= 1480 Bytes and this MTU is advertised to LAN on his setup too then step 2, 3, 4, 5 and 6 in your diagram does not take place. The client would send a full-sized 1480 Byte (or larger) package to the network (of course also using MSS >= 1480). The only way this could fail now is that a router on Swisscom network side (or the firewall) dropping the package and omitting/dropping the ICMP response to the client.

 

So following the explanation does not fully explain this effect on a "large-MTU" capable IPv6 connection.

 

 

I don't claim to be an expert on the topic and your explanation perfectly makes sense with a 6rd tunnel on a Swisscom IB where clients use wrong MSS of 1480 even though the IB announces 1472 on router advertisement frames to the LAN.

But what if you're using any internet connection which seriously is capable of routing 1480 Bytes long packages? Will they reach the web server in Swisscom setup? Will the Webserver be able to respond with equally sized packages?

 

 

There are 2 possibilities why also the requests via HE tunnel are blackholing:

  • The HE tunnel is also limited to less than 1480 bytes while the clients are wrongly advertised 1480 Bytes MTU (I would say this is unlikely)
  • Also the request or response from the web server is dropped due to the fact this server is also limited to 1472 Bytes and also the ICMP responses asking for fragmentation are never sent back to the client (and yes, my tcpdump has shown I never got any ICMP response when sending 1480 Bytes packages over the 6rd link to the Swisscom webserver).

 

Maybe I am missing something here. The explanation makes sense but does not cover the case the user is using a real "large-MTU" capable link. In such case the ICMP packet too big message should be received as an answer to the request (6 in your diagram).

 

 

EDIT: I just changed my MTU on the 6rd tunnel back to 1480 Bytes as well as advertising an MTU of 1480 Bytes to my LAN clients. And I can confirm that I can successfully ping Google using "ping www.google.com -l 1432" This results in an IPv6 packet with 1440 bytes length and 40 Bytes IP header which is a total of 1480 Bytes. So neither my Router nor the Swisscom 6rd gateway is dropping any packets of this size.

Tcpdump:

 

20:01:28.449232 IP6 2a02:120b:xxxx:xxxx::y> 2a00:1450:400a:803::2004: ICMP6, echo request, seq 154, length 1440
20:01:28.457061 IP6 2a00:1450:400a:803::2004 > 2a02:120b:xxxx:xxxx::y: ICMP6, echo reply, seq 154, length 76

So now I can't reach any Swisscom services any more.

Now returning to 1472 Bytes MTU (on Link and advertised to clients via radvd):

"ping www.google.com -l 1432" timing out now

tcpdump showing me I need to fragement at a maximum of 1424 (actually as expected, 8 Bytes less):

20:06:49.969566 IP6 2a02:120b:xxxx:xxxx::y > 2a00:1450:400a:800::2004: frag (0|1424) ICMP6, echo request, seq 159, length 1424
20:06:49.969567 IP6 2a02:120b:xxxx:xxxx::y> 2a00:1450:400a:800::2004: frag (1424|16)

Sure I can do this:

"ping www.google.com -l 1424"

tcpdump confirming success

20:09:26.150286 IP6 2a02:120b:xxxx:xxxx::y > 2a00:1450:400a:800::2004: ICMP6, echo request, seq 184, length 1432
20:09:26.157539 IP6 2a00:1450:400a:800::2004 > 2a02:120b:xxxx:xxxx::y: ICMP6, echo reply, seq 184, length 76

So somehow it's not my link which cannot carry 1480 Bytes and telling my Client that I need to fragment. There must be another router in the path which cannot carry the 1480 Bytes and it's response to me asking me to fragment is dropped.

 

At least this is my conclusion. However I am open to an explanation to this.

Edited
IPv6@swisscom
Level 2
26 of 40

@SkyBeam: the 6rd link of your's can support an MTU of 1480 in the upstream direction, because you don't have a PPPoE header (which would add another 8 Bytes of overhead). The 6rd Border Relay however also has to works for such links, so it is configured with an MTU of 1472. This asymmetry explains the behaviour that you describe.

SkyBeam
Level 2
27 of 40

IPv6@swisscom wrote:

@SkyBeam: the 6rd link of your's can support an MTU of 1480 in the upstream direction, because you don't have a PPPoE header (which would add another 8 Bytes of overhead). The 6rd Border Relay however also has to works for such links, so it is configured with an MTU of 1472. This asymmetry explains the behaviour that you describe.


Yes, that perfectly explains it; didn't think about the possibility to have asymmetric MTU 😯

 

Still not fully sure about 3rd party tunnels like HE where I guess in both directions larger MTUs might be possible. According to your description it shouldn't then. As the web server and all links in between would be capable of larger MTU.

 

So asymmetric MTU is actually explains at least my issue. But I might not fully understand yet why the border relay is using smaller MTU for its uplink and not just on PPPoE endpoints (as MTU is per link). But I am not into PPPoE design.

 

 

Bottom line is that an MTU of 1472 for Swisscom 6rd customers is anyway preferred as the return packets would have to be re-sent if an MSS of 1480 is requested but the answers of this size are getting rejected with the request to fragment them (causing additional load, traffic and overhead).

 

At least the issue was identified and hopefully in the future Swisscom services will be available also if the MSS value is too large and the packet needs to be fragmented and re-sent for any reason.

c.jaquier
Level 2
28 of 40

I looked at my HE tunnel settings and the maximum MTU that can be set for the tunnel is 1480. This is the default value and the one my tunnel is using.

 

I changed back the MTU to default on my IPv6 interface on my pfsense router and it seems that nowhere a PMTU of 1480 is advertised.

> tracepath -6 www.swisscom.com
1?: [LOCALHOST] 0.044ms pmtu 1500
1: pfsense.localdomain 3.611ms
1: pfsense.localdomain 1.552ms
2: tunnelxxxxxx.tunnel.tserv23.zrh1.ipv6.he.net 44.482ms
3: e0-16.core2.zrh3.he.net 44.857ms
4: equinix-zrh-v6.ip-plus.net 65.396ms
5: zhh-015-lo0-0.ip6.ip-plus.net 40.757ms
6: 2001:918:ce::45 42.078ms
7: 2a02:a90:4024:fff:8000::15 44.938ms asymm 8
8: no reply

I then changed the MTU of my IPv6 interface on the pfsense router to 1480 and can successfully connect to swisscom.ch or cff.ch.

> tracepath -6 www.swisscom.com 
1?: [LOCALHOST] 0.036ms pmtu 1500
1: pfsense.localdomain 3.762ms
1: pfsense.localdomain 2.410ms
2: pfsense.localdomain 1.874ms pmtu 1480
2: tunnelxxxxxx.tunnel.tserv23.zrh1.ipv6.he.net 41.883ms
3: e0-16.core2.zrh3.he.net 45.583ms
4: as3303.swissix.ch 42.456ms
5: zhh-015-lo0-0.ip6.ip-plus.net 41.472ms
6: 2001:918:ce::47 45.712ms
7: 2a02:a90:4024:fff::15 36.467ms
8: no reply

As you can see, my router advertises a PMTU of 1480.

 

I wonder what happened when my IPv6 interface was set to an MTU of 1500. Packets larger than 1480 should not have made it through the tunnel and I guess the tunnel interface should have replied back with a ICMP packet so that my host sends smaller packets. But then the same should have happened with swisscom.com and I shouldn't have had any issues!?

 

I probably have to read all your replies again 😉

SkyBeam
Level 2
29 of 40

@c.jaquier wrote:
As you can see, my router advertises a PMTU of 1480.

 

I wonder what happened when my IPv6 interface was set to an MTU of 1500. Packets larger than 1480 should not have made it through the tunnel and I guess the tunnel interface should have replied back with a ICMP packet so that my host sends smaller packets. But then the same should have happened with swisscom.com and I shouldn't have had any issues!?

 

I probably have to read all your replies again 😉


Just read our last discussions here. This will likely explain it. In short:

  • Your HE tunnel can likely handle 1480 Bytes in both directions
  • Your client is advertised MTU of 1500 on LAN as of your trace
  • Your pfsense will tell your client that a maximum path MTU of 1480 can be sent and asks for fragmentation
  • Your client framents the request into 2 packets, none larger than 1480 Bytes, however the TCP MSS size of the client remains set to 1500 Bytes
  • Your request packet(s) are forwarded to the Swisscom web server
  • The web server will try to answer with maximum MSS and will try responding at 1500 Bytes
  • The packet will hit the HE tunnel on HE-side
  • HE will respond with "packet too large" ICMP message asking the Swisscom server to fragment as it cannnot forward the 1500 Bytes response
  • The ICMP message is dropped (inbound) at Swisscom network
  • The web server will never see the request to fragment and silently time out
  • Your client in turn will never receive any answer

So the problem also here is that any system in the path between the client and the server cannot transfer the packets at the size the client is requesting it. So also on the return path the web server does not know this but fails to correctly respond as it never receives the informational ICMP message from the HE router.

 

So what you could do is to advertise a proper 1480 Bytes MTU on your !LAN! interface. So the clients know that no packet larger than 1480 will pass and they will set the MSS to 1480. This 1480 Bytes will also flow from the web server back to your client without any problem (assuming there is no other device in between with a smaller MTU).

 

As explained the problem will always happen when the client "thinks" the actual MTU on the device it's bound to (LAN) is higher than any device in the path to the server. So the client will tell the server it's maximum segment size (MSS) and in case this MSS is larger than the minimum MTU somewhere in the path the Swisscom server would be unable to adapt to this fact.

 

Most safe would be to advertise an MTU of 576 on your LAN. So no device would use any larger package (MSS) than this. 576 Bytes is the minimum MTU for IP links. However this is also causing efficiency issues and a high overhead. So it's not recommended.

c.jaquier
Level 2
30 of 40

Thanks for the summary, greatly appreciated. I got it now 👍

lucaberta
Level 2
31 of 40

IPv6@swisscom wrote:

Let me give you some more details on this problem. It actually is a consequence of two mistakes.

 

[...]

 

I've asked the firewall in front of the www.swisscom.com web server to be corrected. Rest assured that the IPv6 backbone of Swisscom does not filter ICMPv6 Packet Too Big messages. Eric would be telling me off if we did that, and rightfully so 🙂


thanks so much for looking into the issue and providing a detailed explanation from the Swisscom side!

 

I will email Éric right away and tell him to look for the updates to this thread, I am sure he will be happy to see that there has been quite a rapid development on this issue! And since he has been an IPv6 pioneer since day 1, I am sure he will be proud of seeing the IPv6 team at Swisscom acknowledging the issue and implementing a solution for it.

 

Back in my Cisco days, as Product Manager in EMEA for the PIX and later ASA firewall, asking the folks in San Jose to implement IPv6 was a very challenging task, let me tell you that! It's been more than 15 years ago, and now we are reaping the benefits of IPv6, finally!

 

Unless some firewalls somewhere in some data centers are kept a little too “tight“, that is! 😜

 

Again, thanks for looking into this, much appreciated. 👍🏻🙏🏻

 

Ciao, Luca

lucaberta
Level 2
32 of 40

Thanks also to you @c.jaquier together with @SkyBeam, you have given even more information on different scenarios which are still very common, like using an IPv6-in-IPv4 tunnel like that of Hurricane Electric, which I too use in some specific cases, after the sad shutdown of SiXxs, which has been working great for many years.

 

For those who are curios, here is the website:

https://www.tunnelbroker.net

 

This thread has become a real gold mine of IPv6-related information, I am very happy to having been the spark, and am even happier to have found great kindling in other IPv6 aficionados such as yourselves.

 

And like IPv6@swisscom said, hey, we are part of a 0.1% of the people having the issue, what an elite! 😜😄🤣

 

Looking forward to seeing the fixes implemented soon on the Swisscom firewalls.

 

Ciao, Luca

lucaberta
Level 2
33 of 40

I just have one short final reflection on this issue, which came out by re-reading again all the thread and linked messages.

 


IPv6@swisscom wrote:

 

Now, for a problem to occur, two things must happen:

  1. The Router has to incorrectly advertise a too large MTU to the client
  2. The Firewall in front of the web server has to drop inbound Packet Too Big messages

[...]

 

That's why it worked when @lucaberta changed the link MTU directly on the end system.

 

I've asked the firewall in front of the www.swisscom.com web server to be corrected. Rest assured that the IPv6 backbone of Swisscom does not filter ICMPv6 Packet Too Big messages. Eric would be telling me off if we did that, and rightfully so 🙂


clearly point #2 has been addressed correctly by IPv6@swisscom and I look forward to the changes being implemented on the firewalls protecting the webservers.

 

Yet, point #1 falls on to each user's lap, and my reflection follows.

 

I have used the standard Swisscom-provided CPE for years, since 2014, and it was an InternetBox Plus, which worked quite well also as a 6RD tunnel endpoint for my always-successful connection to the IPv4 and IPv6 internet.

 

I never had a single problem with the swisscom.ch/.com website, in spite of the issue mentioned at point #2, simply because the radvd daemon running on the ISP-provided CPE was correctly configured to announce an MTU of 1472.

 

This broke when, two weeks ago, I decided to change CPE and bought a new Zyxel XMG3927-B50A router, which is listed on the BBCS list as an approved CPE by Swisscom for their wholesale service:

 

E_BBCS_Supporting-Document_Proved-Equipment

(see page 4 almost at the bottom, where the Zyxel XMG3927-B50A is listed)

 

I have documented the setup of the box which was quite easy for someone as geeky as me, including the DHCP option 60 and 6RD configuration, in this other thread on this community:

Report on good VDSL2 experience with Zyxel XMG3927...               

 

What I was missing is the fact that the advertised MTU on the Zyxel's implementation of radvd *CANNOT* be changed from the GUI, and most likely it defaults to the LAN MTU since I cannot find any indication of an MTU advertisment done by the Zyxel radvd implementation:

 

radvd packet trace.png

 

So in the end it was the router change, and the inability to announce a smaller MTU by the Zyxel router, that created this whole situation.

 

Had I stayed with the Swisscom-provided CPE, such as the new IB3, I would never have had the issue.

 

And the Swisscom firewalls would NOT have been fixed, like they should... 😜👍🏻💪🏻

 

Thanks everyone, this was a most enjoyable group troubleshooting experience, and I am convinced that we all gained a lot from it, and many users will too as they will access the IPv6 versions of the Swisscom website, without knowing what happened behind the scene!

 

Ciao, Luca

Edited
IPv6@swisscom
Level 2
34 of 40

A last update on this thread from my side: The firewall configuration was corrected last night, so the problem should no longer be there, even for CPEs that are configured with improper MTU settings.

SkyBeam
Level 2
35 of 40

IPv6@swisscom wrote:

A last update on this thread from my side: The firewall configuration was corrected last night, so the problem should no longer be there, even for CPEs that are configured with improper MTU settings.


I just tried to verify but nope. I still can't reach swisscom.ch pages on IPv6 using 6rd tunnel MTU (and LAN MTU including TCP MSS) of 1480.

 

Also it doesn't really sound correct to me claiming this to be "improper" MTU settings if it's entirely a limitation of Swisscom network and caused by firewalls dropping ICMP messages. There are other providers out there which might use even larger MTU and any router in between the return path will send ICMP messages about required fragementation. So this is more a common use case and not in any case an improper MTU setting.

 

Does it work for the others now? For me it still doesn't.

lucaberta
Level 2
36 of 40

I am having the same experience as @SkyBeam.

 

If I keep the MTU of my machine at 1472 or lower, things work.

 

If I set it any higher, then swisscom.ch/.com does not reply.

 

Apparently the changes implemented by IPv6@swisscom and his team do not seem to be having the expected result.

 

Thanks, Luca

lucaberta
Level 2
37 of 40

IPv6@swisscom wrote:

A last update on this thread from my side: The firewall configuration was corrected last night, so the problem should no longer be there, even for CPEs that are configured with improper MTU settings.


the “problem“ is never there for “for CPEs“, in that their only fault is not to announce the default MTU of 1472 which is what the Swisscom-provided CPEs instead do.

 

But the CPEs are NOT affected by the issue in that they are not going to be acting like clients in any way and won't care nor notice of any changes of firewall configurations.

 

On the other hand, LAN clients who receive their IPv6 via radvd would NOT be affected by the issue if radvd announces an MTU of 1472.

Which my Zyxel CPE does NOT do, and that is why I either manually force the MTU on my LAN card, or I am affected by the issue.

 

In short, your last sentence should better be phrased like this:

 

[...]so the problem should no longer be there, even for those IPv6 clients that are configured with MTU settings above 1472, which is the Swisscom suggested value for the current 6RD/IPv6 setup.

Curious to see where the issue is, now!

 

Many thanks for your continuous engagement on this issue!

 

Bye, Luca

lucaberta
Level 2
38 of 40

Good news! I have just tried again by setting the MTU back to the default of 1500, and this time things seem to be working fine!

 

@SkyBeam can you confirm on your side?

 

Maybe the firewall rules set up by IPv6@swisscom needed some tweaking, and now seem to be working fine.

 

I wonder if the issues that some lamented on the SBB/CFF/FFS website are linked to the same problem. I read in other threads that this was a longtime outstanding issue. Maybe the railroads are in the same data center and are suffering from the same too restrictive rules on ICMPv6?

 

Can anyone comment on SBB/CFF/FFS?

 

Thanks all for the great teamwork! 👍🏻🙏🏻

 

And Merry Christmas to you all! 🎄🎁

 

Ciao, Luca

SkyBeam
Level 2
39 of 40

@lucaberta wrote:

Good news! I have just tried again by setting the MTU back to the default of 1500, and this time things seem to be working fine!

 

@SkyBeam can you confirm on your side?

No. Unfortunately a link MTU and local MTU/MSS of 1480 does NOT work for me on swisscom.ch web-pages. Still timing out (not getting any response at all).

But I do not have any issues with sbb.ch or train-schedule lookups.

I did actually have problems in the past with SBB park & rail app (prail.sbb.ch) but the problem there was different: The app seems to use SSL CONNECT method on port 2345 and 10433 which was disallowed in my proxy config (SSL_ports). So I needed to allow it.

lucaberta
Level 2
40 of 40

@SkyBeam wrote:
No. Unfortunately a link MTU and local MTU/MSS of 1480 does NOT work for me on swisscom.ch web-pages. Still timing out (not getting any response at all).

bizarre. Now it also has stopped working for me. Go figure.

 

Looks like we're still stuck with the issue. Time to put my machine's MTU back at 1472.

 

I wonder why it was working before...

 

L

Back to top