Hi everyone, Over the last two/three days, I have been noticing having connection issues with the swisscom.ch website via IPv6, which is my default as I use a 6RD connection from my router, connected to the internet via a Swisscom VDSL2 line. This is independent of the device and OS, since I am experiencing the issue on macOS, Windows, iOS and iPadOS. My macOS laptop with Safari shows this error when trying to connect: [upl-image-preview liId="39689i389FE826042AEBBF" alt="image.png" uuid="52138548-5fa1-433f-98c6-f68bd9446580"] If I switch off wifi on my iPad, and pass via the 4G network, the page opens fine, as the 4G connection is only IPv4 as confirmed by the excellent test website: [https://ipv6-test.com/](https://ipv6-test.com/) On the other hand, this website [https://community.swisscom.ch/](https://community.swisscom.ch/) is not affected by the issue, and I can access it regularly over IPv6. Am I the only one experiencing these connection issues with the swisscom.ch website? Many thanks, Luca

Hello @"lucaberta"#110906, Effectively the ipv6 webserver swisscom.ch (2a02:a90:c400:4001::2) isn't reachable. @"x"#103702could you report this issue?

Hello @"Biorn1950"#571597 @"lucaberta"#110906 Thank you for your contributions. I forwarded your reporting to our specialists. Let's wait for their answer. Have a nice day

I have reported and resolved this already. See [here](https://community.swisscom.ch/t5/Internet-Allgemein/IPv6-6rd-MTU/td-p/641943). Solution is to reduce your MTU from 1480 to 1472. And yes, there is no technical reason and the issue is caused by Swisscom creating a blackholing by blocking the ICMP response packets from the hosts which would notify your host to reduce the MTU. Basically Swisscom services are unavailable on any IPv6 enabled internet connection with an MTU larger than 1472.

@"SkyBeam"#1560 I just tried with mtu 1472 and the server still not answer.

⁉️ Issues with IPv6 access to swisscom.ch website

Biorn1950

@SkyBeam

I just tried with mtu 1472 and the server still not answer.

SkyBeam

@Biorn1950 wrote:

I just tried with mtu 1472 and the server still not answer.

Are you using any additional encapsulation? You might try even lower values then. However for me it works fine using MTU of 1472 on 6rd tunnel to 6rd.swisscom.com. Also make sure the MTU of 1472 is announced properly on your LAN devices (router advertisements). I am operating radvd on my LAN and use "AdvLinkMTU 1472" property.

Biorn1950

@"x"#1560thanks for your answer.

Just to be sure, at this moment you have an answer if you: ping6 swisscom.ch?

Could you try it please and confirm me which ip is reached.

SkyBeam

@Biorn1950 wrote:

@"x"#1560thanks for your answer.

Just to be sure, at this moment you have an answer if you: ping6 swisscom.ch?

Could you try it please and confirm me which ip is reached.

No. Absolutely not of course. This is part of the problem. Swisscom seems to filter all (yes all, not just echo-request/reply) ICMP traffic on swisscom.ch. So you will never get an ICMP echo reply. Even worse you will also not get ICMP error messages from the host's you want to reach.

And exactly this is the problem:

You are sending a package with a size larger than 1472 to the swisscom.ch host
The host receives the package (or a router in between) but it cannot forward it due to an internal link having a maximum MTU of 1472 (6rd tunneling + PPPoE encapsulation). The router/host will reply with ICMP "Fragementation Needed" reply (Type 3, Code 4).
Unfortunately the reply is also dropped on it's way back to you.
Your client will wait forefer or any response. It will neither get any TCP answer nor any ICMP fragmentation needed message. So you're lost and the connection just times out.

The solution is to assure that your client will never even try to send a packet larger than 1472 bytes to the swisscom.ch host (TCP packets on Port 443) so it's not getting dropped. Of course this is completely against any standard and a misconfiguration on the network but there are companies which still believe ICMP can be simply blocked and there is no real value of those information. Some of them only know ICMP type "echo request/reply" and think that's not needed. But specifically with IPv6 it's important also to forward ICMP traffic. So I am afraid that actually nobody using any provider which provides native or large-mtu IPv6 access will be able to reach any swisscom.ch services (with some exceptions like this community pages which seems to be on a host which can transport larger IPv6 packets (working with 1480 bytes MTU at least). But the majority of swisscom.ch services including MyCloud and the customer center simply will not work. The only workaround known to me is to announce a maxium MTU ofr 1472 bytes or smaller to your clients - at least for the path towards Swisscom services.

Perhaps they will fix it once in a while, but I wouldn't hold my breath as I am facing this issue since years.

Biorn1950

@"x"#1560it's most clear now, that why I can't get any answer with an mtu set to 1472 or below.

Thanks for your explanations, I was disturbed by the fact I could reach and diagnose while ping6 mtu size for community.swisscon.ch and not for the main domain.

Perfect, thanks again.

lucaberta

Thanks @SkyBeam for shedding a little light on this issue.

It looks like changing the MTU down to 1472 did not change things for me. I also tried 1460 in case of additional padding from other (which ones?) tunnels, but no luck, still.

Also, I am using a Zyxel router and I don't believe I can change radvd's configuration to advertise a smaller MTU to the clients on the LAN, I am afraid.

The question now is, why would Swisscom block important ICMPv6 frames altogether, when it's best practice to allow them?

There is an RFC which gives best practices on the topic:
https://tools.ietf.org/html/rfc4890

On the other hand, most ICMPv6 error messages traveling end-to-end or
any-to-end are essential to the establishment and maintenance of
communications. These messages must be passed through firewalls and
might also be sent to and from firewalls to assist with establishment
and maintenance of communications. For example, the Packet Too Big
error message is needed to determine the MTU along a path both when a
communication session is established initially and later if the path
is rerouted during the session.

I wonder if all these issues I am seeing would simply be solved by implementing the correct best practices at Swisscom, rather than zapping all the ICMPv6 frames altogether...

Any further comment is much appreciated.

Ciao, Luca

SkyBeam

@lucaberta wrote:

I wonder if all these issues I am seeing would simply be solved by implementing the correct best practices at Swisscom, rather than zapping all the ICMPv6 frames altogether...

Sure it would be resolved. As written some host/router in between the client and the server will actually respond with an ICMP package indicating that it cannot forward the request due to its size. But as this response is dropped your client is waiting forever, not knowing the request has actually been silently dropped.

Actually it should be enough to lower the MTU of your 6rd tunnel on your gateway to 1472. Assuming your gateway does not suffer from the same issues (you don't silently drop ICMP to your clients) then your gateway should actually tell your client to send smaller packages if it cannot forward them on the MTU-1472 6rd tunnel. So for your local network it would be sufficient. However it's perhaps a bit less efficient if your RA messages advertise 1480 on your LAN and your clients will detect on each TCP connection establishment that the actual MTU for global-unique IPs are limited to 1472 so the request needs to be split and re-sent.

You can actually check the MTU on your client. On Linux just use 'ip a' command. On Windows open a command prompt and use the 'netsh interface ipv6 show subinterface' command to see the MTU advertised on your local link.

Likely this one is 1480 but if your gateway needs to forward it on a link with MTU 1472 it will tell your client to send smaller packets (via ICMP). Since you don't block this (hopefully) it will work.

Not sure however if your settings on the gateway are properly done. I am using a Linux router with 6rd tunnel. I had to modify the init script to allow me to set a custom MTU (ip lin kset mtu 1472 <6rd-interface>). I guess your Zyxel perhaps allows to set a manual MTU on the 6rd tunnel. But you need to make sure it is in effect for the 6rd tunnel and not for any other interface.

I am still very convinced that this is exactly your problem but I can't test your setup as I am not using any closed/embedded devices.

In regards to why Swisscom is blocking ICMP all over... I don't know. I guess some "smart" security guy was living in the age where "ping of death" was a real thing and thought "why do we actually need ICMP? We just open TCP port 443, that's enought as the service just listen on this port". Well security engineers are usually not network engineers and I am sure there are many wrongly configured gateways, routers and hosts out there.

lucaberta

SkyBeam wrote:

In regards to why Swisscom is blocking ICMP all over... I don't know. I guess some "smart" security guy was living in the age where "ping of death" was a real thing and thought "why do we actually need ICMP? We just open TCP port 443, that's enought as the service just listen on this port". Well security engineers are usually not network engineers and I am sure there are many wrongly configured gateways, routers and hosts out there.

thanks @SkyBeam, I very much appreciate your comments!

Now things have finally clicked on why my score on

https://ipv6-test.com

could never go above 18/20!

Except that it is **NOT** my router or firewall. It's my ISP that's doing it!!! 😤😡🤬

I really wonder what can be done to fix this very annoying situation with Swisscom and their faulty IPv6 implementation.

I will ping my longtime friend Éric at Cisco, he is an IPv6 and security extraordinaire and author of multiple RFCs, and I should be getting some good comments from him.

Would any email thread on Swinog help too?

Bye, Luca

c.jaquier

I'm on CATV (soon to move to Wingo DSL) and I'm using an IPv6 tunnel from HE.net. I was not able to connect to swisscom.ch or sbb.ch. That's the main 2 websites that failed for me on IPv6, most others works fine. Reading this thread, I just lowered my MTU to 1472 on my tunnel interface in pfsense and I can confirm that I can now successfully access both website via IPv6 🙂

SkyBeam

@c.jaquier wrote:

IPv6 tunnel from HE.net. I was not able to connect to swisscom.ch or sbb.ch.

Thanks for the feedback.

This does indeed confirm that the hypothesis is correct as well as confirming that also non-Swisscom IPv6 users will be affected.

I also believe that not only Swissco and SBB service are affected but actually there is a bunch of other services out there which suffer from the same problem. Although I guess many services are at least IPv6 and IPv4 dual-stacked and moving to IPv4 still provides a fallback. However it doesn't leave a good impression if a potential customer willing to get information about Swisscom services will just not get any response from the official web pages. And the average user will not know the reason, but they will of course be able to surf to competitor pages.

For SBB it might be even worse as there is basically no way around their services when traveling with public transport.

Sad situation, but the solution actually needs to be implemented by these companies and they don't even acknowledge the problem.

Biorn1950

@SkyBeam wrote:

In regards to why Swisscom is blocking ICMP all over... I don't know. I guess some "smart" security guy was living in the age where "ping of death" was a real thing and thought "why do we actually need ICMP? We just open TCP port 443, that's enought as the service just listen on this port". Well security engineers are usually not network engineers and I am sure there are many wrongly configured gateways, routers and hosts out there.

🤣 good summary 🤣

Yes that could be lots of things, miss configuration somewhere, issue with load balancing like ecmp, firewall, or a genius idea from a smart guy 😁

SkyBeam

lucaberta wrote:

Now things have finally clicked on why my score on

https://ipv6-test.com

could never go above 18/20!

Except that it is **NOT** my router or firewall. It's my ISP that's doing it!!! 😤😡🤬

To be honest I am not fully sure if here the ISP is to be blamed.

In fact this test will test the connectivity between the test-website and your local computer. This is unrelated to the issue trying to reach swisscom.ch services where the firewall protecting those servers seem to overdo their job slightly.

Swisscom does actually not filter ICMP in general. At least not ICMP echo requests as you can for sure ping your IPv6 address assigned to a LAN device from outside (from any other IPv6 internet connection).

However I did not test yet if the ISP/Swisscom network would filter some other ICMP types in general. Unfortunately also ipv6-test.com does not show any more details which type of ICMPv6 message they are testing. I don't think this is ICMP echo request as echo requests to my LAN devices work just fine on IPv6 on Swisscom network. At this point I would be unaware about any (ICMP) filtering Swisscom applies to consumer connectivity. Also note that a firewall on your client machine can also be configured to block ICMP from/to the client directly.

At least I think we are looking for two different problems:

ICMP responses from Swisscom internal networks for public services (like swisscom.ch or mycloud.ch web-pages) are dropped
ICMP responses from customer access network to/from other public services might be dropped (not fully confirmed yet)

Edit: I am not 100% sure but I think there might be a general problem with ipv6-test.com regarding ICMP testing. First of all in the IPv6 section ICMP is reported to be "Not tested". And also the Ping test they offer does not work for me - neither for IPv4 nor for IPv6. I am not sure yet if this is eventually blocked by browser policies recently too or it's a service problem. I also don't see any ICMP packets sent from my client when running any of those tests. So I think the test there is not reliable in regards to ICMP. Which supports my second point above - I don't believe ICMPv6 is blocked on network side by the ISP/Swisscom. The problem is limited to Swisscom services being over-protected.

IPv6swisscom

Let me give you some more details on this problem. It actually is a consequence of two mistakes.

First, let me explain why it works for 99.9% of all cases:

The scenario is shown below. We have a situation with a link with a smaller MTU than normal, like the mentioned 6rd or HE tunnel. The Router properly advertises this small MTU to the end system, which takes it into account to determine the Maximum Segment Size of the TCP session with the web server, making sure the packets never exceed the small link MTU. As the TCP client communicates the MSS to the server, it will also make sure its packets are not too large.

Now, for a problem to occur, two things must happen:

The Router has to incorrectly advertise a too large MTU to the client
The Firewall in front of the web server has to drop inbound Packet Too Big messages

As the advertised MTU is too large, the TCP client selects an MSS that will result in packets that are too large to be transported across the small MTU link. The router will drop the packet and inform the client about the MTU that it should use. The MSS is not adjusted, however. The packet that was dropped is simply fragmented into two packets, which can be transported across the small MTU link and are reassembled by the server. The web server will reply with a packet that is also too big. Again, the dropping router will generate a Packet Too Big message. However, this message is dropped by the firewall, so the web server is not informed about it and cannot react with fragmentation.

That's why it worked when lucaberta changed the link MTU directly on the end system.

I've asked the firewall in front of the www.swisscom.com web server to be corrected. Rest assured that the IPv6 backbone of Swisscom does not filter ICMPv6 Packet Too Big messages. Eric would be telling me off if we did that, and rightfully so 🙂

LeylaG

Hello @lucaberta and thank you very much @SkyBeam for your help 🙂

According to our specialist, after an analysis with multiple tests, no IPV6 problems are visible on the swisscom.ch website.

I am sorry not being able to provide you more information.

lucaberta

@LeylaG wrote:

According to our specialist, after an analysis with multiple tests, no IPV6 problems are visible on the swisscom.ch website.

I am sorry not being able to provide you more information.

this is NOT your fault @LeylaG, it's an issue with the security architecture design which is flawed, as clearly demonstrated by @SkyBeam's excellent analysis.

And it's not an issue with the website, rather with the way the Swisscom IPv6 backbone deals with important messages such as ICMPv6.

If Swisscom is not willing to implement the best practices on ICMPv6 messages and firewalling, nicely described in RFC4890, then the issue is much higher than the support staff.

This sucks, and I am not sure how it could be further escalated, since Swisscom's IPv6 backbone is NOT correctly implementing best practices which will make the network run better.

And that's really too bad. 😤😡🤬

Bye, Luca

SkyBeam

@LeylaG wrote:

According to our specialist, after an analysis with multiple tests, no IPV6 problems are visible on the swisscom.ch website.

Sure, if you ask the team responsible for operating the web page they do not see a problem. They don't even see any request packet flowing in larger than 1472 Bytes as every packet larger than this size is already dropped before reaching the web server. So sure for them it looks fine, they never get the requests.

Still the page looks "dead" to all the world out there.

Also clearly if Swisscom staff is testing this from their own home connections (on Swisscom internet box) they will not see a problem either as the internet box is setting a maximum MTU of 1472 Bytes.

But still MTU of 1472 is not the largest MTU other connections will offer (like IPv6 from other providers) and therefore they will face issues.

The bad thing about this is that in the future Swisscom will not even be able to offer competitive offerings to potential customers which would like to switch from another provider to Swisscom as basically they can't even access the Swisscom web page to browse the offers and sign up. Sure you can ask them to go to the shop, but it's a bit ridiculous...

lucaberta

Thanks SkyBeam for your additional considerations.

Yes, the website IPv6 test is a whole different issue compared to what I have mentioned vis-à-vis the swisscom.ch website.

It might very well be that some ICMPv6 messages are dropped elsewhere. Surely NOT on my Zyxel box, as I have created explicit rules on the different interfaces to allow such ICMPv6 packets on the local LAN. Rules on the VSDL interface would not matter as the frames would still be encapsulated in the 6RD tunnel.

Like you say, the issues are different and could very well be both coming from the Swisscom IPv6 network, or maybe from the firewalls in front on the web servers, at least for what concerns the issue with MTU.

Meanwhile, I have manually configured to MTU to 1420 on my macOS machine, and I can now access the IPv6 Swisscom website fine.

I will experiment with other settings too. I was not even aware that the MTU setting could be manually configured on macOS via the GUI, I have done from the CLI first, as an experiment!

This is the easiest way for me to fix things, as the radvd configurations is fixed on my Zyxel router. I should be running an OPNsense or pfSense box someday, I just don't have time to configure it properly right now!

And yes, I have done a few tests myself and most likely the tighter configs for ICMPv6 only seem to affect the swisscom.ch website, so maybe it's a local configuration on that load balancing cluster.

Ciao, Luca

LeylaG

Dear @SkyBeam @lucaberta

I forwarded your concern to the specialist in charge.

I cannot do more at my level I am afraid.

Thank you for your understanding.

lucaberta

Thanks @LeylaG!

I have involved a longtime friend of mine who has been at Cisco for more than twice the years I have been in that company (which is still a good 11 years for me!) and is an IPv6 expert, and he believes that the discussion with @SkyBeam is correct.

These are the remarks from my friend Éric, who clearly has read the thread... 👍🏻

Interesting and educated thread BTW. I suspect that it is a MTU issue 😞 either inbound to the server or outbound from the server. You can put your local MTU (on your device) to 1280 and see what happens... Of course, if this is inbound to you, then you would need to set your TCP MSS to 1240 or so...

ANYWAY, the issue should be solved by Swisscom, I can forward your email to the Swiss IPv6 Council and/or some Swisscom engineers (but unsure of the result of course)

Let's wait for an analysis from the higher level support engineers, and since this is a very easy to repeat issue, only if we don't get the appropriate level of response, we might escalate things outside of the "normal" chain of command...😜

Meanwhile, I can access the IPv6 version of the website after having manually lowered the MTU of my Mac's interface to 1420.

Bye, Luca

lucaberta

@lucaberta wrote:

Meanwhile, I have manually configured to MTU to 1420 on my macOS machine, and I can now access the IPv6 Swisscom website fine.

I will experiment with other settings too.

just tried with MTU of 1472, and it worked too. Anything above it doesn't work, like it was mentioned before.

lucaberta

IPv6swisscom

I just have one short final reflection on this issue, which came out by re-reading again all the thread and linked messages.

IPv6@swisscom wrote:

Now, for a problem to occur, two things must happen:

The Router has to incorrectly advertise a too large MTU to the client

The Firewall in front of the web server has to drop inbound Packet Too Big messages

[...]

That's why it worked when @lucaberta changed the link MTU directly on the end system.

I've asked the firewall in front of the www.swisscom.com web server to be corrected. Rest assured that the IPv6 backbone of Swisscom does not filter ICMPv6 Packet Too Big messages. Eric would be telling me off if we did that, and rightfully so 🙂

clearly point #2 has been addressed correctly by IPv6@swisscom and I look forward to the changes being implemented on the firewalls protecting the webservers.

Yet, point #1 falls on to each user's lap, and my reflection follows.

I have used the standard Swisscom-provided CPE for years, since 2014, and it was an InternetBox Plus, which worked quite well also as a 6RD tunnel endpoint for my always-successful connection to the IPv4 and IPv6 internet.

I never had a single problem with the swisscom.ch/.com website, in spite of the issue mentioned at point #2, simply because the radvd daemon running on the ISP-provided CPE was correctly configured to announce an MTU of 1472.

This broke when, two weeks ago, I decided to change CPE and bought a new Zyxel XMG3927-B50A router, which is listed on the BBCS list as an approved CPE by Swisscom for their wholesale service:

E_BBCS_Supporting-Document_Proved-Equipment

(see page 4 almost at the bottom, where the Zyxel XMG3927-B50A is listed)

I have documented the setup of the box which was quite easy for someone as geeky as me, including the DHCP option 60 and 6RD configuration, in this other thread on this community:

Report on good VDSL2 experience with Zyxel XMG3927...

What I was missing is the fact that the advertised MTU on the Zyxel's implementation of radvd *CANNOT* be changed from the GUI, and most likely it defaults to the LAN MTU since I cannot find any indication of an MTU advertisment done by the Zyxel radvd implementation:

radvd packet trace.png

So in the end it was the router change, and the inability to announce a smaller MTU by the Zyxel router, that created this whole situation.

Had I stayed with the Swisscom-provided CPE, such as the new IB3, I would never have had the issue.

And the Swisscom firewalls would NOT have been fixed, like they should... 😜👍🏻💪🏻

Thanks everyone, this was a most enjoyable group troubleshooting experience, and I am convinced that we all gained a lot from it, and many users will too as they will access the IPv6 versions of the Swisscom website, without knowing what happened behind the scene!

Ciao, Luca

Nächste Seite »