Subject: IPv6 Anycast has been killed by LINUX patch in 2016 - who cares?

From: Vasilenko Eduard <[email-masked]>
Date: Tue, 3 Aug 2021 09:44:28 +0000

Hi all,
I am writing to this alias because I do not know the proper one for such type of a problem (OS/LINUX/BSD).
The history of how Alexander Azimov (Yandex) has found the problem is below.

In short: if TCP loses connectivity for 200ms (or 1s in SYN stage) then TCP changes IPv6 flow label (for the active TCP session!) to push traffic to a different path.
Current networks are extensively ECMP, if intermediate nodes support flow label for hash calculation then a high probability that the path would be changed.
LINUX/BSD does not want to wait till the network will fix its problem.

If the final destination was anycast then the final destination would be changed too by the same hash calculation.
The stateful session would be broken as a result (see the second part of Alexander’s presentation below).

Since the time LINUX has made the default RTO flow label recalculation (2016), IPv6 Anycast is broken.
People would have one more reason not to migrate to IPv6. Flow label does not exist in IPv4 – OS is not capable to break IPv4 Anycast similarly.

Is anybody would like to spend his/her karma to save IPv6 Anycast OR let it die?
It is broken already for 5 years and nobody has spotted it up to now. Is it needed?
(I have seen a few drafts heavily dependent on IPv6 anycast)

What is proper WG for such a problem?

I am concerned that Anycast has been killed, it is not an easily replaceable tool.
Maybe somebody would propose something better but if not
then LINUX should be returned to 2015 when flow label change on RTO was a non-default configuration.
Such LINUX behavior could be valuable in some restricted domains (see below) when the administrator is sure that Anycast is not possible on the traffic path.

Eduard
From: Vasilenko Eduard
Sent: Tuesday, August 3, 2021 12:05 PM
To: 'Jeff Tantsura' <[email-masked]>; Alexander Azimov <[email-masked]>
Cc: Alexander Azimov <[email-masked]>; routing WG <[email-masked]>
Subject: RE: Self-healing Networking with Flow Label

Hi all,
Not many people worldwide read this alias and understand
That RTO could be leveraged to fight “silent drops” in the DC environment.
It is a good use case to publish/document (with more details that it was in the presentation).
I hope that in the future OAM would be used for this purpose – it is better from architecture point of view.
Eduard
From: Jeff Tantsura [mailto:[email-masked]]
Sent: Tuesday, August 3, 2021 1:08 AM
To: Alexander Azimov <[email-masked]<mailto:[email-masked]>>
Cc: Vasilenko Eduard <[email-masked]<mailto:[email-masked]>>; Alexander Azimov <[email-masked]<mailto:[email-masked]>>; routing WG <[email-masked]<mailto:[email-masked]>>
Subject: Re: Self-healing Networking with Flow Label

Eduard,

The idea of the draft to come is to explain what to do - when and how.
The goal is not to regulate (we really don’t) but to provide, similarly to RFC7938 a set of guidelines that community can use to build better and more resilient networks.

Cheers,
Jeff

On Aug 2, 2021, at 04:01, Alexander Azimov <[email-masked]<mailto:[email-masked]>> wrote:

Eduard,

пн, 2 авг. 2021 г. в 13:45, Vasilenko Eduard <[email-masked]<mailto:[email-masked]>>:
It is the key in this presentation “This behavior MUST be switched off by default”
It has been shown on slides 7-10 that flow label change on RTO is enabled by default for BSD and LINUX.
It needs regulation. It needs a standard RFC. Because it kills Anycast otherwise.
As I'm partially responsible for the key points of the presentation, I can stress that it is a bit different.

  *   We have an opportunity for self-healing TCP on top of IPv6, it should be preserved;
  *   The Linux defaults should be changed to a safe mode to prevent session timeouts;
  *   The hash recalculation behavior should be documented;
I'm not sure what you mean by the term 'regulation'.

The story of how to use RTO to work-around “silent drop” vendor’s bugs could be a good informational RFC.
My be people developing iOAM would pay more attention to this use case.

IMHO: these are 2 separate drafts.
I'm not sure about it, we'll try to provide -00 before the next IETF meeting, let's see how it progresses.

Eduard
From: Alexander Azimov [mailto:[email-masked]<mailto:[email-masked]>]
Sent: Monday, August 2, 2021 1:20 PM
To: Vasilenko Eduard <[email-masked]<mailto:[email-masked]>>; Jeff Tantsura <[email-masked]<mailto:[email-masked]>>
Cc: routing WG <[email-masked]<mailto:[email-masked]>>
Subject: Re: Self-healing Networking with Flow Label

Eduard,

Please see the quote from the slide 28. My suggestion was:

Client – sends SYN, Server – responds with SYN&ACK

  *   In case of SYN_RTO or RTO events Server SHOULD recalculate its TCP socket hash, thus change Flow Label. This behavior MAY be switched on by default;
  *   In case of SYN_RTO or RTO events Client MAY recalculate its TCP socket hash, thus change Flow Label. This behavior MUST be switched off by default;
This looks like a safe default behavior, that saves the part of the improvements, but makes the work with stateful anycast services safe.

And yes, IMO it's ok to have a knob to enable it in the controlled environment. If you ask how to enable it in the presence of internal anycast services - there was also a suggestion in the slides: eBPF. It gives a good way to make this kind of separation.

02.08.2021, 11:48, "Vasilenko Eduard" <[email-masked]<mailto:[email-masked]>>:

Hi Jeff,

The situation when Control Plane does not understand what the Forwarding pane doing is a bug.

Yes, RTO in TCP helps to find a work-around for this bug. And yes, Anycast is typically absent inside DC – it does not create the problem in the DC environment.

But the same LINUX is used outside DC. RTO Flow Label change here would create even more problems if Anycast would happen on the traffic path (not much predictable for client).

Do we need separate LINUX distribution for DC and separate distribution for other environments?

Or should we rely on the proper non-default configuration for different environments? (Admin should not forget to change)

What if Anycast would become needed in DC?

RTO flow label recalculation is mutually exclusive with Anycast on the traffic part.

What is more valuable for the public?

IMHO: It is better to fight the problem of such type of a bug with iOAM than to cancel Anycast.

IMHO: It is better to have Flow Label recalculation on RTO as “off” by default.

DC Admin has the higher qualification to activate it in a controlled environment than every client worldwide that should not forget to disable it.

Eduard

From: Jeff Tantsura [mailto:[email-masked]]
Sent: Monday, August 2, 2021 6:56 AM
To: Vasilenko Eduard <[email-masked]<mailto:[email-masked]>>
Cc: [email-masked]<mailto:[email-masked]>; routing WG <[email-masked]<mailto:[email-masked]>>
Subject: Re: Self-healing Networking with Flow Label

Eduard,

The issue is present not in link/device case, if well implemented - fast rehash takes care of updating forwarding within a number of ms. The problem is with  “gray” failures,  when the link in question is UP from routing/forwarding prospective but drops traffic (mostly occasionally and when a HW bug occurs has a distinct flow attributes).

In many large DC fabrics, the majority of the traffic is east-west, between end-points that aren’t anycast. In such deployments - the solution solves  issues rather elegantly and without any interventions from the operator.

The issues/side effects are well understood and will be documented.

The best way to receive RTGWG emails is well… to subscribe to RTGWG ;-)

Cheers,

Jeff

On Aug 1, 2021, at 09:47, Vasilenko Eduard <[email-masked]<mailto:[email-masked]>> wrote:

Hi  Alexander,

Have I understood your presentation right?

The client SHOULD change IPv6 flow label after SYN RTO to have a chance to be moved to the working path inside DC fabric (if DC fabric supports flow label for hash calculation)

But at the same time

The client SHOULD NOT change the IPv6 flow label after SYN RTO to avoid being switched to a different TCP proxy engine.

Looks like a deadlock, especially if both things should happen for the same traffic:

it should reach DC fabric

and

it should be hash load-balanced between different TCP proxy engines (or applications) inside DC Fabric.

I see one bad solution (“Disable Flow Label”):

Routers up to TCP proxy engine SHOULD be configured not to use flow label (by the way these are all routers on the Internet),

TCP flow engines SHOULD be outside of the DC Fabric (CLOS) – probably in front of it.

Routers/Switches inside DC Fabric SHOULD use flow labels.

I see another bad solution (“Disable Anycast”):

Disable anycast on routers in principle, use only stateful LB.

It has been commented in the chat that Anycast is not possible in principle for stateful connection. It is too general a statement.

Anycast is just not compatible with Flow Label. It is not a problem for IPv4 anycast even if the connection is stateful (TCP) because 5-tuple for hash would not change.

Hence, IPv6 anycast has become dead at the time when Flow Label change has been added in LINUX for active TCP session.

Among 3 thins:

-          Anycast

-          Flow Label load balancing (basic Flow Label functionality)

-          Flow Label change on the active session for application to be more active in new path search

You have to choose which one to kill – all 3 are not compatible with each other at the same.

I vote to disable Flow Label change in LINUX. Then wait till the network would fix itself.

We have so many fancy TE tools our days. A broken link or a broken node could be excluded from routing for 50ms.

PS: I am not subscribed to the RTGWG alias, please keep me on a copy of this thread.

Best Regards

Eduard Vasilenko

Senior Architect

Europe Standardization & Industry Development Department

Tel: +7(985) 910-1105, +7(916) 800-5506

_______________________________________________
rtgwg mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/rtgwg

--
Best regards,
Alexander Azimov

_______________________________________________
rtgwg mailing list
[email-masked]<mailto:[email-masked]>
https://www.ietf.org/mailman/listinfo/rtgwg

--
Best regards,
Alexander Azimov

---

From: Michael Tuexen <[email-masked]>
Date: Tue, 3 Aug 2021 12:09:37 +0200

> On 3. Aug 2021, at 11:44, Vasilenko Eduard <[email-masked]> wrote:
> 
> Hi all,
As far as I know, Linux implements something you describe, but I'm not aware on this behaviour being
implemented in *BSD, at least not in FreeBSD.
> What is proper WG for such a problem?
At IETF 110 Alexander gave a presentation on this in TCPM and V6OPS. See the Minutes and the corresponding slides at
https://datatracker.ietf.org/meeting/110/proceedings

At least at the TCPM meeting, it was suggested that an ID would be written.

However, the behaviour you are describing, is implementation specific to Linux, this is not described or
recommended by an RFC.

Best regards
Michael
> To: Alexander Azimov <[email-masked]>
> Cc: Vasilenko Eduard <[email-masked]>; Alexander Azimov <[email-masked]>; routing WG <[email-masked]>
> On Aug 2, 2021, at 04:01, Alexander Azimov <[email-masked]> wrote:
> 
> 
> Eduard, 
>  
> пн, 2 авг. 2021 г. в 13:45, Vasilenko Eduard <[email-masked]>:
> 	• We have an opportunity for self-healing TCP on top of IPv6, it should be preserved;
> 	• The Linux defaults should be changed to a safe mode to prevent session timeouts;
> 	• The hash recalculation behavior should be documented;
> From: Alexander Azimov [mailto:[email-masked]] 
> To: Vasilenko Eduard <[email-masked]>; Jeff Tantsura <[email-masked]>
> Cc: routing WG <[email-masked]>
> 	• In case of SYN_RTO or RTO events Server SHOULD recalculate its TCP socket hash, thus change Flow Label. This behavior MAY be switched on by default;
> 	• In case of SYN_RTO or RTO events Client MAY recalculate its TCP socket hash, thus change Flow Label. This behavior MUST be switched off by default;
> 02.08.2021, 11:48, "Vasilenko Eduard" <[email-masked]>:
> Hi Jeff,
> To: Vasilenko Eduard <[email-masked]>
> Cc: [email-masked]; routing WG <[email-masked]>
> On Aug 1, 2021, at 09:47, Vasilenko Eduard <[email-masked]> wrote:
> 
> 
> Hi  Alexander,
>  
> [email-masked]
> [email-masked]

---

From: Brian Carpenter <[email-masked]>
Date: Tue, 3 Aug 2021 22:45:29 +1200

The issue of anycast and unstable routes is hardly a new discovery; this
Linux feature is not creating a new problem. I suggest reading RFC7094 and
RFC4786 before continuing this conversation.

I certainly wouldn't design a protocol that relied on stable transport
connections to an anycast address.

Regards,
    Brian Carpenter
    (via tiny screen & keyboard)

On Tue, 3 Aug 2021, 22:10 Michael Tuexen, <[email-masked]>
wrote:

> > In short: if TCP loses connectivity for 200ms (or 1s in SYN stage) then
> TCP changes IPv6 flow label (for the active TCP session!) to push traffic
> to a different path.
> > Current networks are extensively ECMP, if intermediate nodes support
> flow label for hash calculation then a high probability that the path would
> be changed.
> > If the final destination was anycast then the final destination would be
> changed too by the same hash calculation.
> > People would have one more reason not to migrate to IPv6. Flow label
> does not exist in IPv4 – OS is not capable to break IPv4 Anycast similarly.
> >
> At IETF 110 Alexander gave a presentation on this in TCPM and V6OPS. See
> the Minutes and the corresponding slides at
> > Such LINUX behavior could be valuable in some restricted domains (see
> below) when the administrator is sure that Anycast is not possible on the
> traffic path.
> >
> > Eduard
> > From: Vasilenko Eduard
> > Cc: Vasilenko Eduard <[email-masked]>; Alexander Azimov <
> [email-masked]>; routing WG <[email-masked]>
> > The goal is not to regulate (we really don’t) but to provide, similarly
> to RFC7938 a set of guidelines that community can use to build better and
> more resilient networks.
> >
> > Cheers,
> > Jeff
> >
> >
> >       • In case of SYN_RTO or RTO events Server SHOULD recalculate its
> TCP socket hash, thus change Flow Label. This behavior MAY be switched on
> by default;
> >       • In case of SYN_RTO or RTO events Client MAY recalculate its TCP
> socket hash, thus change Flow Label. This behavior MUST be switched off by
> default;
> > This looks like a safe default behavior, that saves the part of the
> improvements, but makes the work with stateful anycast services safe.
> >
> > And yes, IMO it's ok to have a knob to enable it in the controlled
> environment. If you ask how to enable it in the presence of internal
> anycast services - there was also a suggestion in the slides: eBPF. It
> gives a good way to make this kind of separation.
> >
> > Yes, RTO in TCP helps to find a work-around for this bug. And yes,
> Anycast is typically absent inside DC – it does not create the problem in
> the DC environment.
> >
> > But the same LINUX is used outside DC. RTO Flow Label change here would
> create even more problems if Anycast would happen on the traffic path (not
> much predictable for client).
> > Or should we rely on the proper non-default configuration for different
> environments? (Admin should not forget to change)
> > DC Admin has the higher qualification to activate it in a controlled
> environment than every client worldwide that should not forget to disable
> it.
> >
> > Eduard
> > The issue is present not in link/device case, if well implemented - fast
> rehash takes care of updating forwarding within a number of ms. The problem
> is with  “gray” failures,  when the link in question is UP from
> routing/forwarding prospective but drops traffic (mostly occasionally and
> when a HW bug occurs has a distinct flow attributes).
> >
> > In many large DC fabrics, the majority of the traffic is east-west,
> between end-points that aren’t anycast. In such deployments - the solution
> solves  issues rather elegantly and without any interventions from the
> operator.
> > The client SHOULD change IPv6 flow label after SYN RTO to have a chance
> to be moved to the working path inside DC fabric (if DC fabric supports
> flow label for hash calculation)
> > But at the same time
> > The client SHOULD NOT change the IPv6 flow label after SYN RTO to avoid
> being switched to a different TCP proxy engine.
> >
> > Routers up to TCP proxy engine SHOULD be configured not to use flow
> label (by the way these are all routers on the Internet),
> > It has been commented in the chat that Anycast is not possible in
> principle for stateful connection. It is too general a statement.
> > Anycast is just not compatible with Flow Label. It is not a problem for
> IPv4 anycast even if the connection is stateful (TCP) because 5-tuple for
> hash would not change.
> > Hence, IPv6 anycast has become dead at the time when Flow Label change
> has been added in LINUX for active TCP session.
> >
> > Among 3 thins:
> > -          Anycast

---

From: Christian Huitema <[email-masked]>
Date: Tue, 3 Aug 2021 08:45:31 -0700

The anycast connection stability issue is addressed in QUIC. During the 
initial handshake, QUIC servers can provide a "preferred address" (IP or 
IPv6 and port number), and the client can immediately migrate the 
connection to use that address. This was specifically designed to 
support servers listening on anycast addresses, so the connection can be 
pinned to an unicast address specific to the server.

-- Christian Huitema

---

From: Töma Gavrichenkov <[email-masked]>
Date: Tue, 3 Aug 2021 19:31:45 +0300

Peace,

>From the operational standpoint, this only addresses one particular
scenario where anycast ensures latency reduction (think CDNs).  However, in
the today's Internet, the unicast nodes most adjacent to the client are
already advertised through DNS records, which also saves a handshake if
compared to the solution QUIC offers.

There are use cases where unicast is not fit, and the server address must
be kept anycast at all times.

--
Töma

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Tue, 3 Aug 2021 13:53:52 -0400

Does ANYCAST UDP work?

It would not make me in the least bit sad if ANYCAST TCP doesn't work
because I don't think it should.

ANYCAST for DNS and other naming/discovery schemes makes perfect sense. If
your discovery scheme requires more than one request packet, it is probably
not going to work very well with ANYCAST, you certainly have no reason to
expect it to work at all over ANYCAST. Which in my view makes it a bad
discovery scheme.

Since I am always going to do discovery before my application, I see
absolutely no reason I would use TCP over multicast. Even if I decide DNS
discovery isn't discovering enough, I am going to be making an
alternative/supplemental discovery, to a geo-location or network-location
sensitive service then doing my application.

TCP over ANYCAST is crossing the streams. Not surprised it doesn't work and
see no reason to change that.

On Tue, Aug 3, 2021 at 5:45 AM Vasilenko Eduard <[email-masked]>
wrote:

> Current networks are extensively ECMP, if intermediate nodes support flow
> label for hash calculation then a high probability that the path would be
> changed.
>
> People would have one more reason not to migrate to IPv6. Flow label does
> not exist in IPv4 – OS is not capable to break IPv4 Anycast similarly.
>
>
>
> *From:* Vasilenko Eduard
> *Sent:* Tuesday, August 3, 2021 12:05 PM
> *To:* 'Jeff Tantsura' <[email-masked]>; Alexander Azimov <
> [email-masked]>
> *Cc:* Alexander Azimov <[email-masked]>; routing WG <
> [email-masked]>
> *Subject:* RE: Self-healing Networking with Flow Label
>
>
>
> Hi all,
>
> *From:* Jeff Tantsura [mailto:[email-masked]
> <[email-masked]>]
> *Sent:* Tuesday, August 3, 2021 1:08 AM
> *To:* Alexander Azimov <[email-masked]>
> *Cc:* Vasilenko Eduard <[email-masked]>; Alexander Azimov <
> *Subject:* Re: Self-healing Networking with Flow Label
>
>
>
> Eduard,
>
>
>
> The goal is not to regulate (we really don’t) but to provide, similarly to
> RFC7938 a set of guidelines that community can use to build better and more
> resilient networks.
>
>
>
> Cheers,
>
> Jeff
>
>
>
> пн, 2 авг. 2021 г. в 13:45, Vasilenko Eduard <[email-masked]
> >:
>
>    - We have an opportunity for self-healing TCP on top of IPv6, it
>    should be preserved;
>    - The Linux defaults should be changed to a safe mode to prevent
>    session timeouts;
>    - The hash recalculation behavior should be documented;
>
> *From:* Alexander Azimov [mailto:[email-masked]]
> *Sent:* Monday, August 2, 2021 1:20 PM
> *To:* Vasilenko Eduard <[email-masked]>; Jeff Tantsura <
> [email-masked]>
> *Cc:* routing WG <[email-masked]>
> *Subject:* Re: Self-healing Networking with Flow Label
>
>
>
> Eduard,
>
>
>
>    - In case of SYN_RTO or RTO events Server SHOULD recalculate its TCP
>    socket hash, thus change Flow Label. This behavior MAY be switched on by
>    default;
>    - In case of SYN_RTO or RTO events Client MAY recalculate its TCP
> Yes, RTO in TCP helps to find a work-around for this bug. And yes, Anycast
> is typically absent inside DC – it does not create the problem in the DC
> environment.
>
>
>
> *From:* Jeff Tantsura [mailto:[email-masked]
> <[email-masked]>]
> *Sent:* Monday, August 2, 2021 6:56 AM
> *To:* Vasilenko Eduard <[email-masked]>
> *Cc:* [email-masked]; routing WG <[email-masked]>
> *Subject:* Re: Self-healing Networking with Flow Label
>
>
>
> Eduard,
>
>
>
> The client SHOULD change IPv6 flow label after SYN RTO to have a chance to
> be moved to the working path inside DC fabric (if DC fabric supports flow
> label for hash calculation)
>
> But at the same time
>
> Routers up to TCP proxy engine SHOULD be configured not to use flow label
> (by the way these are all routers on the Internet),
>
> Hence, IPv6 anycast has become dead at the time when Flow Label change has
> been added in LINUX for active TCP session.
>
>
>
> Among 3 thins:
>
> -          Anycast
>

---

From: Töma Gavrichenkov <[email-masked]>
Date: Tue, 3 Aug 2021 21:55:47 +0300

Peace,

On Tue, Aug 3, 2021, 8:54 PM Phillip Hallam-Baker <[email-masked]>
wrote:

> TCP over ANYCAST is crossing the streams. Not surprised it doesn't work
> and see no reason to change that.
>

Does that imply that you believe no TCP-based protocol deserves protection
from DDoS attacks?  Because anycast is ultimately the one and the only
basement for that protection.

--
Töma

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Tue, 3 Aug 2021 20:00:14 -0400

> basement for that protection.

?? I had better get me to a patent lawyer then because I have multiple DDoS
protection ideas and none involve ANYCAST and only a few TCP.

---

From: Jeff Tantsura <[email-masked]>
Date: Tue, 3 Aug 2021 17:08:10 -0700

Eduard,

The reports about IPv6 anycast death are greatly exaggerated ;-)

The issue is well know, e.g. - IPv6 flow label: misuse in hashing | APNIC
Blog <https://blog.apnic.net/2018/01/11/ipv6-flow-label-misuse-hashing/>

Don't forget that flow hashing is a local decision that could be influenced
by configuration/capabilities.Relying on constant (sticky) hashing is
unhealthy and error prone, when other keys, such as index-ingress-port are
used, this is completely unpredictable.
Flow label based hashing is default by default on/off as per vendor/release.
Stateless anycast is not affected by flow label changes as well as
statefull services that use OOB sync (think public cloud).
Interesting point - this could affect IPv4 in host based overlays cases.
e.g. the value of UDP source port in VXLAN/Geneve header. (skb->hash used
for UDP source port)

---

From: Töma Gavrichenkov <[email-masked]>
Date: Wed, 4 Aug 2021 07:55:26 +0300

Peace,

On Wed, Aug 4, 2021, 3:00 AM Phillip Hallam-Baker <[email-masked]>
wrote:

>> Does that imply that you believe no TCP-based protocol deserves
>> protection from DDoS attacks?  Because anycast is ultimately the one and
>> the only basement for that protection.
>>
>
> ?? I had better get me to a patent lawyer then because I have multiple
> DDoS protection ideas and none involve ANYCAST and only a few TCP.
>

Either that, or you might want to test your ideas against the specialized
field expertise.

Anycast has been the ultimate DDoS mitigation tool for a decade already,
and for a reason.  Basically, it all comes down to a simple idea: DDoS
traffic, generated in thousands of locations on the globe, cannot possibly
be handled when accumulated in one of such locations.  But it's surely more
complicated than that.

Either you have multiple traffic termination points on the net (a.k.a.
anycast), each as close to some traffic generation point as possible, or
you'll end up having capacity overload around your last mile.  This is
fundamental, kind of.

--
Töma

---

From: Vasilenko Eduard <[email-masked]>
Date: Wed, 4 Aug 2021 08:33:08 +0000

Hi Jeff,
Thanks. The article is good. But the reference of Brian to RFC 7094 is better, because it is RFC (more formal). No one should assume that Flow Label would stay constant.
Hence, FlowLabel is a misleading name. It is the ”random changing seed” in reality that is enough for load equalization on the ECMP.
It could abruptly change latency by jumping to the different path.
Eduard
From: ietf [mailto:[email-masked]] On Behalf Of Jeff Tantsura
Sent: Wednesday, August 4, 2021 3:08 AM
To: Töma Gavrichenkov <[email-masked]>
Cc: Phillip Hallam-Baker <[email-masked]>; IETF Rinse Repeat <[email-masked]>
Subject: Re: IPv6 Anycast has been killed by LINUX patch in 2016 - who cares?

Eduard,

The issue is well know, e.g. - IPv6 flow label: misuse in hashing | APNIC Blog<https://blog.apnic.net/2018/01/11/ipv6-flow-label-misuse-hashing/>

Don't forget that flow hashing is a local decision that could be influenced by configuration/capabilities.Relying on constant (sticky) hashing is unhealthy and error prone, when other keys, such as index-ingress-port are used, this is completely unpredictable.
Flow label based hashing is default by default on/off as per vendor/release.
Stateless anycast is not affected by flow label changes as well as statefull services that use OOB sync (think public cloud).
Interesting point - this could affect IPv4 in host based overlays cases. e.g. the value of UDP source port in VXLAN/Geneve header. (skb->hash used for UDP source port)

On Tue, Aug 3, 2021 at 11:56 AM Töma Gavrichenkov <[email-masked]<mailto:[email-masked]>> wrote:
Peace,
On Tue, Aug 3, 2021, 8:54 PM Phillip Hallam-Baker <[email-masked]<mailto:[email-masked]>> wrote:
It would not make me in the least bit sad if ANYCAST TCP doesn't work because I don't think it should.

TCP over ANYCAST is crossing the streams. Not surprised it doesn't work and see no reason to change that.

Does that imply that you believe no TCP-based protocol deserves protection from DDoS attacks?  Because anycast is ultimately the one and the only basement for that protection.

--
Töma

---

From: "Salz, Rich" <[email-masked]>
Date: Wed, 4 Aug 2021 14:35:14 +0000

*   Anycast has been the ultimate DDoS mitigation tool for a decade already

It’s certainly *a* tool.  Ultimate is in the eyes of the beholder.  My employer doesn’t use it. As far as I can tell, Google’s protections don’t either.

---

From: Töma Gavrichenkov <[email-masked]>
Date: Wed, 4 Aug 2021 17:37:38 +0300

Peace,

On Wed, Aug 4, 2021 at 5:35 PM Salz, Rich <[email-masked]> wrote:
>> Anycast has been the ultimate DDoS mitigation tool for a decade already
> It’s certainly *a* tool.  Ultimate is in the eyes of the beholder.
> My employer doesn’t use it.

Well it does.
https://www.akamai.com/us/en/multimedia/documents/product-brief/prolexic-proxy-product-brief.pdf

--
Töma

---

From: "Salz, Rich" <[email-masked]>
Date: Wed, 4 Aug 2021 14:49:16 +0000

Oops.

On 8/4/21, 10:37 AM, "Töma Gavrichenkov" <[email-masked]> wrote:

    Peace,

    Well it does.

    --
    Töma

---

From: Warren Kumari <[email-masked]>
Date: Wed, 4 Aug 2021 10:50:04 -0400

> https://www.akamai.com/us/en/multimedia/documents/product-brief/prolexic-proxy-product-brief.pdf
... as do a large number of (TCP) CDNs and content providers.

The CacheFly writeup is one of the better descriptions -
https://www.cachefly.com/anycast-think-before-you-talk-part-i/
and this LinkedIn post is also not bad:
https://www.linkedin.com/pulse/build-your-own-anycast-network-9-steps-samir-jafferali/
and
https://www.slideshare.net/shawnzandi/how-linkedin-used-tcp-anycast-to-make-the-site-faster

To be honest I wasn't going to get involved in this thread due to the tone
of some of the messages, but...

W

-- 
The computing scientist’s main challenge is not to get confused by the
complexities of his own making.
  -- E. W. Dijkstra

---

From: Christopher Morrow <[email-masked]>
Date: Wed, 4 Aug 2021 11:58:08 -0400

> of some of the messages, but...
also, also, isn't the 'ipv6 anycast is ded in the linhux colonel' really
about the anycast address per prefix in ipv6 and not about bgp / igp
anycast?

---

From: "Theodore Ts'o" <[email-masked]>
Date: Wed, 4 Aug 2021 12:52:54 -0400

> be handled when accumulated in one of such locations.  But it's surely more

If you have an established TCP flow, and there is a DDOS which kills a
particular endpoint, replicating all TCP state across dozens of
location across the globe for hundreds of thousands of TCP connections
per second, such that an Anycast endpoint could be redirected from
say, Hong Kong to Singapore or Australia, is simply not practical.

Quoting RFC 7094:

   Most stateful transport protocols (e.g., TCP), without modification,
   do not understand the properties of anycast; hence, they will fail
   probabilistically, but possibly catastrophically, when using anycast
   addresses in the presence of "normal" routing dynamics.
   Specifically, if datagrams associated with a given active transaction
   are routed to a new anycasted end system and that end system lacks
   state data associated with the active transaction, the session will
   be reset; hence, it will need to be reinitiated.

> you'll end up having capacity overload around your last mile.  This is

Sure, and using UDP with Anycast (DNS, QUIC) works quite well in DDOS
solutions.  But the fact that *any* stateful protocol (such as TCP) is
not compatible with Anycast across the global internet without making
modifications is *also* fundamental.

The changes in Linux five years ago may have made certain broken
architecures more likely to fail.  The reason why no some seems to
care is because those were broken designs in the first place, which
RFC-7094 warns against, and there are plenty companies of which are
actively solving the DDOS problem for thier customers --- with many of
these companies using modern Linux kernels and Anycast, without the
problems cited by the original poster in this thread.

						- Ted

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Wed, 4 Aug 2021 13:28:43 -0400

> you'll end up having capacity overload around your last mile.  This is

The flaw in your argument is 'multiple traffic termination points on the
net (a.k.a. anycast)'

Of course if you call 'multiple traffic termination' anycast, this is true
but you are then taking the leap to TCP over ANYCAST is the only way to do
things which is false.

It is time to abandon TCP entirely and not just because of QUIC. Once one
alternative transport is allowed in, there will be others.

In my RUD transport, the source address, port and protocol are simply
discarded. It is entirely valid to send three packets via HTTP and the next
five over UDP.

So yes, I can use ANYCAST techniques to conceal the internals of my system.
But I am not obliged to. And I certainly don't depend on ANYCAST working
over TCP.

---

From: Töma Gavrichenkov <[email-masked]>
Date: Wed, 4 Aug 2021 21:06:35 +0300

Peace,

> say, Hong Kong to Singapore or Australia, is simply not practical.

This is pretty specific use case, it's not expected to occur
frequently.  In this case, it has always been easier to reset the
existing connections and establish new ones.  This results in some
negotiation overhead and some delays in transmission, but you must
expect that once a part of your infrastructure suddenly died.

Things like that should never happen under a DDoS attack, because you
risk getting a chain reaction over your PoPs otherwise.

>    addresses in the presence of "normal" routing dynamics.

Yep, and you need to account for that in your application
architecture.  This is what RFC 7094 is about.  This is fine.

This is not a death sentence to TCP over anycast.

> RFC-7094 warns against

Surely there are several ways to read 7094!

> and there are plenty companies of which are

This experience might have something to do with the poor state of IPv6
adoption, though.

--
Töma

---

From: Töma Gavrichenkov <[email-masked]>
Date: Wed, 4 Aug 2021 21:09:39 +0300

Peace,

On Wed, Aug 4, 2021 at 8:28 PM Phillip Hallam-Baker
<[email-masked]> wrote:
> The flaw in your argument is 'multiple traffic termination points on the net (a.k.a. anycast)'
> Of course if you call 'multiple traffic termination' anycast, this is true but you are then taking the leap to TCP over ANYCAST is the only way to do things which is false.

There are applications and protocols already built on top of TCP.  I
believe we're using some of these right now to communicate with each
other.
Are you saying here that such applications and protocols do not
deserve to get DDoS protection?

--
Töma

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Wed, 4 Aug 2021 17:11:25 -0400

I say that people who assume the spec does things it does not and then
build a business model based on that assumption deserve it to fail horribly
and painfully.

On Wed, Aug 4, 2021 at 2:09 PM Töma Gavrichenkov <[email-masked]> wrote:

> > Of course if you call 'multiple traffic termination' anycast, this is
> true but you are then taking the leap to TCP over ANYCAST is the only way
> to do things which is false.
>

---

From: Töma Gavrichenkov <[email-masked]>
Date: Thu, 5 Aug 2021 01:09:04 +0300

Peace,

> build a business model based on that assumption deserve it to fail horribly

Well, in this conceptual model, the entire Web 1.0 and Web 2.0 business
ecosystem was born upon the wrong assumption that the HTTP protocol (and
all other protocols with long-living connections in general, by the way)
could've provided business continuity with external threats in mind.

This is now the world we live in.  It's not perfect, nor that it ever
pretended to be, but companies within that ecosystem are worth trillions,
so I guess the initial assumption for all practical intents and purposes is
here to stay, would one like it or not.

--
Töma

---

From: "Templin (US), Fred L" <[email-masked]>
Date: Fri, 6 Aug 2021 20:13:18 +0000

Eduard, as you know AERO/OMNI do some things with both IPv4 and IPv6 Anycast which
can provide useful functions. Consideration of anycast in the specs is not perfect currently,
but while the specs are not currently subject to change I have captured suitable resolutions
in the issue tracker.

Fred

---

From: Toerless Eckert <[email-masked]>
Date: Sat, 7 Aug 2021 03:47:30 +0200

[bitching]
I apologize for attempting to respond to the original post topic instead
of derailing the thread into my pet side topic without changing subject,
which seems to be expected behavior on [email-masked].
[/bitching]

Adding [email-masked] as that seems to be the closest WG list for the topic.

Brian reminded us that we have ample RFCs to elaborate on the fact that you can not
reasonably expect for connections to an anycast address to work when persistently
using the anycast address. 

Christian pointed out how QUIC does the right thing. Great! Maybe we should
have a an anycast support hall of fame and shame for protocols: DOes or does it
not support single round-trip resolution of anycast to unicast address.

But back to what seems to be the root cause, which isn't anycast, but IPv6
flow label "abuse" ?!

I specifically had not heard of this Linux "hack" to change flow-label
mid-connection after TCP RTO to overcome a seemingly broken path and hope for 
the new flow-label to pick another, working path (most likely in a data cener).

I do not think that this endpoint behavior is endorsed by RFC6437 and by absence
of a description in RFC6437 under which circumstances an endpoint could or should
change the Flow label for an in-progress transport connection, i would conclude 
that this linux behavior is not in compliance with how the normative RFC6437
describes proper host assignment of flow label.

Could someone who sleeps with RFC6437 under his pillow comment on whether
or not my assessment is accurate ? I think to remember Bob mentioned he just
carefully read through all those flow label RFCs...

Of course, pointing out to linux that what it does is a hack would not make
TCP to an anycast address any less of a hack.

Ultimately, i have not a lot of sympathy for the linux behavior, even if it
was blessed by RFC6437, because i think good networks should fix broken paths
fast enough for this hack to be not necessary...

Cheers
    Toerless

-- 
---
[email-masked]

---

From: Mark Smith <[email-masked]>
Date: Sat, 7 Aug 2021 13:29:44 +1000

> using the anycast address.

Yes, even the first anycast RFC (RFC1546) recognised that.

> not support single round-trip resolution of anycast to unicast address.

MPTCP and SCTP could support that too:

https://tools.ietf.org/id/draft-smith-6man-form-func-anycast-addresses-01.html#rfc.section.5.7.7

> the new flow-label to pick another, working path (most likely in a data cener).

The flow label is supposed to be a hint from hosts to the network as
to what a flow is, so that the network can try to provide a better
service to the hosts beyond the minimum that is provided by
destination addresses based forwarding.

If these non-flow representing flow labels start making it harder for
network operators to manage traffic flows, the answer is easy and will
be inevitable. Network operators will ignore the flow label when doing
ECMP or LAG.

My perspective is both as a network operator, and as somebody who
helped with RFC6437 in the interests of getting those flow hints to be
able to provide a better network service.

<snip>

> fast enough for this hack to be not necessary...

Yes, and that is what a network and network operators' job is.

If hosts were to be properly involved in that then they'd need to
participate in IGPs and EGPs, and somehow attain knowledge of the
traffic management policies that network operators apply to IGPs and
EGPs.

Regards,
Mark.

> --------------------------------------------------------------------
> IETF IPv6 working group mailing list
> [email-masked]
> Administrative Requests: https://www.ietf.org/mailman/listinfo/ipv6
> --------------------------------------------------------------------

---

From: Masataka Ohta <[email-masked]>
Date: Sat, 7 Aug 2021 14:00:26 +0900

> using the anycast address.

But, the reality is that no practical protocols need persistent
stability.

> Christian pointed out how QUIC does the right thing.

Wrong. It only makes an anycast server victim of DDOS by the
alternative unicast address.

Remember how IPv6 was improved to be useless with so many
fancy ideas.

							Masataka Ohta

---

From: Jeff Tantsura <[email-masked]>
Date: Fri, 6 Aug 2021 22:13:53 -0700

It is easy to route around broken links, in a well designed DC we can reroute (really re-forward) within 50ms after the failure has been detected without FRR (aka fast-rehash).
Gray failures are a very different kind of animals, while you might think - how often do the optics degrade or fibers get dirty, at a large scale this is a constant problem, and it takes time (minutes if we are lucky) to detect and safely isolate.
While it might be not to someone’ taste  - it is a working solution for the aforementioned problem for large DC fabrics. 

Cheers,
Jeff

> On Aug 6, 2021, at 18:48, Toerless Eckert <[email-masked]> wrote:
> 
> [bitching]

---

From: Lloyd W <[email-masked]>
Date: Sat, 7 Aug 2021 16:04:44 +1000

> On 5 Aug 2021, at 00:51, Warren Kumari <[email-masked]> wrote:
> 
> .. as do a large number of (TCP) CDNs and content providers. 
> 
> The CacheFly writeup is one of the better descriptions - https://www.cachefly.com/anycast-think-before-you-talk-part-i/

well, apart from the broken link to part ii at the end of that article. (Why split an article into two parts?)

you can find part ii at
https://www.cachefly.com/anycast-think-before-you-talk-part-ii/

maybe contact@cachefly can fix the link at the end of part i.

L.

I find it hard to trust an authority on content provision when they have
broken links, but that's just me.

---

From: Mark Smith <[email-masked]>
Date: Sat, 7 Aug 2021 17:12:17 +1000

> While it might be not to someone’ taste  - it is a working solution for the aforementioned problem for large DC fabrics.

Linux isn't just used in DCs. IPv6 is a general purpose
internetworking protocol, not a DC fabric protocol. The Internet isn't
a single large DC fabric.

It seems working groups in the IETF think they can take IPv6 and
modify it to suit their special case purposes regardless of what the
6man WG think and have specified.

The SPRING working group have been allowed to specify EH insertion in
one of their RFCs despite it being objected to by the 6man WG. Rough
consensus in the SPRING WG on IPv6 EH processing is not rough
consensus from the 6man working group.

If the IESG and IETF approve this flow label behaviour, despite 6man
being the owners and specifiers of the Flow Label field, then the IETF
is a broken organisation.

---

From: Töma Gavrichenkov <[email-masked]>
Date: Sat, 7 Aug 2021 14:31:50 +0300

Peace,

> But, the reality is that no practical protocols need persistent

Exactly, and...

> alternative unicast address.

...precisely.

--
Töma

---

From: Robert Raszuk <[email-masked]>
Date: Sat, 7 Aug 2021 13:54:02 +0200

Hi Eduard,

I am not sure about the history but it seems that natural solution to the
issue you reported would be to allow app to set flow label.

It also seems there was such attempt 20 years ago, but I have no idea where
it ended up or why not it was progressed fwd:

https://tools.ietf.org/search/draft-itojun-ipv6-flowlabel-api-01

Thx
R.

---

From: Toerless Eckert <[email-masked]>
Date: Sat, 7 Aug 2021 17:17:16 +0200

Thanks, Mark, Jeff,

but let me rephrase my core question as its not answered, maybe
i wasn't asking it crisply enough or i didn't recognize the answer:

Do our RFCs say or imply anything about whether or not hosts can change
the IPv6 flow label field of a single flow during the lifetime
of a flow (MUST, SHOULD, MAY, ... MAY NOT, SHOULD NOT, MUST NOT)

Cheers
    Toerless

-- 
---
[email-masked]

---

From: Töma Gavrichenkov <[email-masked]>
Date: Sat, 7 Aug 2021 18:37:52 +0300

Peace,

> of a flow (MUST, SHOULD, MAY, ... MAY NOT, SHOULD NOT, MUST NOT)

Well, it is, of course, sort of implied that the flow label is attached to
the particular flow.

As per normative references, RFC 6437 goes (underscores are mine):

> It is therefore RECOMMENDED
> that source hosts support the flow label by setting the flow label
> field for _all_packets_of_a_given_flow_ to the _same_value_ chosen from
> an approximation to a discrete uniform distribution.

--
Töma

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Sat, 7 Aug 2021 13:32:40 -0400

Traffic schemes have a principle that once a right of way is established,
any competing mode of transport wanting to cross that right of way is
responsible for building the necessary bridges etc.

I raise this not to propose it as a principle for the Internet but rather
the opposite, to call it out as an anti-pattern.

If the IAB has an architectural role, it is maintaining the thinness of the
thin waist: Keeping complexity out of the Internet core is important and
requires vigilance.

The anycast over TCP discussion reminds me of a much older argument in the
industry when people who got used to hooking MSDOS system calls at
particular addresses were wont to scream like banshees each time a new
version of MSDOS came out with the entry points moved.

>From an architectural point of view, ANYCAST is not a thing, it is a hack.
A hack that is useful but one that was never guaranteed to support TCP. The
Internet architecture was designed to support UNICAST. The fact that
architecture allows us to share an IP address between hosts within a BGN
ASN is interesting and useful but not a core commitment by the
architecture. If people want to build stuff on that quicksand, so be it.
But those who do so have to understand they are building within very
limited constraints and there is no long term commitment to support any use
they might establish outside those constraints.

Sure, there will be people who have come up with some ingenious scheme that
allows systems to work because the deployed infrastructure currently
provides stronger guarantees. But fortunately the IETF does not have the
ability to insist such guarantees be continued so we really don't have to
debate whether we should do something we can't.

The cost of insisting that squatters rights be preserved would be a gradual
ossification of the core as every change is required to preserve the rights
of the owners of riparian rights.

As for QUIC, Christian is half right. Yes, QUIC does behave in a way that
is more compatible with ANYCAST. But QUIC was never designed to work in
that way and hence does not guarantee it will work that way.

Fortunately, QUIC is not the replacement for TCP, it is one tool that does
the same job and the most important single change in QUIC is that from this
point on every application can choose their own transport. The principle
that the platform providers are gatekeepers for transport provision has
been broken.

Censorship resistant protocols is a very longstanding interest of mine and
I would like to see innovation in that area. I really can't see much that
ANYCAST brings to the table beyond the discovery phase but Multicast
certainly interests me a lot.

There are three types of transport protocol are possible:

121) One to One
12M) One to Many
M2M) Many to Many

The last is of course the holy grail for some people, peer to peer goodness
and all that. As an applications guy, I have absolutely no interest in that
type of transport, reliable or not. Even when my application is peer to
peer.

A reliable transport with DDoS resistance properties supporting 121 and 12M
that could make use of multicast and hide the fact it had been used from
the application layer would be very very useful.

QUIC arrived at IETF as an optimization that Google was using and wanted to
standardize. Which is a completely valid mode of IETF work. But if we want
to do interesting stuff, QUIC is not the test bed to build upon. We need to
do research first and challenge the prior assumptions baked in before the
effort began.

PHB

---

From: Toerless Eckert <[email-masked]>
Date: Sat, 7 Aug 2021 19:54:10 +0200

Sure, that is the basic recommendation,
but let me play devils advocate (linux developer):

rfc6437: "However, a flow is not necessarily 1:1 mapped to a transport connection"

Aka: what linux does is a perfect lightweight form of
MP_TCP just without the overhead of different IP addresses
or port numbers to identify different flows. Just different
IPv6 flow labels. And in MP-TCP it would of course be perfectly
legitimate to use two flows sequentially, e.g.: when one flow
has RTO issues, using the second flow. Thats just a transport
protocol policy choice.

In other words: linux does not violate this SHOULD because it really
uses two different flows.

Cheers
    Toerless

-- 
---
[email-masked]

---

From: Töma Gavrichenkov <[email-masked]>
Date: Sat, 7 Aug 2021 21:39:47 +0300

Peace,

On Sat, Aug 7, 2021, 8:54 PM Toerless Eckert <[email-masked]> wrote:

> or port numbers to identify different flows.
>

Please correct me if I'm wrong, but this is exactly the opposite of what
MP-TCP is/was going to offer, which is the same flow, transferred via
multiple 5-tuples.

And here we have different flows over the same 5-tuple.

--
Töma

---

From: Tom Herbert <[email-masked]>
Date: Sat, 7 Aug 2021 12:41:18 -0700

> uses two different flows.

Correct, it's the host prerogative to set the flow as it sees fits.
Generally, we want the flow label to be consistent for the lifetime of the
flow, that is why the flow label is only changed at RTO when the connection
is failing. So this is really intended as a last ditch effort to salvage a
failing connection.

As for relying on the network to fix problems, the host cannot assume that.
The host has detected a problem with a connection's path, there is no
reason to believe that the networks, or networks, in the path will even
detect the problem much less fix it quickly. It would be nice if the host
had a standard means to report to the network that it sees a problem, but
that doesn't exist.

Also, as the subject states these patches have been in Linux and deployment
for 5 years now. The claim that it kills anycast seems overly melodramatic.
Is there any real data on the problem?

I'll also point out that any route change in the network could also have
the same effect. If anycast, or other stateful mechanisms, requires that
all packets of transport flow are routed consistently for the lifetime of a
flow then inevitably failures will occur. Consistent outing per flow is
only best effort and not a standard requirement.

This flow label modulation patches  were presented in routing area WG at
IETF 111. The intent is to document it and we are looking at mitigations
(the algorithm might be a little too aggressive for sending into the public
Internet).

Tom

---

From: Brian E Carpenter <[email-masked]>
Date: Sun, 8 Aug 2021 07:42:30 +1200

On 07-Aug-21 23:54, Robert Raszuk wrote:
> Hi Eduard,
> 
> I am not sure about the history but it seems that natural solution to the issue you reported would be to allow app to set flow label. 
> 
> It also seems there was such attempt 20 years ago, but I have no idea where it ended up or why not it was progressed fwd: 
> 
> https://tools.ietf.org/search/draft-itojun-ipv6-flowlabel-api-01 <https://tools.ietf.org/search/draft-itojun-ipv6-flowlabel-api-01>

It vanished somewhere in POSIX-land, in the same way as RFC3542. Very frustrating.

   Brian

> 
> Thx
> R.
> 
> 
> 
> 
> On Tue, Aug 3, 2021 at 11:45 AM Vasilenko Eduard <[email-masked] <mailto:[email-masked]>> wrote:
> 
>     Hi all,____
> 
>     I am writing to this alias because I do not know the proper one for 
such type of a problem (OS/LINUX/BSD).____
> 
>     The history of how Alexander Azimov (Yandex) has found the problem is below.____
> 
>     __ __
> 
>     In short: if TCP loses connectivity for 200ms (or 1s in SYN stage) then TCP changes IPv6 flow label (for the active TCP session!) to push traffic to a different path.____
> 
>     Current networks are extensively ECMP, if intermediate nodes support flow label for hash calculation then a high probability that the path would be changed.____
> 
>     LINUX/BSD does not want to wait till the network will fix its problem.____
> 
>     __ __
> 
>     If the final destination was anycast then the final destination would be changed too by the same hash calculation.____
> 
>     The stateful session would be broken as a result (see the second part of Alexander’s presentation below).____
> 
>     __ __
> 
>     Since the time LINUX has made the default RTO flow label recalculation (2016), IPv6 Anycast is broken.____
> 
>     People would have one more reason not to migrate to IPv6. Flow label does not exist in IPv4 – OS is not capable to break IPv4 Anycast similarly.____
> 
>     __ __
> 
>     Is anybody would like to spend his/her karma to save IPv6 Anycast OR let it die?____
> 
>     It is broken already for 5 years and nobody has spotted it up to now. Is it needed?____
> 
>     (I have seen a few drafts heavily dependent on IPv6 anycast)____
> 
>     __ __
> 
>     What is proper WG for such a problem?____
> 
>     __ __
> 
>     I am concerned that Anycast has been killed, it is not an easily replaceable tool.____
> 
>     Maybe somebody would propose something better but if not____
> 
>     then LINUX should be returned to 2015 when flow label change on RTO 
was a non-default configuration.____
> 
>     Such LINUX behavior could be valuable in some restricted domains (see below) when the administrator is sure that Anycast is not possible on the traffic path.____
> 
>     __ __
> 
>     Eduard____
> 
>     *From:*Vasilenko Eduard
>     *To:* 'Jeff Tantsura' <[email-masked] <mailto:[email-masked]>>; Alexander Azimov <[email-masked] <mailto:[email-masked]>>
>     *Cc:* Alexander Azimov <[email-masked] <mailto:[email-masked]>>; routing WG <[email-masked] <mailto:[email-masked]>>
>     *Subject:* RE: Self-healing Networking with Flow Label____
> 
>     __ __
> 
>     Hi all,____
> 
>     Not many people worldwide read this alias and understand____
> 
>     That RTO could be leveraged to fight “silent drops” 
in the DC environment.____
> 
>     It is a good use case to publish/document (with more details that it was in the presentation).____
> 
>     I hope that in the future OAM would be used for this purpose – it is better from architecture point of view.____
> 
>     Eduard____
> 
>     *From:*Jeff Tantsura [mailto:[email-masked] <mailto:[email-masked]>]
>     *To:* Alexander Azimov <[email-masked] <mailto:[email-masked]>>
>     *Cc:* Vasilenko Eduard <[email-masked] <mailto:[email-masked]>>; Alexander Azimov <[email-masked] <mailto:[email-masked]>>; routing WG <[email-masked] <mailto:[email-masked]>>
>     *Subject:* Re: Self-healing Networking with Flow Label____
> 
>     __ __
> 
>     Eduard,____
> 
>     __ __
> 
>     The idea of the draft to come is to explain what to do - when and how.____
> 
>     The goal is not to regulate (we really don’t) but to provide, similarly to RFC7938 a set of guidelines that community can use to build better and more resilient networks.____
> 
>     __ __
> 
>     Cheers,____
> 
>     Jeff____
> 
>     __ __
> 
>         On Aug 2, 2021, at 04:01, Alexander Azimov <[email-masked] <mailto:[email-masked]>> wrote:____
> 
>         ____
> 
>         Eduard, ____
> 
>         __ __
> 
>         пн, 2 авг. 2021 г. в 13:45, Vasilenko Eduard <[email-masked] <mailto:[email-masked]>>:____
> 
>             It is the key in this presentation “This behavior MUST be switched off by default”____
> 
>             It has been shown on slides 7-10 that flow label change on RTO is enabled by default for BSD and LINUX.____
> 
>             It needs regulation. It needs a standard RFC. Because it kills Anycast otherwise.____
> 
>         As I'm partially responsible for the key points of the presentation, I can stress that it is a bit different.____
> 
>           * We have an opportunity for self-healing TCP on top of IPv6, it should be preserved;____
>           * The Linux defaults should be changed to a safe mode to prevent session timeouts;____
>           * The hash recalculation behavior should be documented;____
> 
>         I'm not sure what you mean by the term 'regulation'. ____
> 
>          ____
> 
>             The story of how to use RTO to work-around “silent drop” vendor’s bugs could be a good informational RFC.____
> 
>             My be people developing iOAM would pay more attention to this use case.____
> 
>              ____
> 
>             IMHO: these are 2 separate drafts.____
> 
>         I'm not sure about it, we'll try to provide -00 before the next 
IETF meeting, let's see how it progresses.____
> 
>          ____
> 
>             Eduard____
> 
>             *From:*Alexander Azimov [mailto:[email-masked] <mailto:[email-masked]>]
>             *To:* Vasilenko Eduard <[email-masked] <mailto:[email-masked]>>; Jeff Tantsura <[email-masked] <mailto:[email-masked]>>
>             *Cc:* routing WG <[email-masked] <mailto:[email-masked]>>
>             *Subject:* Re: Self-healing Networking with Flow Label____
> 
>              ____
> 
>             Eduard,____
> 
>              ____
> 
>             Please see the quote from the slide 28. My suggestion was:____
> 
>              ____
> 
>             Client – sends SYN, Server – responds with SYN&ACK____
> 
>               * In case of SYN_RTO or RTO events Server SHOULD recalculate its TCP socket hash, thus change Flow Label. This behavior MAY be switched on by default;____
>               * In case of SYN_RTO or RTO events Client MAY recalculate 
its TCP socket hash, thus change Flow Label. This behavior MUST be switched off by default;____
> 
>             This looks like a safe default behavior, that saves the part of the improvements, but makes the work with stateful anycast services safe.____
> 
>              ____
> 
>             And yes, IMO it's ok to have a knob to enable it in the controlled environment. If you ask how to enable it in the presence of internal anycast services - there was also a suggestion in the slides: eBPF. It gives a good way to make this kind of separation.____
> 
>              ____
> 
>             02.08.2021, 11:48, "Vasilenko Eduard" <[email-masked] <mailto:[email-masked]>>:____
> 
>                 Hi Jeff,____
> 
>                 The situation when Control Plane does not understand what the Forwarding pane doing is a bug.____
> 
>                 Yes, RTO in TCP helps to find a work-around for this bug. And yes, Anycast is typically absent inside DC – it does not create the problem in the DC environment.____
> 
>                  ____
> 
>                 But the same LINUX is used outside DC. RTO Flow Label change here would create even more problems if Anycast would happen on the 
traffic path (not much predictable for client).____
> 
>                 Do we need separate LINUX distribution for DC and separate distribution for other environments?____
> 
>                 Or should we rely on the proper non-default configuration for different environments? (Admin should not forget to change)____
> 
>                 What if Anycast would become needed in DC?____
> 
>                  ____
> 
>                 RTO flow label recalculation is mutually exclusive with 
Anycast on the traffic part.____
> 
>                 What is more valuable for the public?____
> 
>                  ____
> 
>                 IMHO: It is better to fight the problem of such type of 
a bug with iOAM than to cancel Anycast.____
> 
>                  ____
> 
>                 IMHO: It is better to have Flow Label recalculation on RTO as “off” by default.____
> 
>                 DC Admin has the higher qualification to activate it in 
a controlled environment than every client worldwide that should not forget to disable it.____
> 
>                  ____
> 
>                 Eduard____
> 
>                 *From:*Jeff Tantsura [mailto:[email-masked] <mailto:[email-masked]>]
>                 *To:* Vasilenko Eduard <[email-masked] <mailto:[email-masked]>>
>                 *Cc:* [email-masked] <mailto:[email-masked]>; routing WG <[email-masked] <mailto:[email-masked]>>
>                 *Subject:* Re: Self-healing Networking with Flow Label____
> 
>                  ____
> 
>                 Eduard,____
> 
>                  ____
> 
>                 The issue is present not in link/device case, if well implemented - fast rehash takes care of updating forwarding within a number of ms. The problem is with  “gray” failures,  when the link in question is UP from routing/forwarding prospective but drops traffic (mostly occasionally and when a HW bug occurs has a distinct 
flow attributes).____
> 
>                  ____
> 
>                 In many large DC fabrics, the majority of the traffic is east-west, between end-points that aren’t anycast. In such deployments - the solution solves  issues rather elegantly and without any interventions from the operator.____
> 
>                 The issues/side effects are well understood and will be 
documented.____
> 
>                  ____
> 
>                 The best way to receive RTGWG emails is well… to subscribe to RTGWG ;-)____
> 
>                 Cheers,____
> 
>                 Jeff____
> 
> 
>                  ____
> 
>                     On Aug 1, 2021, at 09:47, Vasilenko Eduard <[email-masked] <mailto:[email-masked]>> wrote:____
> 
>                      ____
> 
>                     Hi  Alexander,____
> 
>                      ____
> 
>                     Have I understood your presentation right?____
> 
>                     The client SHOULD change IPv6 flow label after SYN RTO to have a chance to be moved to the working path inside DC fabric (if 
DC fabric supports flow label for hash calculation)____
> 
>                     But at the same time____
> 
>                     The client SHOULD NOT change the IPv6 flow label after SYN RTO to avoid being switched to a different TCP proxy engine.____
> 
>                      ____
> 
>                     Looks like a deadlock, especially if both things should happen for the same traffic:____
> 
>                     it should reach DC fabric____
> 
>                     and____
> 
>                     it should be hash load-balanced between different TCP proxy engines (or applications) inside DC Fabric.____
> 
>                      ____
> 
>                     I see one bad solution (“Disable Flow Label”):____
> 
>                     Routers up to TCP proxy engine SHOULD be configured 
not to use flow label (by the way these are all routers on the Internet),____
> 
>                     TCP flow engines SHOULD be outside of the DC Fabric 
(CLOS) – probably in front of it.____
> 
>                     Routers/Switches inside DC Fabric SHOULD use flow labels.____
> 
>                      ____
> 
>                     I see another bad solution (“Disable Anycast”):____
> 
>                     Disable anycast on routers in principle, use only stateful LB.____
> 
>                      ____
> 
>                      ____
> 
>                     It has been commented in the chat that Anycast is not possible in principle for stateful connection. It is too general a statement.____
> 
>                     Anycast is just not compatible with Flow Label. It is not a problem for IPv4 anycast even if the connection is stateful (TCP) because 5-tuple for hash would not change.____
> 
>                     Hence, IPv6 anycast has become dead at the time when Flow Label change has been added in LINUX for active TCP session.____
> 
>                      ____
> 
>                     Among 3 thins:____
> 
>                     -          Anycast____
> 
>                     -          Flow Label load balancing (basic Flow Label functionality)____
> 
>                     -          Flow Label change on the active session for application to be more active in new path search____
> 
>                     You have to choose which one to kill – all 3 are not compatible with each other at the same.____
> 
>                     I vote to disable Flow Label change in LINUX. Then wait till the network would fix itself.____
> 
>                     We have so many fancy TE tools our days. A broken link or a broken node could be excluded from routing for 50ms.____
> 
>                      ____
> 
>                     PS: I am not subscribed to the RTGWG alias, please keep me on a copy of this thread.____
> 
>                     ____
> 
>                     Best Regards____
> 
>                     Eduard Vasilenko____
> 
>                     Senior Architect____
> 
>                     Europe Standardization & Industry Development Department____
> 
>                     Tel: +7(985) 910-1105, +7(916) 800-5506____
> 
>                      ____
> 
>                     [email-masked] <mailto:[email-masked]>
>                     https://www.ietf.org/mailman/listinfo/rtgwg <https://www.ietf.org/mailman/listinfo/rtgwg>____
> 
>              ____
> 
>              ____
> 
>             -- ____
> 
>             Best regards,____
> 
>             Alexander Azimov____
> 
>              ____
> 
>             [email-masked] <mailto:[email-masked]>
>             https://www.ietf.org/mailman/listinfo/rtgwg <https://www.ietf.org/mailman/listinfo/rtgwg>____
> 
> 
>         ____
> 
>         __ __
> 
>         -- ____
> 
>         Best regards,____
> 
>         Alexander Azimov____
>

---

From: Brian E Carpenter <[email-masked]>
Date: Sun, 8 Aug 2021 07:57:46 +1200

On 08-Aug-21 07:41, Tom Herbert wrote:
> 
> 
> On Sat, Aug 7, 2021, 10:55 AM Toerless Eckert <[email-masked] <mailto:[email-masked]>> wrote:
> 
> 

I can't speak for what the IETF thought when it reached consensus on that 
statement. I can speak for what I thought when it was drafted: we didn't know enough about how the flow label would work in practice to tighten the definition of "flow". In the context 
of unicast destinations, that isn't catastrophic for on-path load balancing. It may be catastrophic for server load balancing, however. Or taking a different view, it may be desirable for server load balancing to use the same flow label for different transport associations that happen to be part of the same application layer transaction. (Hence the need for an API as Robert Raszuk noted on another sub-thread.)

What I don't like is Linux deciding this for me. It may be convenient in some DC scenarios, and highly inconvenient/damaging in other scenarios. It clearly needs to be a settable option per transport session.

    Brian

> Correct, it's the host prerogative to set the flow as it sees fits. Generally, we want the flow label to be consistent for the lifetime of the flow, that is why the flow label is only changed at RTO when the connection is failing. So this is really intended as a last ditch effort to salvage a failing connection.
> 
> As for relying on the network to fix problems, the host cannot assume that. The host has detected a problem with a connection's path, there is no reason to believe that the networks, or networks, in the path will even 
> 
> Also, as the subject states these patches have been in Linux and deployment for 5 years now. The claim that it kills anycast seems overly melodramatic. Is there any real data on the problem?
> 
> I'll also point out that any route change in the network could also have the same effect. If anycast, or other stateful mechanisms, requires that all packets of transport flow are routed consistently for the lifetime of a flow then inevitably failures will occur. Consistent outing per flow 
is only best effort and not a standard requirement.
> 
> This flow label modulation patches  were presented in routing area 
WG at IETF 111. The intent is to document it and we are looking at mitigations (the algorithm might be a little too aggressive for sending into the public Internet).
>     > On Sat, Aug 7, 2021, 6:19 PM Toerless Eckert <[email-masked] <mailto:[email-masked]>> wrote:
>     >
>     [email-masked] <mailto:[email-masked]>
> 
>     [email-masked] <mailto:[email-masked]>
<https://www.ietf.org/mailman/listinfo/ipv6>

---

From: Michael Tuexen <[email-masked]>
Date: Sat, 7 Aug 2021 22:30:19 +0200

> On 7. Aug 2021, at 21:41, Tom Herbert <[email-masked]> wrote:
> 
> 
> 
I think it comes down to the question of what a flow is...

Some might consider a TCP connection being a single a flow, some seems to think that a TCP connection
consists of multiple flows, especially a new flow being started on a timer based retransmission.

Best regards
Michael
> 
> As for relying on the network to fix problems, the host cannot assume that. The host has detected a problem with a connection's path, there is no reason to believe that the networks, or networks, in the path will even detect the problem much less fix it quickly. It would be nice if the host had a standard means to report to the network that it sees a problem, but that doesn't exist.
> 
> I'll also point out that any route change in the network could also have the same effect. If anycast, or other stateful mechanisms, requires that all packets of transport flow are routed consistently for the lifetime of a flow then inevitably failures will occur. Consistent outing per flow is only best effort and not a standard requirement.
> 
> This flow label modulation patches  were presented in routing area WG at IETF 111. The intent is to document it and we are looking at mitigations (the algorithm might be a little too aggressive for sending into the public Internet).
> 
> Tom
> 
> 
> Cheers
>     Toerless
>

---

From: Tom Herbert <[email-masked]>
Date: Sat, 7 Aug 2021 14:22:52 -0700

> consists of multiple flows, especially a new flow being started on a timer based retransmission.
Right.

There's also QUIC where a NAT may evict a UDP 4-tuple state and later
instantiate a new tuple for the QUIC connection with different port
numbers. QUIC is designed to survive this since the UDP port numbers
are part of its connection tuple, but changing the UDP port numbers
mid-flow could change ECMP routing such that an anycast endpoint could
become unreachable.

In reality, it's not that flow label modulation or NAT break anycast,
it's that anycast is inherently broken since it makes assumptions that
are true only most of the time.

Tom

---

From: Töma Gavrichenkov <[email-masked]>
Date: Sun, 8 Aug 2021 00:54:30 +0300

Peace,

On Sun, Aug 8, 2021 at 12:24 AM Tom Herbert <[email-masked]> wrote:
> numbers.

Just so that we're all on the same page, we're talking of IPv6 where
NAT is considered an enormously unprofessional practice.  This is what
the entire protocol was designed for: to eliminate the necessity of
NAT where it's not really supposed to happen.

> are true only most of the time.

The anycast isn't broken, it's fundamental to the Internet due to the
architecture of the latter.

Broken are the protocol and the application designs which rely on a
transport flow being stable and uninterrupted over the course of time.
This is just not how the Internet works, and it never did work this
way, and it never would.

--
Tòma

---

From: Robert Raszuk <[email-masked]>
Date: Sun, 8 Aug 2021 00:09:51 +0200

Töma,

> architecture of the latter.

Unfortunately I do think Tom is correct.

So allow me to attempt to expand a bit his point.

Internet last time I checked uses IP. IP is connection less by design. So
is entire Internet architecture.

Anycast in a connection less fashion - meaning stateless - will work just
fine - pretty much in all cases.

But using connection less forwarding plane and transporting connection
oriented protocols (like TCP) only works between solid anchors (read
endpoints).

Anycast does not meet the requirement of solid endpoint - so it should be
of no surprise to anyone that connections may break.

And this is not only about flow label and hashing. Internet routing
continues to churn. Every second routing may decide for you to prefer some
other BGP path and your anycast destination may end up in different
destination naturally only by following the destination routing paradigm.

Best,
Robert.

PS. Of course there are more smart ways to use anycast for TCP. Those
involve attracting packets only for further forwarding to the consistently
selected servers/gateways. Especially useful where global network may be
more reliable then scenic paths.

---

From: Töma Gavrichenkov <[email-masked]>
Date: Sun, 8 Aug 2021 01:10:25 +0300

Peace,

> consists of multiple flows, especially a new flow being started on a timer based retransmission.

The TCP connection, or a QUIC connection, or an SCTP connection, or
whatever there is...
The flow is just not the _concept_ of the layer where all of those
types of connections keep grazing.

The flow is the _application_concept_. It's entirely up to the
application whether the latter considers certain requests,
encapsulated in IPv6 packets, to be a part of the same flow (and these
thus should be all treated by the network, transport-wise, in a
similar fashion), or not.

Due to the incredibly long IPv6 deployment timeline, this concept has
sort of slipped away from the protocol implementors.  Therefore, the
perfect scenario where the _application_ itself decides whether it
wants to keep the flow label on the transport connection or not
practically vanished in implementations.  There just were no requests
for the control over the flow label function to be available to the
application layer, because no one cared about the performance of
applications over IPv6...

...because no one really have had any real demands for the IPv6
performance, as it has, quite unfortunately, become a fine practice to
just switch off IPv6 once there seem to be any performance issues with
the network.

Either it's the application which is the entity controlling the flow
label function, or the flow label concept should be abandoned, here's
the
gambit.

--
Töma

---

From: Töma Gavrichenkov <[email-masked]>
Date: Sun, 8 Aug 2021 01:23:02 +0300

Peace,

On Sun, Aug 8, 2021 at 1:09 AM Robert Raszuk <[email-masked]> wrote:
> Internet last time I checked uses IP. IP is connection less by design. So is entire Internet architecture.
> Anycast in a connection less fashion - meaning stateless - will work just fine - pretty much in all cases.
> But using connection less forwarding plane and transporting connection oriented protocols (like TCP) only works between solid anchors (read endpoints).

The entire idea that there are any "solid" anchors on the Internet
kinda defeats the purpose of the Internet, doesn't it?

We're now accustomed to the Internet where the _packet_flows_ are
kinda stable.  This is how it works, in practice, 95% of the time b/c
of giant intercontinental fiber lines, b/c ISPs have their SLAs, etc.

This is probably the shiny moment in time this is true, but it's not
supposed to stay for much longer, in part because the ISPs are
merging, the controlling entities want some control over that,
peerings break, but mainly because it was never supposed to go this
way, since the Internet is _inherently_peer_to_peer_, with no
guarantee over the paths, with no SLAs over the paths and the RTTs.

It's not the anycast that doesn't meet the concept of long-living
stable connections, it's the concept of stable connections that
doesn't fit into the architecture of the Internet.

> And this is not only about flow label and hashing. Internet routing continues to churn. Every second routing may decide for you to prefer some other BGP path and your anycast destination may end up in different destination naturally only by following the destination routing paradigm.

Exactly, and you need to get your applications to be aware of that,
and not to shift this responsibility to the network which actually
bears no responsibility, in particular over that.

> PS. Of course there are more smart ways to use anycast for TCP. Those involve attracting packets only for further forwarding to the consistently selected servers/gateways.

I believe this was covered before on the thread, and I don't see any
answers here to the issues raised before.

--
Töma

---

From: Robert Raszuk <[email-masked]>
Date: Sun, 8 Aug 2021 00:29:35 +0200

That is why IRTF worked for few years on Locator-ID split where endpoints
are stable and there is mapping plane (of some sort - solution dependent)
decoupling transport from end to end connection IDs.

Today ILNP and LISP solutions offer such decoupling in practice.

So now we fully diverged from the original topic :)  But anycast was not
the original problem anyway ...

Cheers,
R.

On Sun, Aug 8, 2021 at 12:23 AM Töma Gavrichenkov <[email-masked]> wrote:

> selected servers/gateways.
>

---

From: Töma Gavrichenkov <[email-masked]>
Date: Sun, 8 Aug 2021 01:44:11 +0300

Peace,

> But using connection less forwarding plane and transporting connection oriented protocols (like TCP) only works between solid anchors (read endpoints).

Tell you what.
Some 12 years ago I was working in HP, and I had to do some weird
things with OpenVMS.

OpenVMS (formerly VMS, and VAX/VMS before that) is/was a true real
time system.  Lots of life-controlling systems were built on top of
OpenVMS these days.  I recall the ISS is still flying because it's
powered by VAX/VMS but I might really be wrong about it.

Anyhow, the TCP protocol wasn't considered reliable in VAX/VMS because
it didn't guarantee reliable data delivery.  By "reliable", the VMS
architects meant the reliable delivery over the reliable maximum
timeframe.

This reliable delivery required actual network and transport network
architecture to be worked on.  Copper line maps from every endpoint to
every endpoint were a part of every project, with redundancy ensured,
and network continuity protocols were signed off by the customer, with
multi-country regulation entities also having their signoffs,
guarantees, and insurances.

And it was expensive as hell, but it worked just the way you want it
to work: stable, reliable, and constant over time.

The Internet isn't that expensive because it's built on compromises.
TCP is one of these compromises.  It doesn't guarantee reliable
delivery in terms of VAX/VMS — and neither do SCTP nor QUIC —  but all
of these are cheap, and the rest is up to the application, and it
should be the application that could handle that.

Yet, no one should never expect a perfect performance from a
compromise.  It just was never paid for.

--
Töma

---

From: Töma Gavrichenkov <[email-masked]>
Date: Sun, 8 Aug 2021 02:27:25 +0300

Peace,

Now, speaking of compromises,

On Sun, Aug 8, 2021 at 1:09 AM Robert Raszuk <[email-masked]> wrote:
> Every second routing may decide for you to prefer some other BGP path and your anycast destination may end up in different destination naturally only by following the destination routing paradigm.

In my opinion, this is a perfect sample of a bad engineering approach.
Here's why.

In practice, routing on the Internet does never change this way.  A
single BGP route announcement takes some 30 seconds to propagate, and
sometimes a route withdrawal takes more than 4 minutes.  Severe
routing changes by an AS are just not expected to happen more
frequently than a few times in an hour, and massive changes shouldn't
happen more often than once in a few hours, because they might cause
routing loops, weird routing issues, and prolonged network
unavailability otherwise.

This is the limitation to rely on for an application, or, somewhat
worse, for a transport protocol.

Engineering is basically a process of developing compromises over
limitations.  You need to thoroughly understand the latter to build a
system.  Engineers don't build houses capable of resisting the energy
of an intergalactic meteor landing.  Your car would break and kill you
if you smash it into the wall cruising 900 mph, and I don't even need
to know the make to tell you that, because I already know your car
certainly isn't a Batmobile, and it's therefore built on compromises,
over limitations, and there are warning signs on every road because of
that.

An attempt to pretend the Internet engineering doesn't have to keep
any limitations in mind just leads to bad engineering practices.
There's always a compromise, and you need to know the limitations.

And there are no limitations on the Internet of the past, present, or
the future, which could possibly make the anycast impractical.
Application designs may do, because of poor engineering practices, but
that's it.

--
Töma

---

From: Robert Raszuk <[email-masked]>
Date: Sun, 8 Aug 2021 01:47:32 +0200

> In practice, routing on the Internet does never change this way.

I recommend you spend a moment on RIPE BGP Play to observe how real life
routing paths change to any anycast address sourced by more then one ASN.

If my TCP session is supposed to last 1s - maybe routing changes will not
impact it. But if someone uses anycast to establish long lasting TCP
sessions (say 12h) - Good Luck !

And that is with or without stable flow lable. Just look at IPv4 where flow
label does not exist.

Thx,
R.

On Sun, Aug 8, 2021 at 1:27 AM Töma Gavrichenkov <[email-masked]> wrote:

> > Every second routing may decide for you to prefer some other BGP path
> and your anycast destination may end up in different destination naturally
> only by following the destination routing paradigm.
>

---

From: Töma Gavrichenkov <[email-masked]>
Date: Sun, 8 Aug 2021 02:53:04 +0300

Peace,

On Sun, Aug 8, 2021 at 2:47 AM Robert Raszuk <[email-masked]> wrote:
> if someone uses anycast to establish long lasting TCP sessions (say 12h) - Good Luck !

Transport sessions cannot reliably last for 12 hours on the Internet.
That's the thing.

An application should never rely on that.  12 minutes are already kind
of long, but 12 hours sound ridiculous.
You can achieve that over multiple paths (MP-TCP was mentioned along
the thread) with some higher level abstraction (also subject to
eventually break) but that's it.

This is the limitation.  There's no way 'round it.

--
Töma

---

From: Töma Gavrichenkov <[email-masked]>
Date: Sun, 8 Aug 2021 02:57:47 +0300

Peace,

On Sun, Aug 8, 2021 at 2:47 AM Robert Raszuk <[email-masked]> wrote:
> I recommend you spend a moment on RIPE BGP Play to observe how real life routing paths change to any anycast address sourced by more then one ASN.

I had maintained a completely anycast network for 10 consecutive
years, until 2021, with zero to no availability issues.  And, with no
modern features like "we've got anycast for dispatch only but we will
serve you a unicast address once you try to connect".
Is there anything new on the BGPlay for me in 2021 to take a look at
with that regard?  Please advise.

--
Töma

---

From: Tom Herbert <[email-masked]>
Date: Sat, 7 Aug 2021 17:27:09 -0700

> Transport sessions cannot reliably last for 12 hours on the Internet.

Toma,

Transport connections follow the end-to-end model. Transport state is
maintained at the endpoints and not in the network, and hence as long
as there remains a reachable path between the running end points the
connection is viable. That is the fundamental architecture of the
Internet. Reliability diminishes when the network is invasive in the
transport layer and E2E model is not adhered to.

> of long, but 12 hours sound ridiculous.

No application relies on connections being guaranteed to be viable for
any specified amount of time, but that is not the same thing as saying
that there aren't long lived connections that productively exist or
that the network should proactively disallow long lived connections.
If a connection is still useful after 12 hours, and everyone is happy,
why tear it down and start over?

> eventually break) but that's it.
How does anycast work with Multipath TCP? Wouldn't each subflow
potentially be routed to a different anycast host?

Tom

---

From: Tom Herbert <[email-masked]>
Date: Sat, 7 Aug 2021 17:49:50 -0700

Toma,

Yes, a major selling point of IPv6 was that it would eliminate the
need for NAT because of the enormity of the address space. I think
that the same rationale could be applied that anycast should be
unnecessary in IPv6 because there is a lot of address space to work
with. For instance, instead of defining anycast endpoints behind one
address and hoping that the network can consistently route packets
based on per-connection layer transport information, why not just
embed the transport layer information in the IP address? e.g. maybe
the ephemeral port number, so that the packet is always routed
properly to the server instance regardless of how it's routed to the
destination. To make this work, DNS could return an address range
record instead of a single address and then a client to randomly
select an address to connect to from the range to achieve desired load
balancing.

> This is just not how the Internet works, and it never did work this

But consistent routing per flow is a requirement of anycast, hence why
I think anycast is broken...

Tom

---

From: Töma Gavrichenkov <[email-masked]>
Date: Sun, 8 Aug 2021 04:06:02 +0300

Peace,

On Sun, Aug 8, 2021 at 3:50 AM Tom Herbert <[email-masked]> wrote:
> I think
> with.

No, that doesn't work

> For instance, instead of defining anycast endpoints behind one
> embed the transport layer information in the IP address?

Because DDoS, as has been pointed out in this thread a dozen times

--
Töma

---

From: Töma Gavrichenkov <[email-masked]>
Date: Sun, 8 Aug 2021 04:09:20 +0300

Peace,

On Sun, Aug 8, 2021 at 3:27 AM Tom Herbert <[email-masked]> wrote:
> maintained at the endpoints and not in the network

Ideally, yes, this is what we have IPv6 for: to eliminate NAT which is
the (part of the) transport state maintained in the network.

> why tear it down and start over?

Yep, no need, why?
You just need to be prepared that it would eventually break.

> potentially be routed to a different anycast host?

Depends on the architecture of the application.  Might work well, or
might not work.

--
Töma

---

From: Töma Gavrichenkov <[email-masked]>
Date: Sun, 8 Aug 2021 04:13:14 +0300

Peace,

On Sun, Aug 8, 2021, 3:50 AM Tom Herbert <[email-masked]> wrote:

> why not just
> destination.

How's that better than the IPv6 anycast (which is a very different beast
that the BGP anycast, by the way)?

--
Töma

---

From: Masataka Ohta <[email-masked]>
Date: Sun, 8 Aug 2021 11:16:20 +0900

Robert Raszuk wrote:

> But if someone uses anycast to establish long lasting TCP

What if someone uses windows to establish long lasting
unicast TCP sessions?

							Masataka Ohta

---

From: Tom Herbert <[email-masked]>
Date: Sat, 7 Aug 2021 19:19:49 -0700

Toma,

I don't see DDOS mentioned in this thread. If the concern is that an
attacker could DDOS an individual address then I'd point out that all
unicast addresses are susceptible to that anyway. Using anycast as a
mitigation to DDoS doesn't seem like a great idea considering the
problems being discussed here.

Tom

---

From: Töma Gavrichenkov <[email-masked]>
Date: Sun, 8 Aug 2021 10:27:15 +0300

Peace,

On Sun, Aug 8, 2021, 5:20 AM Tom Herbert <[email-masked]> wrote:

> I don't see DDOS mentioned in this thread.

You might have some serious procmail configuration...
Check out the archives: https://mailarchive.ietf.org/arch/browse/ietf/

If the concern is that an
> unicast addresses are susceptible to that anyway.

Exactly my point.

Using anycast as a
> problems being discussed here.

It's quite the opposite: using anycast to mitigate DDoS is the only proper
way to do it, because, basically, DDoS traffic, generated in thousands of
locations on the globe, cannot be handled when accumulated in one place.

Either you have multiple traffic termination points on the net (a.k.a.
anycast), each as close to some traffic generation point as possible, or
you'll end up having capacity overload around your last mile.  This is the
equation fundamental to the Internet, while the implementation issues
discussed here are hardly more than just typical software engineering tasks.

--
Töma

---

From: Robert Raszuk <[email-masked]>
Date: Sun, 8 Aug 2021 12:31:18 +0200

> Transport sessions cannot reliably last for 12 hours on the Internet.

The thing is that 1000s of people come to their desks at 8:00, they turn on
their financial application - which uses TCP - and turn it off at 17:00
local time. And the single session stays up just fine for 9h.  In fact
session stays much longer such that in such application there is hard stop
at 18 hours to kill the session. And such forced session killing does
happen a lot every day.

Of course - to make it crystal clear I would never endorse designing any
application like this - but when such apps were written X years ago
networking was taken as balck box abstraction shielded by a socket.  It is
just real proof that your point is simply not true.

> A single BGP route announcement takes some 30 seconds to propagate, and
> sometimes a route withdrawal takes more than 4 minutes.

That's a pure occurance of spreading FUD.

I do recommend running your own measurements or consult those who do it
well. For example Geoff. In fact his cats and data presented  illustrate
very well the point I was making earlier:

 https://blog.apnic.net/2019/05/15/the-speed-of-bgp-network-propagation/

And with the average of 4.3 of average AS path length maybe was something
in late 1990s where withdraws or first updates were subject to MRAI timer
in some implementations - that is just history.

Best,
R.

---

From: Simon Hobson <[email-masked]>
Date: Sun, 8 Aug 2021 13:53:24 +0100

> doesn't fit into the architecture of the Internet.

Perhaps it's me missing something here, but I don't follow you. Surely a globally unique address is a "solid anchor" in that (unless something has gone badly wrong), that will belong to one and only one node* - and it will remain belonging to that one node for as long as the admins keep it that way.
So ignoring anycast, we send a packet to <some address> and "somehow" it arrives at <some address> and hence the one node that own that address. In general we don't care how it gets there, but hopefully it does. We don't care if one packet goes direct via a short route, while another packet goes all round the world, either the protocol (TCP) or the application level re-assembles the packets into the order they are needed, or some get dropped if time is more important than completeness (e.g. real-time audio stream). I've certainly had SSH sessions running for long times (sometimes days), even when I know that there's been a temporary failure (e.g. our local DSL line has dropped), though in the latter case, it's reliant on either not sending any packets while there is no available path, or the network becoming available again within the timeout limit for the protocols involved. More often I've had to add keepalives to deal with aggressive NAT gateways (IPv4) that drop inactive mappings even if they are TCP and the gateway has not seen it teared down.

Of course anycast changes that - if routing changes then the other end may change. But I would not consider an anycast address to be a "solid anchor" for a long lived connection unless the protocols concerned don't care if the other end node gets swapped part way through a session.

* If that address happens to be something like a load balancer then that doesn't really matter since it's down to the operator of that cluster of devices to manage things so that remote devices can establish long lived (where "'long lived" will be different for different connections involved) connections without the load balancer switching nodes on it - and thus give the outward appearance of there being "one node".

Simon

---

From: Tom Herbert <[email-masked]>
Date: Sun, 8 Aug 2021 07:40:07 -0700

> discussed here are hardly more than just typical software engineering tasks.

Yes, a service needs multiple points of entry, but anycast isn't the only
way to do that. For instance, DNS can return different addresses to users
in different geographic regions, mirrors have long been used for file
download.

Tom

---

From: Töma Gavrichenkov <[email-masked]>
Date: Sun, 8 Aug 2021 17:57:19 +0300

Peace,

On Sun, Aug 8, 2021, 1:30 PM Robert Raszuk <[email-masked]> wrote:

> The thing is that 1000s of people come to their desks at 8:00, they turn
> on their financial application - which uses TCP - and turn it off at 17:00

*sigh* yes, there might be some legacy applications which don't work well
with connection reestablishment.  Such legacy applications won't be getting
any DDoS protection except for the basic one.  This is a fine trade-off for
many.

I, however, don't see the point.  For seemingly every network feature out
there there's an application which breaks on that feature.  Anycast is not
an exception.

An order of magnitude more applications break when they encounter MTUs less
than 1500, but we're not going to abandon the concept of MTU, right?

It is just real proof that your point is simply not true.

Not really, as you can see.

>  https://blog.apnic.net/2019/05/15/the-speed-of-bgp-network-propagation/

Kinda nothing in this article contradicts what I said before.  Well, it was
sort of implied it's rather box-and-whiskers than a constant reliable time,
as this is a statistical process, if _this_ is your concern.  Sorry if it
wasn't unclear.

Whether you count not pointing out that a statistical process is, in fact,
statistical as FUD or not is entirely up to you.

--
Töma

---

From: Töma Gavrichenkov <[email-masked]>
Date: Sun, 8 Aug 2021 18:01:06 +0300

Peace,

> in different geographic regions, mirrors have long been used for file

And the cybercriminal would just take down all of these unicast addresses
with a DDoS, one by one, until so many scrubbing centers are down that the
rest cannot handle the traffic, resulting in a domino effect.

No, that doesn't work.

--
Töma

---

From: Tom Herbert <[email-masked]>
Date: Sun, 8 Aug 2021 08:05:04 -0700

> rest cannot handle the traffic, resulting in a domino effect.

And yet these have techniques have been successfully deployed for many
years. Not every address seen on the Internet is anycast.

Tom

---

From: Töma Gavrichenkov <[email-masked]>
Date: Sun, 8 Aug 2021 18:08:30 +0300

Peace,

On Sun, Aug 8, 2021, 5:40 PM Tom Herbert <[email-masked]> wrote:

> For instance, DNS can return different addresses to users in different
> geographic regions, mirrors have long been used for file download.
>

I want to highlight that because this is important.

There surely are methods to distribute endpoint addresses in such a way
that the client would have options on which endpoint to choose: the one you
advertise to the IPs in the client's region, or the one you advertised to
them w weeks ago, or the one the client has obtained via collaboration
(read C2C) with clients in other regions, et cetera.

But anycast is the only way to distribute the address(es) in such a way
that the client has _no_choice_.

And this is crucial in DDoS mitigation because otherwise you leave the
control over your traffic engineering to the attacker.  The resulting
complete downtime is just a matter of time then.

--
Töma

---

From: Töma Gavrichenkov <[email-masked]>
Date: Sun, 8 Aug 2021 18:12:07 +0300

Peace,

On Sun, Aug 8, 2021, 6:05 PM Tom Herbert <[email-masked]> wrote:

> DNS can return different addresses to users in different geographic
>>> regions, mirrors have long been used for file download.
>>>
>>

Not every address ever gets attacked, and when there's an attack, it isn't
_always_ complicated.

And there's no incentive now for the criminals to do research in that
matter because with TCP working fine over anycast, it's too complicated to
overcome the traffic engineering obstacles.

Take anycast out of the game, the situation would change.

--
Töma

---

From: Masataka Ohta <[email-masked]>
Date: Mon, 9 Aug 2021 01:21:37 +0900

> their financial application - which uses TCP - and

And, some people will suffer from OS crash before 17:00.

							Masataka Ohta

---

From: David Farmer <[email-masked]>
Date: Sun, 8 Aug 2021 14:03:37 -0500

> discussed here are hardly more than just typical software engineering tasks.

Anycast is only one of several mitigation strategies for DDoS, yes, it is a
good one for web type services, it might even be the best for that type of
service, especially against large volumetric attacks. However, there are
many other types of attacks to protect against and services that need
protection and anycast is a lousy mitigation strategy for many of them,
especially for client networks or peer to peer services.

While I agree with you, anycast is an important capability in the Internet
architecture, nevertheless it has many limitations, and is not the panacea
you claim it to be, even for DDoS.

Furthermore, I’m not sure what you or the original reporter of this problem
expect the IETF to do to fix the problem that was reported. I’ll remind you
of the well worn trope, “the IETF is not the protocol police.” Any fix to
the problem reported is squarely in hands of Linux developers, not the IETF.

Thanks.

===============================================
David Farmer               Email:[email-masked]
Networking & Telecommunication Services
Office of Information Technology
University of Minnesota
2218 University Ave SE        Phone: 612-626-0815
Minneapolis, MN 55414-3029   Cell: 612-812-9952
===============================================

---

From: Robert Raszuk <[email-masked]>
Date: Sun, 8 Aug 2021 21:05:12 +0200

> on

>                                                         Masataka Ohta

Maybe .. but I am not sure about you and others but I do not remember when
was the last time any of my linux or my windows crashed.

Apples - sure I do remember, but do not use it. iOS and Android too - do
crash from time to time.

But the point was that using a real wild Internet TCP session can stay for
hours or days irrespective of Internet routing or ECMP hashing when you set
it up with non anycast addresses.

Thx,
R.

PS. Someone keeps mentioning DDoS ... so how do you protect anycast
addresses from DDoS ? Today the most common method is to blacklist
destination ip address + port(s)... irrespective if it is done by NOC or
with automation using RFC5575(bis) I co-authored.

But if this protection as it should is happening as fast from target as as
possible, if you happen to use anycast your service is dead with not too
much effort from the attacker. To me actually the risk of DDoS would be
something to seriously consider before advertising anycast as a service
destination.

---

From: Töma Gavrichenkov <[email-masked]>
Date: Sun, 8 Aug 2021 22:19:55 +0300

Peace,

On Sun, Aug 8, 2021, 10:06 PM Robert Raszuk <[email-masked]> wrote:
> how do you protect anycast addresses from DDoS ?
> Today the most common method is to blacklist destination ip address +
port(s)...

With all due respect, this is not a protection, this is the
_implementation_ of the attack.

You're only protecting your lovely network from the collaterals of an
attack towards a service application hosted on the network.  The
application itself is then basically dead.  This is exactly what the
attacker wants.

--
Töma

---

From: Tom Herbert <[email-masked]>
Date: Sun, 8 Aug 2021 12:23:25 -0700

On Sun, Aug 8, 2021, 12:03 PM David Farmer <[email-masked]> wrote:

>> It's quite the opposite: using anycast to mitigate DDoS is the only
>> proper way to do it, because, basically, DDoS traffic, generated in
>> thousands of locations on the globe, cannot be handled when accumulated in
>> one place.
>>
> Anycast is only one of several mitigation strategies for DDoS, yes, it is
> a good one for web type services, it might even be the best for that type
> of service, especially against large volumetric attacks. However, there are
> Furthermore, I’m not sure what you or the original reporter of this
> problem expect the IETF to do to fix the problem that was reported. I’ll
> remind you of the well worn trope, “the IETF is not the protocol police.”
> Any fix to the problem reported is squarely in hands of Linux developers,
> not the IETF.
>

David,

That's true, however we, Linux developers, certainly value the input and
discussion on IETF lists. IMO, the later patch that started to recompute
the hash on each RTO probably is too aggressive to be the default behavior
on the Internet. I plan to post a patch that makes default less aggressive
by restoring the original default behavior to recompute hash only after
multiple RTOs. The rehash behavior will be configurable by sysctl and also
I'll add a socket option to allow the application to control the behavior
per connection thereby resolving Brian's concern.

Tom

---

From: Töma Gavrichenkov <[email-masked]>
Date: Sun, 8 Aug 2021 22:28:00 +0300

Peace,

On Sun, Aug 8, 2021, 10:03 PM David Farmer <[email-masked]> wrote:

> architecture

This is sort of an achievement already given some reckless claims on the
IETF mailing list!  Thank you.

nevertheless it has many limitations, and is not the panacea you claim it
> to be, even for DDoS.

If you consider the client-server model, it *is* the panacea,
fundamentally, but I agree that p2p and other models of communications
might require something else.  My bad, didn't consider these before.

> expect the IETF to do to fix the problem that was reported.
>

Honestly, I've just been answering the questions other people have, sharing
my experience.

I, too, believe the practical discussion rather belongs to LKML.

--
Töma

---

From: "Theodore Ts'o" <[email-masked]>
Date: Sun, 8 Aug 2021 18:40:38 -0400

> Take anycast out of the game, the situation would change.

Which of the top 5, 10, 100 sites on the Internet use anycast?

If Facebook, Amazon, Google, Wikipedia, etc., are using standard IPv4
and IPv6 endpoints and are *not* using anycast, and they have
successly fielded defenses against DDOS's without using anycast,
wouldn't that tend to blow a gigantic, gaping hole in your assertion?

Cheers,

						- Ted

---

From: Nick Hilliard <[email-masked]>
Date: Mon, 9 Aug 2021 00:08:42 +0100

Theodore Ts'o wrote on 08/08/2021 23:40:
> Which of the top5, 10,  100 sites on the Internet use anycast?

for starters, all the dns root servers. For content delivery, some of 
Cloudflare's content is delivered to end users using anycast on the 
front side.  Are the DNS root servers top-5, top-10 or top-100 sites 
(asking for a friend)?

> wouldn't that tend to blow a gigantic, gaping hole in your assertion?

It's the norm to build ddos defenses without anycast, but it has its 
place as a technology.

Otherwise: anycast is one of many tools in the box; rewriting the ipv6 
flow label hurts ipv6 anycast when DDOS traffic sinkers use ECMP for 
load balancing; tcp anycast is a hack which works quite nicely for 
short-lived tcp sessions and barely at all for long-lived sessions (this 
is well-understood in network engineering circles).

@Tom your suggestions for tuning down the flow label rewriting 
aggression level sound reasonable.

Nick

---

From: Masataka Ohta <[email-masked]>
Date: Mon, 9 Aug 2021 08:27:50 +0900

> it up with non anycast addresses.

That is a wrong point.

That you can often keep TCP sessions for days does not mean your
sessions are guaranteed to be kept for nine hours.

As such, no real world applications rely on such persistence.

						Masataka Ohta

---

From: Brian E Carpenter <[email-masked]>
Date: Mon, 9 Aug 2021 12:13:08 +1200

Tom,
On 09-Aug-21 07:23, Tom Herbert wrote:
> 
> 
> On Sun, Aug 8, 2021, 12:03 PM David Farmer <[email-masked] <mailto:[email-masked]>> wrote:
> 
> 
> 
>     On Sun, Aug 8, 2021 at 02:27 Töma Gavrichenkov <[email-masked] <mailto:[email-masked]>> wrote:
> 
>         Peace,
> 
>         On Sun, Aug 8, 2021, 5:20 AM Tom Herbert <[email-masked] <mailto:[email-masked]>> wrote:
> 
>             Using anycast as a
>         It's quite the opposite: using anycast to mitigate DDoS is the only proper way to do it, because, basically, DDoS traffic, generated in thousands of locations on the globe, cannot be handled when accumulated in one place.
> 
>         Either you have multiple traffic termination points on the net (a.k.a. anycast), each as close to some traffic generation point as possible, or you'll end up having capacity overload around your last mile.  
This is the equation fundamental to the Internet, while the implementation issues discussed here are hardly more than just typical software engineering tasks.
> 
> 
>     Anycast is only one of several mitigation strategies for DDoS, yes, 
it is a good one for web type services, it might even be the best for that type of service, especially against large volumetric attacks. However, there are many other types of attacks to protect against and services that need protection and anycast is a lousy mitigation strategy for many of them, especially for client networks or peer to peer services. 
> 
>     While I agree with you, anycast is an important capability in the Internet architecture, nevertheless it has many limitations, and is not the panacea you claim it to be, even for DDoS. 
> 
>     Furthermore, I’m not sure what you or the original reporter 
of this problem expect the IETF to do to fix the problem that was reported. I’ll remind you of the well worn trope, “the IETF is not the protocol police.” Any fix to the problem reported is squarely in hands of Linux developers, not the IETF.
> 
> 
> David,
> 
> That's true, however we, Linux developers, certainly value the input and discussion on IETF lists. IMO, the later patch that started to recompute the hash on each RTO probably is too aggressive to be the default behavior on the Internet. I plan to post a patch that makes default less aggressive by restoring the original default behavior to recompute hash only after multiple RTOs. The rehash behavior will be configurable by sysctl and also I'll add a socket option to allow the application to control the behavior per connection thereby resolving Brian's concern.

Yes, thanks, that looks good. How about a global default setting for a given host, too? That could be at sysctl level.

Thanks
    Brian

>     David Farmer               Email:[email-masked] <mailto:[email-masked]>

---

From: Tom Herbert <[email-masked]>
Date: Sun, 8 Aug 2021 17:22:54 -0700

On Sun, Aug 8, 2021, 5:13 PM Brian E Carpenter <[email-masked]>
wrote:

> >         It's quite the opposite: using anycast to mitigate DDoS is the
> only proper way to do it, because, basically, DDoS traffic, generated in
> >         Either you have multiple traffic termination points on the net
> (a.k.a. anycast), each as close to some traffic generation point as
> possible, or you'll end up having capacity overload around your last mile.
> This is the equation fundamental to the Internet, while the implementation
> issues discussed here are hardly more than just typical software
> engineering tasks.
> >
> >
> it is a good one for web type services, it might even be the best for that
> type of service, especially against large volumetric attacks. However,
> there are many other types of attacks to protect against and services that
> need protection and anycast is a lousy mitigation strategy for many of
> them, especially for client networks or peer to peer services.
> >
> >     While I agree with you, anycast is an important capability in the
> Internet architecture, nevertheless it has many limitations, and is not the
> panacea you claim it to be, even for DDoS.
> >
> of this problem expect the IETF to do to fix the problem that was
> reported. I’ll remind you of the well worn trope, “the IETF is not the
> protocol police.” Any fix to the problem reported is squarely in hands of
> Linux developers, not the IETF.
> >
> >
> > David,
> >
> Yes, thanks, that looks good. How about a global default setting for a
> given host, too? That could be at sysctl level.
>

Yes, that will be there.

Tom

---

From: Tom Herbert <[email-masked]>
Date: Sun, 8 Aug 2021 17:37:07 -0700

> front side.  Are the DNS root servers top-5, top-10 or top-100 sites

Route changes wouldn't impact stateless UDP use of anycast.

> is well-understood in network engineering circles).

Perhaps, but I would hope users are aware of the susceptibility of anycast
to arbitrary routing changes in the path (flow label modulation being just
one example). It's also a question of how much we should accommodate
protocols like this that aren't aligned with the core architecture of the
Internet. At some point such accomodations impede evolution of protocols
and the Internet

---

From: Jen Linkova <[email-masked]>
Date: Mon, 9 Aug 2021 11:44:59 +1000

> Route changes wouldn't impact stateless UDP use of anycast.

....unless the UDP packet is fragmented...

Ever tried RADIUS over IPv6? ;)

-- 
SY, Jen Linkova aka Furry

---

From: Patrik Fältström <[email-masked]>
Date: Mon, 09 Aug 2021 08:11:00 +0200

>  Route changes wouldn't impact stateless UDP use of anycast.

Route changes do not happen enough to impact TCP in reality, and in the cases where this happens, as many have said, applications do restart their flows. There are many reasons why flows die.

Yes, we at Netnod run Anycast for both TCP and UDP and it just works. This as one of many tools in the toolbox to maximise the ability for end users to reach whatever services we provide.

We use Anycast in the form of multiple termination points for the flows announced in BGP from the same AS but multiple locations, is in use, and it just works.

   Patrik

---

From: Ole Troan <[email-masked]>
Date: Mon, 9 Aug 2021 09:23:54 +0200

Patrick,

> We use Anycast in the form of multiple termination points for the flows announced in BGP from the same AS but multiple locations, is in use, and it just works.

How do you know?
It would be interesting if you have measurements of application behavior when routing changes.
If applications generally handle that well then it would affect how we reason about flash renumbering (and state redundancy for NATs) too.

Cheers 
Ole

---

From: Patrik Fältström <[email-masked]>
Date: Mon, 09 Aug 2021 09:42:25 +0200

>> We use Anycast in the form of multiple termination points for the flows announced in BGP from the same AS but multiple locations, is in use, and it just works.

By measuring number of complaints from customers and end users after many many years of using anycast for both DNS (TCP and UDP) and HTTPS. Last year also NTS.

I think it also depends on how you deploy your anycast, how you do announcements and other things. It is simply one of many tools you have to be used to and use the way that suits your environment.

Sure, there are customers that do not get a response, but that is as have been explained when one of the nodes get unresponsive, the BGP announcement is not withdrawn (intentionally or unintentionally) a situation you sometimes want to have. Simply because other customers should get a response (other nodes are not impacted by whatever has happened).

> If applications generally handle that well then it would affect how we reason about flash renumbering (and state redundancy for NATs) too.

Yes.

Randy Bush did some measurements on number of route changes "in general" to see whether TCP and anycast would work or not. The answer was clear. The route changes are so small that you have other issues that impact operation more. For example that you withdraw a prefix too late compared with when you turn off a node, so traffic is still flowing, or that some nodes simply do not return the expected response.

What we have seen is that applications do restart, happy eyeballs is your friend, or that the user do "reload" if a browser gives up.

When having an anycast based set of termination points this still only happens for the users that happen to have traffic routed to the node that just stopped returning packets. The rest of the users are not harmed.

No, we do not try to sync flow state between termination points. We rely on applications to restart.

Ole, just fire up your network of Ripe Atlas nodes and give it a try!!! :-)

Fragmentation of IPv6 UDP packets is a larger headache for me than anycast based TCP.

   Patrik

---

From: Vasilenko Eduard <[email-masked]>
Date: Mon, 9 Aug 2021 08:01:16 +0000

+1

-----Original Message-----
From: ietf [mailto:[email-masked]] On Behalf Of Töma Gavrichenkov
Sent: Sunday, August 8, 2021 1:10 AM
To: Michael Tuexen <[email-masked]>
Cc: Tom Herbert <[email-masked]>; 6man WG <[email-masked]>; IETF discussion list <[email-masked]>
Subject: Re: IPv6 Anycast has been killed by LINUX patch in 2016 - who cares?

Peace,

On Sat, Aug 7, 2021 at 11:31 PM Michael Tuexen <[email-masked]> wrote:
> Some might consider a TCP connection being a single a flow, some seems 
> to think that a TCP connection consists of multiple flows, especially a new flow being started on a timer based retransmission.

The TCP connection, or a QUIC connection, or an SCTP connection, or whatever there is...
The flow is just not the _concept_ of the layer where all of those types of connections keep grazing.

The flow is the _application_concept_. It's entirely up to the application whether the latter considers certain requests, encapsulated in IPv6 packets, to be a part of the same flow (and these thus should be all treated by the network, transport-wise, in a similar fashion), or not.

Due to the incredibly long IPv6 deployment timeline, this concept has sort of slipped away from the protocol implementors.  Therefore, the perfect scenario where the _application_ itself decides whether it wants to keep the flow label on the transport connection or not practically vanished in implementations.  There just were no requests for the control over the flow label function to be available to the application layer, because no one cared about the performance of applications over IPv6...

...because no one really have had any real demands for the IPv6 performance, as it has, quite unfortunately, become a fine practice to just switch off IPv6 once there seem to be any performance issues with the network.

Either it's the application which is the entity controlling the flow label function, or the flow label concept should be abandoned, here's the gambit.

--
Töma

---

From: Vasilenko Eduard <[email-masked]>
Date: Mon, 9 Aug 2021 08:17:44 +0000

Ø  I, too, believe the practical discussion rather belongs to LKML.
No. RFC 7094 permitted Linux people to do what they did.
They are fully compliant with this standard RFC.
They are not compliant with RFC 6437 but it is not their problem to sync RFCs.

From: ietf [mailto:[email-masked]] On Behalf Of Töma Gavrichenkov
Sent: Sunday, August 8, 2021 10:28 PM
To: David Farmer <[email-masked]>
Cc: Tom Herbert <[email-masked]>; 6man WG <[email-masked]>; IETF discussion list <[email-masked]>
Subject: Re: IPv6 Anycast has been killed by LINUX patch in 2016 - who cares?

Peace,
On Sun, Aug 8, 2021, 10:03 PM David Farmer <[email-masked]<mailto:[email-masked]>> wrote:
While I agree with you, anycast is an important capability in the Internet architecture

This is sort of an achievement already given some reckless claims on the IETF mailing list!  Thank you.

nevertheless it has many limitations, and is not the panacea you claim it to be, even for DDoS.

If you consider the client-server model, it *is* the panacea, fundamentally, but I agree that p2p and other models of communications might require something else.  My bad, didn't consider these before.

Furthermore, I’m not sure what you or the original reporter of this problem expect the IETF to do to fix the problem that was reported.

Honestly, I've just been answering the questions other people have, sharing my experience.

--
Töma

---

From: Vasilenko Eduard <[email-masked]>
Date: Mon, 9 Aug 2021 08:56:31 +0000

Hi Phillip,
Philosophy is a bad thing for exact science, for example, engineering.
Let me give you an opposite philosophical example:
If all decisions are cut in stope for ages then mistakes would be only accumulated till the whole system would collapse.
Is it what you imply below pushing “principles”?
You see, it is possible to prove anything by philosophy. Any technological nonsense. And the discussion could be eternal.
Conclusion: Arguments should be only technical and commercial that are verifiable.

Moreover, anycast and multicast are both hacks for IP architecture.
Hence, it is peculiar that you prize multicast but deny anycast.

The rules are important to coordinate people, if everybody would be pulling in different directions at the same time – it would be a mess without any progress.
Hence, I believe that the existence of RFC 7094 is the tombstone for Anycast.
Anycast for stateful traffic is just banned by IETF. Hence, it is better to follow the IETF decision.
Anybody that would assume that Anycast is possible to use (for stateless connections) is acting against standards at his own risk. Do not complain later that your proprietary solution does not work.
Hence, no problem that LINUX has finished it off. Anybody could abuse RFC 7094.
Would like something different – welcome to 7094bis to release this ban.
Just discussion in any alias may help to reach a common understanding but not to release the formal Anycast ban for stateful connections.
Even Linux parch revert is not a full solution to the problem – RFC 7094 is awaiting for the next engineer to abuse it.

Why I believe that Anycast is needed for connections too. IMHO: This general use case is very strong:
The final destination computer would never scale to the size needed.
It would be always many computers, for some use cases many thousands of computers, up to the situation when traffic would be bigger than the capacity of any stateful Load Balancer that may be installed in front of these computers.
Then one would need to load balance on the stateless routers (or even network of routers) to achieve the needed scale for load balancing.
It is exactly what is banned by RFC 7094.
Load balancing by DNS is possible too, but it has many differences – it is not an equal replacement.

Eduard
From: ietf [mailto:[email-masked]] On Behalf Of Phillip Hallam-Baker
Sent: Saturday, August 7, 2021 8:33 PM
To: IETF Discussion Mailing List <[email-masked]>
Subject: The right of way principle... Was: IPv6 Anycast has been killed by

Traffic schemes have a principle that once a right of way is established, any competing mode of transport wanting to cross that right of way is responsible for building the necessary bridges etc.

I raise this not to propose it as a principle for the Internet but rather the opposite, to call it out as an anti-pattern.

If the IAB has an architectural role, it is maintaining the thinness of the thin waist: Keeping complexity out of the Internet core is important and requires vigilance.

The anycast over TCP discussion reminds me of a much older argument in the industry when people who got used to hooking MSDOS system calls at particular addresses were wont to scream like banshees each time a new version of MSDOS came out with the entry points moved.

From an architectural point of view, ANYCAST is not a thing, it is a hack. A hack that is useful but one that was never guaranteed to support TCP. The Internet architecture was designed to support UNICAST. The fact that architecture allows us to share an IP address between hosts within a BGN ASN is interesting and useful but not a core commitment by the architecture. If people want to build stuff on that quicksand, so be it. But those who do so have to understand they are building within very limited constraints and there is no long term commitment to support any use they might establish outside those constraints.

Sure, there will be people who have come up with some ingenious scheme that allows systems to work because the deployed infrastructure currently provides stronger guarantees. But fortunately the IETF does not have the ability to insist such guarantees be continued so we really don't have to debate whether we should do something we can't.

The cost of insisting that squatters rights be preserved would be a gradual ossification of the core as every change is required to preserve the rights of the owners of riparian rights.

As for QUIC, Christian is half right. Yes, QUIC does behave in a way that is more compatible with ANYCAST. But QUIC was never designed to work in that way and hence does not guarantee it will work that way.

Fortunately, QUIC is not the replacement for TCP, it is one tool that does the same job and the most important single change in QUIC is that from this point on every application can choose their own transport. The principle that the platform providers are gatekeepers for transport provision has been broken.

Censorship resistant protocols is a very longstanding interest of mine and I would like to see innovation in that area. I really can't see much that ANYCAST brings to the table beyond the discovery phase but Multicast certainly interests me a lot.

121) One to One
12M) One to Many
M2M) Many to Many

The last is of course the holy grail for some people, peer to peer goodness and all that. As an applications guy, I have absolutely no interest in that type of transport, reliable or not. Even when my application is peer to peer.

A reliable transport with DDoS resistance properties supporting 121 and 12M that could make use of multicast and hide the fact it had been used from the application layer would be very very useful.

QUIC arrived at IETF as an optimization that Google was using and wanted to standardize. Which is a completely valid mode of IETF work. But if we want to do interesting stuff, QUIC is not the test bed to build upon. We need to do research first and challenge the prior assumptions baked in before the effort began.

PHB

---

From: Michael Tuexen <[email-masked]>
Date: Mon, 9 Aug 2021 11:16:11 +0200

> On 8. Aug 2021, at 21:23, Tom Herbert <[email-masked]> wrote:
> 
> 
> 
> Either you have multiple traffic termination points on the net (a.k.a. anycast), each as close to some traffic generation point as possible, or you'll end up having capacity overload around your last mile.  This is the equation fundamental to the Internet, while the implementation issues discussed here are hardly more than just typical software engineering tasks.
> 
> Anycast is only one of several mitigation strategies for DDoS, yes, it is a good one for web type services, it might even be the best for that type of service, especially against large volumetric attacks. However, there are many other types of attacks to protect against and services that need protection and anycast is a lousy mitigation strategy for many of them, especially for client networks or peer to peer services. 
> 
> Furthermore, I’m not sure what you or the original reporter of this problem expect the IETF to do to fix the problem that was reported. I’ll remind you of the well worn trope, “the IETF is not the protocol police.” Any fix to the problem reported is squarely in hands of Linux developers, not the IETF.
> 
> David,
> 
What about making the number of retransmission timeouts after which you change the flow label
configurable via a socket option, having the default value configurable via sysctl.

The value of 0 could mean that you don't change the flow label at all due to retransmission
timeouts.

Maybe use 0 as the default value.

Just to be crystal clear: the same behaviour applies to retransmissions of SYN/SYN-ACK segments
and to segments which have no the SYN bit set.

Best regards
Michael

---

From: Michael Tuexen <[email-masked]>
Date: Mon, 9 Aug 2021 11:18:33 +0200

> On 8. Aug 2021, at 21:28, Töma Gavrichenkov <[email-masked]> wrote:
> 
> Peace,
> 
No sure. In my view this feature of Linux seems to have some use cases at least in some specific environments.
So it might make sense to implement it also in other OSes, which might be used in similar use cases.

Best regards
Michael

---

From: Brian Carpenter <[email-masked]>
Date: Mon, 9 Aug 2021 21:40:43 +1200

RFC7094 is not a standard. Also, RFC6437 does not actually REQUIRE a single
flow label per transport session.
IMHO this is an area where experience with running code is more important
than words in RFCs.

Regards,
    Brian Carpenter

On Mon, 9 Aug 2021, 20:18 Vasilenko Eduard, <[email-masked]>
wrote:

> *From:* ietf [mailto:[email-masked]] *On Behalf Of *Töma
> Gavrichenkov
> *Sent:* Sunday, August 8, 2021 10:28 PM
> *To:* David Farmer <[email-masked]>
> *Cc:* Tom Herbert <[email-masked]>; 6man WG <[email-masked]>; IETF
> discussion list <[email-masked]>
> *Subject:* Re: IPv6 Anycast has been killed by LINUX patch in 2016 - who
> cares?
>
>
>
> Peace,
>
> problem expect the IETF to do to fix the problem that was reported.
>
>
>

---

From: Töma Gavrichenkov <[email-masked]>
Date: Mon, 9 Aug 2021 13:47:53 +0300

Peace,

> Which of the top 5, 10, 100 sites on the Internet use anycast?

You should understand that this is a wrong question to ask, because there's
just no way of reliably figuring that out.

Anycast isn't just something which is written all over your BGP
announcement.  By the nature of it, anycast is the announcement of the same
IP prefix, through BGP, from multiple physical locations.  And, the concept
of a "physical location" is not incorporated within BGP or any globally
available network layer protocol.

You can, probably, carry a research, of course, to a certain level of
reliability only, using something like hundreds of RIPE Atlas probes with a
good geographic AND source network distribution (not the same thing), and
measure which IP flows land within which ranges of expected intervals of
time.  Based if the value of the speed of light, it will then show you
(with some level of reliability) which sites of the group certainly use
anycast, and there's no real way of telling if any of them don't, because
the locations could be just too close to each other.

That is a massive piece of work, and I hope you didn't just suggest that
I'd do it, right?
Anyhow, this doesn't mean a lot, because:

> wouldn't that tend to blow a gigantic, gaping hole in your assertion?

A gigantic, gaping hole in my assertion and experience would be blown by
anyone who's ready to come up with an autonomous system architecture, able
to reliably process and mitigate stateful layer 7-enabled (including
combined vectors) DDoS attacks towards a layer 7 network service with no
(or, insignificant) impact to the legitimate users of the service, with no
particular scrubbing centers likely to overload during the attack, without
anycast.

So far, no one was able to even draft this after a week of chatting,
grumbling, and architecture astronautics.

--
Töma

---

From: Nick Hilliard <[email-masked]>
Date: Mon, 9 Aug 2021 12:29:24 +0100

Töma Gavrichenkov wrote on 09/08/2021 11:47:
> anyone who's ready to come up with an autonomous system architecture, 
> able to reliably process and mitigate stateful layer 7-enabled 
> (including combined vectors) DDoS attacks towards a layer 7 network 
> service with no (or, insignificant) impact to the legitimate users of 
> the service, with no particular scrubbing centers likely to overload 
> during the attack, without anycast.

no doubt about it, ddos mitigation without ddos mitigation design and 
infrastructure is not easy.

Nick

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Mon, 9 Aug 2021 12:46:56 -0400

We have people vigorously asserting that Linux broke IPv6 TCP over Anycast
five years ago and this is serious.

And We have people vigorously asserting that TCP over Anycast works
absolutely perfectly and there are no issues.

And they are the same people. It cannot work and not work at the same time.

As a user, I find that a lot of the commercial systems I end up having to
use are considerably less reliable than they should be. Running a real time
trading desktop over a 1Gbs pipe to a big name brokerage is not 100%
reliable which is rather surprising.

What I am objecting to here is the assumption that people can make use of
some undocumented feature that was never guaranteed to work and then insist
that the rest of the Internet maintain that feature in perpetuity. Nope,
does not work that way because however important your application is to
YOU, it is almost certainly not important to ME.

The core assumption of the Internet is that the narrow waist is an
unreliable transport which makes no guarantees. That is why TCP exists in
the first place.

Like BitCoin, deployment of TCP turns out to be immutable. Whatever clever
features people might think they can add, they cannot because TCP is a
platform feature and requires deep operating system voodoo to change.

So if you want your applications to work, you need to think about moving
away from TCP to a modern transport over UDP that takes account of the
additional capabilities and features of the network as deployed.

I would not design a DDoS resistant transport over IPv6 the same way as
over IPv4. What a waste! With 128 bits of addressing, why on earth give
everyone the same target IP address at the discovery layer??? Of course I
am going to be mixing it up.

What this discussion is doing that is useful is surfacing some of the
operational assumptions that are not supported in the infrastructure but
relied on by real installations. We will never have an exhaustive set of
such assumptions.

At the end of the day, there are two operational arguments that invariably
turn out to be false:

1) This change I am proposing to the infrastructure is so important it MUST
be deployed.
2) This feature I rely on is so important it MUST NOT ever change.

What is really strange is that these arguments seem to be the ones most
commonly made in BOFs and WG kickoffs. We have 30 years of experience
telling us that both are false but they still come up, over and over again.

Making a protocol dependent on deployment of a new DNS record is a really
good way to make sure it will never see deployment and I speak as the
author of the only new DNS record to be successfully deployed since SRV
(nope, TLSA still not in C-Panel, most people can't use it, take a look at
the business model and you will see why that isn't happening). Making a
protocol dependent on TCP fast start was another wrecking choice. The list
goes on.

An internetwork is not the same thing as a network. tThe IETF is not the
administrator of the internetwork because the distinctive feature of an
Internetwork is that there is no administrator.

---

From: Masataka Ohta <[email-masked]>
Date: Tue, 10 Aug 2021 01:53:30 +0900

Phillip Hallam-Baker wrote:

> absolutely perfectly

Wrong. It is merely that TCP over unicast does not work perfectly.

							Masataka Ohta

---

From: Töma Gavrichenkov <[email-masked]>
Date: Mon, 9 Aug 2021 20:07:06 +0300

Peace,

On Mon, Aug 9, 2021, 7:47 PM Phillip Hallam-Baker <[email-masked]>
wrote:

> And they are the same people.
>

a) they're not really the same people,

b) no one said that TCP works _perfectly_ over anycast per se, because it's
understood that perfectionism just doesn't belong in the area or
engineering.
What's been actually said is that it works just fine in a number of
applications, including almost every popular application, and these
applications use it this way on purpose,

c) also, the impact of IPv6 deployment and performance issues is perfectly
limited by the current poor scale of IPv6 deployment.
And, when the existing behaviour of applications working just fine in IPv4
breaks during transition to IPv6, that's not really going to speed the
transition up.

--
Töma

---

From: Warren Kumari <[email-masked]>
Date: Mon, 9 Aug 2021 14:07:53 -0400

On Mon, Aug 9, 2021 at 1:08 PM Töma Gavrichenkov <[email-masked]> wrote:

>> We have people vigorously asserting that Linux broke IPv6 TCP over
>> Anycast five years ago and this is serious.
>>
> b) no one said that TCP works _perfectly_ over anycast per se, because
> it's understood that perfectionism just doesn't belong in the area or
> engineering.

... including a number of content providers.
As examples (many aren't really documented), Fastly (
https://docs.fastly.com/en/guides/using-fastly-with-apex-domains) and
CloudFlare (
https://www.cloudflare.com/learning/cdn/glossary/anycast-network/,
https://blog.cloudflare.com/cloudflares-architecture-eliminating-single-p/)
have offered this.
Fastly and CloudFlare both have some really smart people working for them,
and they collect and analyze lots of transport level stats. I suspect that
they'd be surprised to hear that what they've built doesn't work reliably...

I'm often surprised just how often we end up in discussions in the IETF
where people make an assertion like "Foo will never work. Can't be done, no
way, no how.", and then someone else points at a bunch of existing
implementations. This feels like another instance of this.

W

---

From: Tom Herbert <[email-masked]>
Date: Mon, 9 Aug 2021 11:23:53 -0700

On Mon, Aug 9, 2021 at 11:09 AM Warren Kumari <[email-masked]> wrote:
>> On Mon, Aug 9, 2021, 7:47 PM Phillip Hallam-Baker <[email-masked]> wrote:
>>>
>>> We have people vigorously asserting that Linux broke IPv6 TCP over Anycast five years ago and this is serious.
>>>
>>> And We have people vigorously asserting that TCP over Anycast works absolutely perfectly and there are no issues.
>>>
>> b) no one said that TCP works _perfectly_ over anycast per se, because it's understood that perfectionism just doesn't belong in the area or engineering.
>> What's been actually said is that it works just fine in a number of applications, including almost every popular application, and these applications use it this way on purpose,
>
>
> As examples (many aren't really documented), Fastly (https://docs.fastly.com/en/guides/using-fastly-with-apex-domains) and CloudFlare (https://www.cloudflare.com/learning/cdn/glossary/anycast-network/, https://blog.cloudflare.com/cloudflares-architecture-eliminating-single-p/) have offered this.
> Fastly and CloudFlare both have some really smart people working for them, and they collect and analyze lots of transport level stats. I suspect that they'd be surprised to hear that what they've built doesn't work reliably...
>
> I'm often surprised just how often we end up in discussions in the IETF where people make an assertion like "Foo will never work. Can't be done, no way, no how.", and then someone else points at a bunch of existing implementations. This feels like another instance of this.

Warren,

That logic works both ways. The fact that these patches in question
are over five years old and no one has reported a production issue
with them is inconsistent with some of the dire proclamations in this
thread that anycast is broken. Note that the concerns were raised by a
researcher not by any production data, and also note that the authors
of the patches were from Google and Facebook which obviously have a
vested interest in not creating problems on the Internet. In fact,
these points are likely to be counter arguments brought up on LKML if
we try to change the Linux behavior, Linux maintainers are often
loathe to change a deployed default without a strong rationale that
there's a real problem in deployment.

Tom

>
> W
>
>>
>>
>> c) also, the impact of IPv6 deployment and performance issues is perfectly limited by the current poor scale of IPv6 deployment.
>> And, when the existing behaviour of applications working just fine in IPv4 breaks during transition to IPv6, that's not really going to speed the transition up.
>>
>> --
>> Töma
>
>
>
> --

---

From: Vasilenko Eduard <[email-masked]>
Date: Mon, 9 Aug 2021 18:42:50 +0000

Hi Tom,
The problem may be overlooked for years
Because not many people use Linux as a desktop with a browser.
And even after this RTO should happen that is not mandatory for the majority of connections - the network should be overloaded for this.
It is more important for this matter what is implemented on Windows and Android.
Eduard
-----Original Message-----
From: ietf [mailto:[email-masked]] On Behalf Of Tom Herbert
Sent: Monday, August 9, 2021 9:24 PM
To: Warren Kumari <[email-masked]>
Cc: Theodore Ts'o <[email-masked]>; Phillip Hallam-Baker <[email-masked]>; 6man WG <[email-masked]>; IETF discussion list <[email-masked]>
Subject: Re: IPv6 Anycast has been killed by LINUX patch in 2016 - who cares?

Warren,

That logic works both ways. The fact that these patches in question are over five years old and no one has reported a production issue with them is inconsistent with some of the dire proclamations in this thread that anycast is broken. Note that the concerns were raised by a researcher not by any production data, and also note that the authors of the patches were from Google and Facebook which obviously have a vested interest in not creating problems on the Internet. In fact, these points are likely to be counter arguments brought up on LKML if we try to change the Linux behavior, Linux maintainers are often loathe to change a deployed default without a strong rationale that there's a real problem in deployment.

Tom

---

From: Tom Herbert <[email-masked]>
Date: Mon, 9 Aug 2021 11:46:08 -0700

Eduard,

Android runs Linux. It's likely these patches are deployed on over a
billion devices at this time.

Tom

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Mon, 9 Aug 2021 14:46:22 -0400

> implementations. This feels like another instance of this.

The starting point for this is the assertion that Linux broke this five
years ago. Either it works or it does not.

While I agree with what you are saying about Fastly, Cloudflare etc, I am
very much aware of what they are doing. But you are overlooking one very
important qualifier, they didn't just deploy and forget, they are
actively monitoring. and adapting their approach to reflect changing
circumstances.

If someone is going to sell any mitigation measure for any form of attack,
they are going to have to continuously monitor performance or they will be
quickly overwhelmed.

What I am saying is that there is a difference between an undocumented
feature being found to work and a perpetual commitment to making it work.
If you are basing your business model on such a feature fine, just be
prepared to adapt if circumstances change.

Our job here is not to nag people into operating their infrastructure in a
particular way. Our job here is to design an infrastructure that is robust
in the face of incompetence, stupidity, greed, lust and technical failures.

One lesson that most of us in the security area have learned but some
obstinately refuse to learn is that it is the fault of implementers if the
user can't use a system securely and it is the fault of designers and
architects if zero effort security is not possible.

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Mon, 9 Aug 2021 14:54:44 -0400

On Mon, Aug 9, 2021 at 2:46 PM Tom Herbert <[email-masked]> wrote:

> > And even after this RTO should happen that is not mandatory for the
> majority of connections - the network should be overloaded for this.

Possibly. Does Android run the vanilla Linux IPv6 stack or something rather
smaller of their own devising.

Thing is that Linux tends to be latched on to in IETF terms as being the
vanguard of new protocol deployment because patches get pushed out quickly.
Meanwhile macOS and Windows frequently lag in feature deployment by up to a
decade.

While the assumption is commonly made here that this behavior is laziness
and it is the cost of implementation that causes deployment to lag, my
conversations with engineers and very senior managers tells me that in some
cases the lag is because their evaluation of the feature leads them to
consider it should not be deployed at all.

So no, I would not assume that features from Linux automatically make their
way into anything. Not without actually looking at the running code.

---

From: Warren Kumari <[email-masked]>
Date: Mon, 9 Aug 2021 14:54:40 -0400

On Mon, Aug 9, 2021 at 2:24 PM Tom Herbert <[email-masked]> wrote:

> > Fastly and CloudFlare both have some really smart people working for
> them, and they collect and analyze lots of transport level stats. I suspect
> that they'd be surprised to hear that what they've built doesn't work
> reliably...
> >
> thread that anycast is broken.

Yes.... No..... Maybe.... I cannot tell if we are in violent agreement here
or not.

There are dire proclamations that IPv6 TCP anycast cannot work.... And yet
there are a bunch of existing implementations where it is clearly working,
and people have built their business models around it, showing that it is
working fine.

I think that we are in violent agreement; the bit that gave me pause was
the "that logic works both ways", which made me think that you thought I
was using it to argue the other side?

Note that the concerns were raised by a
> vested interest in not creating problems on the Internet.

Yup. Nod. Fully agree.

We've actually had a really similar discussion back in 2016 --
https://mailarchive.ietf.org/arch/msg/v6ops/NLb9D5HBWDubS52EoNrdK8gqzqI/

In that thread you said:
"The flow label for a connection may change if the connection
is failing in hopes of finding a better route-- either the networking
stack detects a bad route (i.e. TCP is retransmitting) or userspace
can request a route change if it has information about path quality.
So flow labels are not necessarily persistent which probably makes
flow label filtering a bad idea at least if persistence for the
lifetime of a connection is required for that"

It's clear what the intent was (it's described in the patch as well).

> In fact,

Indeed. 'tis been working like this for many years, and the Internet is not
on fire.

Ok, so, we *are* in violent agreement :-)

W

>
> Tom
>
> >
> > W
> >
> >>
> >>
> >> c) also, the impact of IPv6 deployment and performance issues is
> perfectly limited by the current poor scale of IPv6 deployment.
> >> And, when the existing behaviour of applications working just fine in
> IPv4 breaks during transition to IPv6, that's not really going to speed the
> transition up.
> >>
> >> --
> >> Töma
> >
> >
> >
> > --

---

From: Viktor Dukhovni <[email-masked]>
Date: Mon, 9 Aug 2021 15:25:43 -0400

> they are going to have to continuously monitor performance or they will be

Right, unmonitored security is an oxymoron.

> architects if zero effort security is not possible.

I don't think you can literally have "zero effort", since monitoring is
never zero effort.  The closest thing we know how to do at close to
"zero effort" is unauthenticated opportunistic security, as a defence
against pervasive monitoring.  And even there, the server operator still
has to make some minimal effort to ensure the TLS stack is not broken.

-- 
    Viktor.

---

From: Christian Huitema <[email-masked]>
Date: Mon, 9 Aug 2021 12:40:29 -0700

> and people have built their business models around it, showing that it is

Or something in between. Whether TCP anycast works or not depends on how 
long one needs the TCP connection to last. If the connection lasts long 
enough, route flaps end up happening, and then the TCP connection 
breaks. How long that lasts also depends on how long the route is. If 
the anycast destination is just one AS hop away from the TCP source, 
then you can expect BGP paths to be fairly stable. If the anycast target 
is multiple ASes away, the frequency of route flaps is going to increase 
markedly.

For big CDN deployments, or for the edge servers of big tech companies, 
there are many servers and the AS paths are quite short. TCP sessions 
will mostly work. On the other hand, if you start a deployment with just 
a couple servers, say one per continent, the AS paths will be long and 
the rate of TCP failure will be high.

-- Christian Huitema

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Mon, 9 Aug 2021 15:55:38 -0400

> has to make some minimal effort to ensure the TLS stack is not broken.

Years ago, I started talking about 'zero trust' as something that was
reachable by using cryptography to encrypt data at rest in the cloud. I
don't believe that to be the best approach any more. Trusting two parties
50% is much better than trusting one party 100% and another 0%. Since some
actions inherently require trust, all you can do is to limit the scope and
manage it.

But even so, zero trust is a pretty useful marketing slogan that gets an
idea across and I discovered someone else used it six months before I did
so I am in the clear.

Zero effort security is the amount of security that can be delivered
without a net effort from the users or the system administrators. I think I
can provide quite a bit - full end to end encryption of data in transit and
at rest.

Is that enough for every application? Of course not. The CIA/NSA etc will
always have more comprehensive needs and they are going to need that
feedback info. But for regular commercial applications with regular people,
that is the best enterprise security we are going to get in the next decade
or so.

---

From: Robert Raszuk <[email-masked]>
Date: Mon, 9 Aug 2021 22:04:45 +0200

Hi Warren,

There are dire proclamations that IPv6 TCP anycast cannot work....

Maybe I missed it but I do not recall anyone said this here.

What have been said is that due to hashing or routing changes you may end
up in different endpoint of the same IP address during TCP session
lifetime.

Now two things needs to be observed:

A)  That endpoint may not be ultimate dst server but LB, application
gateway or smart proxy fully aware about such events and automagically deal
with session recreation and transparent to the user re-authentication such
that eyeball never sees a glitch.

B) Anycast for company  X != anycast for company Z

Take Google - Company x  - they have such great coverage and presence in
all most important IXPs that path from any user to their entrance is just
few hops away. That means that routing changes which can impact the anycast
flows are near zero. Same for different hashing ... worse what can happen
to you you will end up on different port of IXP edge router. That all means
that in such case anycast is here for user convenience and will work just
fine.

In contrast take Company B which will inject same /24 from SF, NY, FRA, CDG
& SIN ... First imagine FRA & CDG - if any of those happen to be served by
same ISP then it is already pretty random where the packet will end up.
Such ISP single link flap may switch to different BGP best path and take
opposite exit.

Bottom line is that when you build a solution consideration needs to be
done at all levels from kernel app runs on via customer base and ISPs/IXPs
involved in plumbing the packets.

Best,
R.

---

From: Warren Kumari <[email-masked]>
Date: Mon, 9 Aug 2021 16:12:43 -0400

On Mon, Aug 9, 2021 at 3:40 PM Christian Huitema <[email-masked]>
wrote:

> will mostly work.

Yup. In many cases the traffic to/from CDN is also HTTP/similar, and the
application layer either expects to have short lived connections, and / or
handles retries gracefully.
If the path changes while my browser is fetching
https://www.example.com/favicon.ico, no-harm, no-foul -- my browser gets
the RST and tries again.

> On the other hand, if you start a deployment with just

Yup, or if I were, for example, doing SSH over TCP over IPv6 over Anycast
IP (for some weird convoluted reason :-)). But, that would be silly, and I
wouldn't try and do it in v4 either...

W

---

From: Warren Kumari <[email-masked]>
Date: Mon, 9 Aug 2021 16:16:30 -0400

On Mon, Aug 9, 2021 at 4:04 PM Robert Raszuk <[email-masked]> wrote:

> In contrast take Company B which will inject same /24 from SF, NY, FRA,
> CDG & SIN ... First imagine FRA & CDG - if any of those happen to be served
> by same ISP then it is already pretty random where the packet will end up.

Indeed. I just wish you'd posted this last sentence right when the thread
started; it would have prevented the needless annihilation of many
electrons...

W

---

From: "Theodore Ts'o" <[email-masked]>
Date: Mon, 9 Aug 2021 23:26:25 -0400

> Maybe I missed it but I do not recall anyone said this here.

How about the orignal message which kicked off this entire thread?
Including the hyperbolic subject line:

*** IPv6 Anycast has been killed by LINUX patch in 2016 - who cares? ***

    	 	     	  	    	  - Ted

---

From: Gyan Mishra <[email-masked]>
Date: Tue, 10 Aug 2021 02:29:11 -0400

Forgive me for top posting - as this thread has forked into many different
topics I try will hit a few of the topics including the subject heading
that started the thread …here goes!!

Starting with the subject heading ..

“ Anycast has been killed by LINUX patch in 2016”

Tom Herbert  -> This is mid way through the thread in response to Brian and
I guess basically this solves this hashing patch backs out  the 2016 patch
making less aggressive hash so not recompute hash on each RTO but only
after a threshold of RTOs - problem solved??

Tom Herbert-> “That's true, however we, Linux developers, certainly value
the input and discussion on IETF lists. IMO, the later patch that started
to recompute the hash on each RTO probably is too aggressive to be the
default behavior on the Internet. I plan to post a patch that makes default
less aggressive by restoring the original default behavior to recompute
hash only after multiple RTOs. The rehash behavior will be configurable by
sysctl and also I'll add a socket option to allow the application to
control the behavior per connection thereby resolving Brian's concern”

Just so I and others understand - the main issue that sparked this thread
is the RFC 7098 use of flow label for TCP application load balancing?

In the thread we transformed the topic into IPV6 flow label TCP Anycast
load balancing, but the original issue was the Linux patch that changed the
hashing for application IPv6 flow label hash for TCP server load balancing
to aggressive each RTO which Tom is changing Default back to multiple
RTOs?  Is that correct ??

RFC 7098 talks about the use of flow label for TCP flow server application
load balancing.  Most load balancing vendors that I know of have their own
variety of methods of load balancing such as round-Robin w/ w/o sticky
Session option, source based load balancing and GEO load balancing based on
an application Geo datatabase which I know of Citrix is which uses MEP
Media Exchange protocol for DNS intercept, GSLB sub domain to be able to
provide DNS reply from GEO application database based on client source
address.

Few other points I will try to address related to ECMP load balancing as
that topic came up in relation to RFC 6437 flow label.

Some general rules and history on L3 router ECMP IGP or EGP load balancing
-: well as other data plane as well in relation to load balancing:

The concept of ECMP load balancing historically has been based on vendor
specific proprietary algorithm ASIC or NPU based and had been flow or
session based load balancing for IPv4 using basic source and destination IP
 as input keys to a hashing function.

For any load balancing to work input key needs to be generated from
information called keys within a packet done on a per packet basis, so that
these keys within the packet can ensure that the packet gets hashed to the
same path to avoid out of order packets. So the per packet input keys in
the case of 5-tuple header hash is SA, DA, SP, DP, Protocol is used to
generate a 20 bit “entropy” label as an input to a hashing function to
provide uniform 50/50 load balancing.

 This basic “entropy” label  principle is applied to MPLS  data plane RFC
6790 SR data plane RFC 8662 and NVO3 VXLAN RFC 7348 source port entropy
label same 5-tuple header hash key info generated random source port and
now the same 5-tuple L3/L4 header hash keys generates the 20 bit flow label
input key to a hashing function to provide uniform 50/50 load balancing.

Most all modern routers support RFC 6437 stateless uniform load balancing
vendor links below of incumbent router / switch vendor stakeholders:

IPv6 flow label RFC 6437 stateless uniform 50/50 load balancing is one
business driver for operators to migrate to SRv6.

Cisco XR
https://www.cisco.com/c/en/us/td/docs/routers/asr9000/software/asr9k-r6-3/ip-addresses/configuration/guide/b-ip-addresses-configuration-guide-asr9000-63x/b-ip-addresses-configuration-guide-asr9000-63x_chapter_0100.html

Juniper:
https://www.juniper.net/documentation/us/en/software/junos/sampling-forwarding-monitoring/topics/concept/hash-computation-mpcs-understanding.html

Nokia:
https://infocenter.nokia.com/public/7750SR215R1A/index.jsp?topic=%2Fcom.nokia.Gx_AVPs_Reference_Guide_21.5.R1%2Fdita_standards.html

Arista:
https://eos.arista.com/ipv6-rfc-compliance/

Huawei:
https://forum.huawei.com/enterprise/en/huawei/m/ViewThread.html?tid=605826&lang=en

BGP convergence metrics based on advertisement and withdrawals and BFD 50ms
detection time and propagation times convergence within seconds:

NANOG 2013 link analysis on convergence times and route propagation
benchmarks:

https://archive.nanog.org/sites/default/files/monday.general.huston.BGP.21.pdf

NANOG 2019 Analysis of internet traffic is 90% CDN traffic:

https://pc.nanog.org/static/published/meetings/NANOG76/1972/20190610_Labovitz_Internet_Traffic_2009-2019_v1.pdf

Anycast:

Anycast proximity based routing has been utilized by operators and
enterprises for more almost 2 decades.

The initial uses of IPv4 Anycast was for DNS followed by NTP as well as
other NMS monitoring services syslog, snmp, IPFIX.

RFC 2526 defines IPv6 Anycast which is the last  7 bits of the IID -
/121-/128  Reserved  for IPv6 Anycast prefixes.

As far as I know I don’t know of any implementations or operators using the
reserved Anycast range for IPv6 TCP Anycast.

I think most all IPv6 Anycast  NMS or Application deployments of Anycast
have been using IPv4 style RFC 7094 proximity routing based IPv6 Anycast.

NANOG link on TCP Anycast:
https://archive.nanog.org/meetings/nanog37/presentations/matt.levine.pdf

As stated in the deck “don’t believe the FUD that TCP Anycast is not
possible”

DDOS and Anycast:
Cloudfare CDN uses TCP Anycast and apparently had DDOS solution using
Anycast.  Makes sense.

https://www.cloudflare.com/learning/cdn/glossary/anycast-network/

As it seems there are proven implementations and deployments of TCP Anycast
both IPv4 application load balancing and IPv6 flow label RFC 6437 / RFC
7098 application load balancing.  My guess is that it works with some
caveats as revealed in the NANOG archive.

I think the question came up in this thread related to L3 ECMP use of RFC
6437 for stateless uniform load balancing and also being able to use RFC
7098 application based  IPv6 GUA TCP or IPv6 Anycast TCP load balancing
simultaneously.  I think that would work as the TCP load balancing is at L4
and L3 ECMP is at L3, and they both reference the per packet 5-tuple
generated 20 bit key for both L3 and L4-7 load balancing.  I think what
would not work in my mind is RFC 6437 stateful signaling from a edge host
for flow label value as that would get overwritten by the per packet
5-tuple stateless 20 bit input key to the 50/50 load balancing hashing
function.

We are thinking of using IPv6 flow label for APN edge marking, which would
be stateful edge signaling from APN edge node,  however  that maybe
difficult per above issue with stateless stepping on stateful.

Kind Regards

Gyan

-- 

<http://www.verizon.com/>

*Gyan Mishra*

*Network Solutions A**rchitect *

*Email [email-masked] <[email-masked]>*

*M 301 502-1347*

---

From: Vasilenko Eduard <[email-masked]>
Date: Tue, 10 Aug 2021 09:16:59 +0000

Hi Gyan,
Thanks. It is no the summary - It is a much bigger pack of information.
Especially this one is the direct answer to my original question:
NANOG link on TCP Anycast: https://archive.nanog.org/meetings/nanog37/presentations/matt.levine.pdf
As stated in the deck “don’t believe the FUD that TCP Anycast is not possible”
Eduard
From: ipv6 [mailto:[email-masked]] On Behalf Of Gyan Mishra
Sent: Tuesday, August 10, 2021 9:29 AM
To: Theodore Ts'o <[email-masked]>
Cc: Phillip Hallam-Baker <[email-masked]>; 6man WG <[email-masked]>; IETF discussion list <[email-masked]>
Subject: Re: IPv6 Anycast has been killed by LINUX patch in 2016 - who cares?

Forgive me for top posting - as this thread has forked into many different topics I try will hit a few of the topics including the subject heading that started the thread …here goes!!

Tom Herbert  -> This is mid way through the thread in response to Brian and I guess basically this solves this hashing patch backs out  the 2016 patch making less aggressive hash so not recompute hash on each RTO but only after a threshold of RTOs - problem solved??

Tom Herbert-> “That's true, however we, Linux developers, certainly value the input and discussion on IETF lists. IMO, the later patch that started to recompute the hash on each RTO probably is too aggressive to be the default behavior on the Internet. I plan to post a patch that makes default less aggressive by restoring the original default behavior to recompute hash only after multiple RTOs. The rehash behavior will be configurable by sysctl and also I'll add a socket option to allow the application to control the behavior per connection thereby resolving Brian's concern”

Just so I and others understand - the main issue that sparked this thread is the RFC 7098 use of flow label for TCP application load balancing?

In the thread we transformed the topic into IPV6 flow label TCP Anycast load balancing, but the original issue was the Linux patch that changed the hashing for application IPv6 flow label hash for TCP server load balancing to aggressive each RTO which Tom is changing Default back to multiple RTOs?  Is that correct ??

RFC 7098 talks about the use of flow label for TCP flow server application load balancing.  Most load balancing vendors that I know of have their own variety of methods of load balancing such as round-Robin w/ w/o sticky Session option, source based load balancing and GEO load balancing based on an application Geo datatabase which I know of Citrix is which uses MEP Media Exchange protocol for DNS intercept, GSLB sub domain to be able to provide DNS reply from GEO application database based on client source address.

Few other points I will try to address related to ECMP load balancing as that topic came up in relation to RFC 6437 flow label.

Some general rules and history on L3 router ECMP IGP or EGP load balancing -: well as other data plane as well in relation to load balancing:

The concept of ECMP load balancing historically has been based on vendor specific proprietary algorithm ASIC or NPU based and had been flow or session based load balancing for IPv4 using basic source and destination IP  as input keys to a hashing function.

For any load balancing to work input key needs to be generated from information called keys within a packet done on a per packet basis, so that these keys within the packet can ensure that the packet gets hashed to the same path to avoid out of order packets. So the per packet input keys in the case of 5-tuple header hash is SA, DA, SP, DP, Protocol is used to generate a 20 bit “entropy” label as an input to a hashing function to provide uniform 50/50 load balancing.

 This basic “entropy” label  principle is applied to MPLS  data plane RFC 6790 SR data plane RFC 8662 and NVO3 VXLAN RFC 7348 source port entropy label same 5-tuple header hash key info generated random source port and now the same 5-tuple L3/L4 header hash keys generates the 20 bit flow label input key to a hashing function to provide uniform 50/50 load balancing.

Most all modern routers support RFC 6437 stateless uniform load balancing vendor links below of incumbent router / switch vendor stakeholders:

IPv6 flow label RFC 6437 stateless uniform 50/50 load balancing is one business driver for operators to migrate to SRv6.

Cisco XR

Juniper:

Nokia:

Arista:

Huawei:

BGP convergence metrics based on advertisement and withdrawals and BFD 50ms detection time and propagation times convergence within seconds:

NANOG 2013 link analysis on convergence times and route propagation benchmarks:

Anycast:

Anycast proximity based routing has been utilized by operators and enterprises for more almost 2 decades.

The initial uses of IPv4 Anycast was for DNS followed by NTP as well as other NMS monitoring services syslog, snmp, IPFIX.

RFC 2526 defines IPv6 Anycast which is the last  7 bits of the IID - /121-/128  Reserved  for IPv6 Anycast prefixes.

As far as I know I don’t know of any implementations or operators using the reserved Anycast range for IPv6 TCP Anycast.

I think most all IPv6 Anycast  NMS or Application deployments of Anycast have been using IPv4 style RFC 7094 proximity routing based IPv6 Anycast.

As stated in the deck “don’t believe the FUD that TCP Anycast is not possible”

DDOS and Anycast:
Cloudfare CDN uses TCP Anycast and apparently had DDOS solution using Anycast.  Makes sense.

As it seems there are proven implementations and deployments of TCP Anycast both IPv4 application load balancing and IPv6 flow label RFC 6437 / RFC 7098 application load balancing.  My guess is that it works with some caveats as revealed in the NANOG archive.

I think the question came up in this thread related to L3 ECMP use of RFC 6437 for stateless uniform load balancing and also being able to use RFC 7098 application based  IPv6 GUA TCP or IPv6 Anycast TCP load balancing simultaneously.  I think that would work as the TCP load balancing is at L4 and L3 ECMP is at L3, and they both reference the per packet 5-tuple generated 20 bit key for both L3 and L4-7 load balancing.  I think what would not work in my mind is RFC 6437 stateful signaling from a edge host for flow label value as that would get overwritten by the per packet 5-tuple stateless 20 bit input key to the 50/50 load balancing hashing function.

We are thinking of using IPv6 flow label for APN edge marking, which would be stateful edge signaling from APN edge node,  however  that maybe difficult per above issue with stateless stepping on stateful.

Kind Regards

Gyan

On Mon, Aug 9, 2021 at 11:28 PM Theodore Ts'o <[email-masked]<mailto:[email-masked]>> wrote:

                                          - Ted

--------------------------------------------------------------------
IETF IPv6 working group mailing list
[email-masked]<mailto:[email-masked]>
Administrative Requests: https://www.ietf.org/mailman/listinfo/ipv6
--------------------------------------------------------------------
--

[http://ss7.vzw.com/is/image/VerizonWireless/vz-logo-email]<http://www.verizon.com/>

Gyan Mishra

Network Solutions Architect

Email [email-masked]<mailto:[email-masked]>

M 301 502-1347

---

From: Töma Gavrichenkov <[email-masked]>
Date: Tue, 10 Aug 2021 12:55:00 +0300

Peace,

On Tue, Aug 10, 2021, 9:31 AM Gyan Mishra <[email-masked]> wrote:

> a patch that makes default less aggressive by restoring the original
> default behavior to recompute hash only after multiple RTOs.
>

Let's now talk about hacks, right?

A flow is basically a stream of similar data within one or more
connections.  This is an application layer concept.  Architecturally, it
may change on a connection if the data flow within the connection changes.

E.g. we've established a connection to [youtube DNS A entry]:443,
downloaded the hypertext, but now we're going to reuse the same established
connection to stream video, so the network should better treat that
connection somehow differently now.

The flow label was never supposed to be a legitimate control over routing.
It shouldn't change over one, two, or a hundred RTOs.  It generally only
changes when the flow becomes different.
I believe this was so obvious to the authors of the original specification
in 2003 that they even forgot to actually state it.

What Tom proposed is, of course, way better than how it works now.
Especially the socket option — yay, Linux is finally going to implement the
"MUST" in RFC3697#3!  We harbour the hope that other operating systems
would do the same good thing.

But the idea I'm trying to drive home is: fixing (temporary) network
delivery issues via the control of a strictly application level feature is
among the dirtiest of the hacks possible.

And it kind of amazes me how people call anycast a hack (while it's
perfectly the behaviour natural to the Internet, a global self-healing
internetwork, as designed in 1970s) and still consider this a legitimate
behaviour.

--
Töma

---

From: Vasilenko Eduard <[email-masked]>
Date: Tue, 10 Aug 2021 12:05:30 +0000

After the hash would be recalculated, the flow may be pushed to the same broken link or node.
Is probability 50% or 75% to repair problem good enough? (for ECMP to 2 or 4 paths)

Should RTO give a chance to IGP to repair the problem in a normal way (1s?)
Ed/
From: ietf [mailto:[email-masked]] On Behalf Of Töma Gavrichenkov
Sent: Tuesday, August 10, 2021 12:55 PM
To: Gyan Mishra <[email-masked]>
Cc: 6man WG <[email-masked]>; Phillip Hallam-Baker <[email-masked]>; Theodore Ts'o <[email-masked]>; IETF discussion list <[email-masked]>
Subject: Re: IPv6 Anycast has been killed by LINUX patch in 2016 - who cares?

Peace,
On Tue, Aug 10, 2021, 9:31 AM Gyan Mishra <[email-masked]<mailto:[email-masked]>> wrote:
a patch that makes default less aggressive by restoring the original default behavior to recompute hash only after multiple RTOs.

A flow is basically a stream of similar data within one or more connections.  This is an application layer concept.  Architecturally, it may change on a connection if the data flow within the connection changes.

E.g. we've established a connection to [youtube DNS A entry]:443, downloaded the hypertext, but now we're going to reuse the same established connection to stream video, so the network should better treat that connection somehow differently now.

The flow label was never supposed to be a legitimate control over routing.  It shouldn't change over one, two, or a hundred RTOs.  It generally only changes when the flow becomes different.
I believe this was so obvious to the authors of the original specification in 2003 that they even forgot to actually state it.

What Tom proposed is, of course, way better than how it works now.  Especially the socket option — yay, Linux is finally going to implement the "MUST" in RFC3697#3!  We harbour the hope that other operating systems would do the same good thing.

But the idea I'm trying to drive home is: fixing (temporary) network delivery issues via the control of a strictly application level feature is among the dirtiest of the hacks possible.

And it kind of amazes me how people call anycast a hack (while it's perfectly the behaviour natural to the Internet, a global self-healing internetwork, as designed in 1970s) and still consider this a legitimate behaviour.

--
Töma

---

From: Gyan Mishra <[email-masked]>
Date: Tue, 10 Aug 2021 08:10:04 -0400

Hi Töma

> may change on a connection if the data flow within the connection changes.

   Gyan> Agreed

> The flow label was never supposed to be a legitimate control over
> routing.  It shouldn't change over one, two, or a hundred RTOs.  It
> generally only changes when the flow becomes different.

    Gyan> Very Good point. So let’s say you have an IPv4 or IPv6 TCP
Anycast connection you should stay on that proximity routed flow throughout
the duration that goes for the long lived TCP.  But now with the Linux hack
we now shift after the first RTO immediately to try a different BGP anycast
path via Linux hack patch and hope for better results in case the first
path was congested or having issues.  This is definitely an application
based network engineering hack by a Linux developer whom had the best
intentions of a application network awareness  self healing network.  From
a technical standpoint as a TCP RST has already been receiving and we are
re-establishing the connection, I am not understanding why this was such a
bad thing understandable that it’s aggressive but the thought process does
makes sense.  The Linux developers thought was that if you got an RTO, then
more then likely that network path is bad and let’s rehash to a different
path immediately.  I can see the down side is that first Anycast path from
a BGP path selection was the best lowest latency path, but now the
application thinks it understands the network better then network engineers
and thinks it’s better to rehash to a different path immediately.  The
MAJOR problem with that is as BGP Anycast is proximity based you could end
up going half was around the world for the second best path and now voila
—> TCP Anycast is now from the Happy Eyeballs (not the RFC 6555) but user
perspective is completely broken thus  the subject heading “IPv6 Anycast
has been killed by Linux patch”.

> would do the same good thing.

Gyan> Given what I stated above I would say let the Network do the
networking and as CDN makes up 90% plus of the internet traffic being GEO
load balanced worldwide, and as we have IETF ALTO WG that does application
based traffic optimization BGP-LS / PCEP CDN  RSVP / SR aware network
optimization based solutions  that  already exist today,  let’s kill the
Linux hack all together.  As the Linux server is completely unaware of
network conditions any rehash is bad thing as that breaks TCP Anycast by
sending you clear around the world when you should be “sticky” based on BGP
Anycast best path selection stay on the optimization proximity based
network path and only shift to alternate BGP path when the path is no
longer available.  Let routing do it’s routing!!

> among the dirtiest of the hacks possible.

  Gyan> Firmly agreed

> internetwork, as designed in 1970s) and still consider this a legitimate

Gyan> Agreed

Gyan> After reading the  feedback from Toma can we not rehash at all for
the Default Linux patch.  See the MAJOR problem that is being created when
you try to rehash with BGP Anycast described above and that basically any
rehash literally breaks IPv6 flow label based TCP Anycast CDN load
balancing.

*Gyan Mishra*

*M 301 502-1347*

---

From: Vasilenko Eduard <[email-masked]>
Date: Tue, 10 Aug 2021 12:34:45 +0000

It is probably too strong: “let’s kill the Linux hack all together”.
IMHO: it should be completely switched off by default. But if some admin would like to use it – let it activate it and use it.
Cross all OSI model optimization looks not the good architecture decision, but if somebody wants to do it – why not.

The original problem that was raised for this Linux feature (original use case):
many vendors already have so bad microcode that it is very often in the big DC environment to have broken PFE that the control plane is not aware of. “Silent drop” up to manual intervention.
Of course, it is better to monitor such a situation in a different way (iOAM, BFD), but if one has already hundreds or thousands of switches – it is not a short-term proposition. Faster work-around is needed.
Ed/
From: ietf [mailto:[email-masked]] On Behalf Of Gyan Mishra
Sent: Tuesday, August 10, 2021 3:10 PM
To: Töma Gavrichenkov <[email-masked]>
Cc: Theodore Ts'o <[email-masked]>; Phillip Hallam-Baker <[email-masked]>; 6man WG <[email-masked]>; IETF discussion list <[email-masked]>
Subject: Re: IPv6 Anycast has been killed by LINUX patch in 2016 - who cares?

Hi Töma

On Tue, Aug 10, 2021 at 5:55 AM Töma Gavrichenkov <[email-masked]<mailto:[email-masked]>> wrote:
Peace,
On Tue, Aug 10, 2021, 9:31 AM Gyan Mishra <[email-masked]<mailto:[email-masked]>> wrote:
a patch that makes default less aggressive by restoring the original default behavior to recompute hash only after multiple RTOs.

   Gyan> Agreed

    Gyan> Very Good point. So let’s say you have an IPv4 or IPv6 TCP Anycast connection you should stay on that proximity routed flow throughout the duration that goes for the long lived TCP.  But now with the Linux hack we now shift after the first RTO immediately to try a different BGP anycast path via Linux hack patch and hope for better results in case the first path was congested or having issues.  This is definitely an application based network engineering hack by a Linux developer whom had the best intentions of a application network awareness  self healing network.  From a technical standpoint as a TCP RST has already been receiving and we are re-establishing the connection, I am not understanding why this was such a bad thing understandable that it’s aggressive but the thought process does makes sense.  The Linux developers thought was that if you got an RTO, then more then likely that network path is bad and let’s rehash to a different path immediately.  I can see the down side is that first Anycast path from a BGP path selection was the best lowest latency path, but now the application thinks it understands the network better then network engineers and thinks it’s better to rehash to a different path immediately.  The MAJOR problem with that is as BGP Anycast is proximity based you could end up going half was around the world for the second best path and now voila —> TCP Anycast is now from the Happy Eyeballs (not the RFC 6555) but user perspective is completely broken thus  the subject heading “IPv6 Anycast has been killed by Linux patch”.

Gyan> Given what I stated above I would say let the Network do the networking and as CDN makes up 90% plus of the internet traffic being GEO load balanced worldwide, and as we have IETF ALTO WG that does application based traffic optimization BGP-LS / PCEP CDN  RSVP / SR aware network optimization based solutions  that  already exist today,  let’s kill the Linux hack all together.  As the Linux server is completely unaware of network conditions any rehash is bad thing as that breaks TCP Anycast by sending you clear around the world when you should be “sticky” based on BGP Anycast best path selection stay on the optimization proximity based network path and only shift to alternate BGP path when the path is no longer available.  Let routing do it’s routing!!

  Gyan> Firmly agreed

Gyan> Agreed

Gyan> After reading the  feedback from Toma can we not rehash at all for the Default Linux patch.  See the MAJOR problem that is being created when you try to rehash with BGP Anycast described above and that basically any rehash literally breaks IPv6 flow label based TCP Anycast CDN load balancing.

--
Töma
--

Gyan Mishra

M 301 502-1347

---

From: Gyan Mishra <[email-masked]>
Date: Tue, 10 Aug 2021 09:09:01 -0400

Hi Eduard

On Tue, Aug 10, 2021 at 8:34 AM Vasilenko Eduard <
[email-masked]> wrote:

> IMHO: it should be completely switched off by default. But if some admin
> would like to use it – let it activate it and use it.
>

    Gyan> Agreed

> Cross all OSI model optimization looks not the good architecture decision,
> but if somebody wants to do it – why not.
>
>  Gyan> Agreed
>
> many vendors already have so bad microcode that it is very often in the
> big DC environment to have broken PFE that the control plane is not aware
> of. “Silent drop” up to manual intervention.
>
> Of course, it is better to monitor such a situation in a different way
> (iOAM, BFD), but if one has already hundreds or thousands of switches – it
> is not a short-term proposition. Faster work-around is needed.
>

     Gyan> I believe the  original problem with hashing reported not in DC
environment but over the internet?  For the general internet scenario  not
rehashing at all as the Default behavior is the best solution.  For the DC
and other scenarios Linux developers can change as they see fit for their
environment is fine.

Ed/
>
> *From:* ietf [mailto:[email-masked]] *On Behalf Of *Gyan Mishra
> *Sent:* Tuesday, August 10, 2021 3:10 PM
> *To:* Töma Gavrichenkov <[email-masked]>
> *Cc:* Theodore Ts'o <[email-masked]>; Phillip Hallam-Baker <
> [email-masked]>; 6man WG <[email-masked]>; IETF discussion list <
> [email-masked]>
> *Network Solutions Architect *
>
-- 

*Gyan Mishra*

*M 301 502-1347*

---

From: Vasilenko Eduard <[email-masked]>
Date: Tue, 10 Aug 2021 13:24:07 +0000

Hi Gyan,
I am not sure why Linux RTO refreshed the flow label initially in 2014-2016.
But the presentation on the last IETF that attracted my attention was about how to mitigate this type of vendor's bug: loss of sync between control plane and data plane. That leads to prolonging “silent drop”.

I have stated in the 1st message: it is a problem that this work-around has become the default for the whole Internet.
I agree that it should not happen by default after any number of RTOs.

But even after activation, it makes sense to give IGP a chance to repair the problem.
IMHO: RTO timer should be configurable or at least 1s to give OSPF a chance.
Because rerouting would happen even if IGP would fix the problem very soon, not only for “hung” PFE.
Eduard
From: Gyan Mishra [mailto:[email-masked]]
Sent: Tuesday, August 10, 2021 4:09 PM
To: Vasilenko Eduard <[email-masked]>
Cc: 6man WG <[email-masked]>; IETF discussion list <[email-masked]>; Phillip Hallam-Baker <[email-masked]>; Theodore Ts'o <[email-masked]>; Töma Gavrichenkov <[email-masked]>
Subject: Re: IPv6 Anycast has been killed by LINUX patch in 2016 - who cares?

Hi Eduard

On Tue, Aug 10, 2021 at 8:34 AM Vasilenko Eduard <[email-masked]<mailto:[email-masked]>> wrote:
It is probably too strong: “let’s kill the Linux hack all together”.
IMHO: it should be completely switched off by default. But if some admin would like to use it – let it activate it and use it.

    Gyan> Agreed

     Gyan> I believe the  original problem with hashing reported not in DC environment but over the internet?  For the general internet scenario  not rehashing at all as the Default behavior is the best solution.  For the DC and other scenarios Linux developers can change as they see fit for their environment is fine.

Ed/
From: ietf [mailto:[email-masked]<mailto:[email-masked]>] On Behalf Of Gyan Mishra
Sent: Tuesday, August 10, 2021 3:10 PM
To: Töma Gavrichenkov <[email-masked]<mailto:[email-masked]>>
Cc: Theodore Ts'o <[email-masked]<mailto:[email-masked]>>; Phillip Hallam-Baker <[email-masked]<mailto:[email-masked]>>; 6man WG <[email-masked]<mailto:[email-masked]>>; IETF discussion list <[email-masked]<mailto:[email-masked]>>
Subject: Re: IPv6 Anycast has been killed by LINUX patch in 2016 - who cares?

Hi Töma

   Gyan> Agreed

  Gyan> Firmly agreed

Gyan> Agreed

--
Töma
--

Gyan Mishra

M 301 502-1347

--

Gyan Mishra

M 301 502-1347

---

From: Gyan Mishra <[email-masked]>
Date: Tue, 10 Aug 2021 12:29:19 -0400

Hi Eduard

> I am not sure why Linux RTO refreshed the flow label initially in

   Gyan> I believe the kernel update / hack was related to rehash of load
balancing to pick the next best proximity BGP Anycast path after first
RTO.  Since it’s load balancing hash related that’s where RFC 6347 flow
label applicability to server load balancing RFC 7098 comes into play.
Which presentation was it on 111.

> But the presentation on the last IETF that attracted my attention was
> about how to mitigate this type of vendor's bug: loss of sync between
> control plane and data plane. That leads to prolonging “silent drop”.
>
> Gyan> Unfortunately I may have stepped away missed that discussion.  So
> was the Linux kernel rehash after first RTO mentioned as a server fix.  A
> much better fix as you and others mentioned is BFD tight timers to detect 1
> way fiber in DC and iOAM.
>
> I have stated in the 1st message: it is a problem that this work-around
> has become the default for the whole Internet.
>
   Gyan> default RTO is I believe 20 seconds.  I agree  it would be nice if
the RTO timer was configurable.

> *From:* Gyan Mishra [mailto:[email-masked]]
>
> *Sent:* Tuesday, August 10, 2021 4:09 PM
> *Cc:* 6man WG <[email-masked]>; IETF discussion list <[email-masked]>;
> Phillip Hallam-Baker <[email-masked]>; Theodore Ts'o <[email-masked]>;
> Töma Gavrichenkov <[email-masked]>
-- 

*Gyan Mishra*

*M 301 502-1347*

---

From: Michael Richardson <[email-masked]>
Date: Tue, 10 Aug 2021 13:05:35 -0400

Warren Kumari <[email-masked]> wrote:
    > There are dire proclamations that IPv6 TCP anycast cannot work.... And
    > yet there are a bunch of existing implementations where it is clearly
    > working, and people have built their business models around it, showing
    > that it is working fine.

It can't work if the anycast mechanism depends upon flow-label remaining the
same during the lifetime of the connection.  It seems that the BGP anycast
that I'm familiar with (as used by many anycast name server systems) can
work, and does work.

Maybe local load-balancers are borked.  There was a talk at netdev about
trying to make the Linux load balancer code more stateless in the face of
hash bucket rebalancing, and I think that work is probably can never work.

I've ignored half of this thread, particularly the half where the quoting was
borked.  It seems to me that there is a v6ops BCP that should get written.

--
Michael Richardson <[email-masked]>   . o O ( IPv6 IøT consulting )
           Sandelman Software Works Inc, Ottawa and Worldwide