Subject: The TCP and UDP checksum algorithm may soon need updating

From: Craig Partridge <[email-masked]>
Date: Thu, 4 Jun 2020 13:12:14 -0600

Hi folks:

This note is intended as an invitation to think a bit about a potential
hard problem.

There's a small body of literature suggesting that the TCP checksum is
regularly failing to detect errors and that we're getting close the point
where using an MD5 authentication check will be insufficient (e.g. the
number of times the TCP checksum fails to detect errors is so large that
TCP passes through enough errors that the md5 check won't catch all of
them).  This situation is due to the growth in both total traffic and the
size of individual data transfers.  This is not a surprise -- it was
anticipated 20 years ago, when studies showed the TCP checksum was quite
weak.

I'm part of a team that is setting out to do a big set of measurements to
figure out if the other reports that we're close to a tipping point are (a)
correct; and (b) what kinds of errors are getting through.  That data will
tell us if a new checksum is warranted.  We hope to know in about a year.
We have time to think.

If we need a new checksum, then we are in an interesting space.  There is a
defined way to negotiate a new checksum in TCP (RFC 1146, now obsolete, but
we can presumably unobsolete it).  But all the middleboxes that like to
muck with the TCP header and data would have to both (a) learn about the
option and (b) implement the new checksum algorithm(s).  Middleboxes are
the problem because if an end system doesn't update the checksum, that's on
the end system owner and their willingness to risk bad data.  But if an end
system updates and can't transfer data due to the middlebox's ignorance,
that's a larger system problem.

Then there's UDP.  UDP has no options.  We could retrofit options by
leveraging the reserved zero checksum value and some magic codes at the
start or end of the UDP data, but that's ugly.  Or we could define a UDPv2
(UDP has no version numbers either!) and give it another IP protocol
number.  But if we don't fix UDP, protocols above UDP, like QIUC, need
fixing...

I don't think we'll need to fix IP and ICMP, as the consequences of
occasional error aren't a big deal for them.  A misrouted packet or
unreadable ICMP in every million packets or so is probably OK.

At any rate, in a spare moment, worth pondering how one might proceed.

Thanks!

Craig

PS: Some folks may wonder if we couldn't protect ourselves by using a
bigger MAC than md5.  Yes but (a) that doesn't solve the problem for
protocols that don't do message authentication; and (b) MACs are lousy
checksums.

That second statement may surprise folks, so here's the one paragraph
point.  A checksum for error checking (i.e. not proof against adversaries)
should be designed to detect all instances of common types of errors and,
for errors other than those common types, detect errors proportionate to
the width (in bits) of the checksum. Thus, for a checksum of width W bits,
we'd expect it to fail to detect an error with a probability of 1 in
2^(W+4) or better.  Some newer checksums may be able to do even better,
like 1 in 2^(2*W). Whereas a MAC of width W bits can only fail to catch
errors with a probability of 1 in 2^W due to the additional requirement to
thwart an adversary (not sure this is a proven property, but it has
consistently been true).  So, for the same width in bits, a checksum
catches many more errors -- and checksums are computationally much cheaper
to compute than MACs.

-- 
*****
Craig Partridge's email account for professional society activities and
mailing lists.

---

From: Joe Touch <[email-masked]>
Date: Thu, 4 Jun 2020 12:33:01 -0700

See draft-ietf-tsvwg-udp-options

> On Jun 4, 2020, at 12:13 PM, Craig Partridge <[email-masked]> wrote:
> 
> Then there's UDP.  UDP has no options.

---

From: Michael Thomas <[email-masked]>
Date: Thu, 4 Jun 2020 12:48:06 -0700

On 6/4/20 12:12 PM, Craig Partridge wrote:
> Hi folks:
>
> This note is intended as an invitation to think a bit about a 
> potential hard problem.
>
> regularly failing to detect errors and that we're getting close the 
> point where using an MD5 authentication check will be insufficient 
> (e.g.. the number of times the TCP checksum fails to detect errors is 
> so large that TCP passes through enough errors that the md5 check 
> won't catch all of them).  This situation is due to the growth in both 
> total traffic and the size of individual data transfers.  This is not 
> a surprise -- it was anticipated 20 years ago, when studies showed the 
> TCP checksum was quite weak.
>
>

How does this interact with (D)TLS? Assuming the error is in the packet 
body which would be most likely for data payloads, the TLS layer would 
detect the error too, right? Obviously ack's coming back would still suffer.

Also: since it's clear that any new and improved checksum is going to 
take forever to get upgraded, do we know what the implications are for 
higher error rates? Is there anything non-linear that could happen if we 
don't fix it, or does it just chug along getting more and more uncaught 
errors due to traffic volumes?

Last given L2, can it be a backstop?

Mike

---

From: Craig Partridge <[email-masked]>
Date: Thu, 4 Jun 2020 14:16:05 -0600

Ah, I somehow had missed that.

Looks like it does the trick if we need it. Not sure if we'd have to update
the optional checksum to use a new checksum too.

Craig

---

From: Richard Barnes <[email-masked]>
Date: Thu, 4 Jun 2020 17:18:52 -0400

Yeah, secure transports already consider the data from the network
untrusted, including TLS and DTLS, but also IPsec/ESP, SSH, SRTP and QUIC.
As long as those protocols are using reasonably modern, AEAD ciphers, the
worst impact of corruption in the transport is DoS.

That said, there might be some pretty hard crashes in cases where the
security protocol relies on the transport for something.  For example, TLS
doesn't have a mechanism for requesting that data be re-sent in the event
of loss/corruption, because it's assumed that TCP provides that.  If
corruption is only detected at the TLS layer, there's not a way to
recover.  That said, I wouldn't be surprised if many TLS stacks close the
connection on bad data anyway.

On the UDP side, there should be no issue.  Anything above UDP has to
tolerate loss anyway, so there should be mechanisms that will require from
corruption-treated-as-loss.

So there may be some feedback loops to close, but that's a more limited,
host-internal problem.

In any case -- one more reason to encrypt everything!

--Richard

On Thu, Jun 4, 2020 at 4:17 PM Craig Partridge <[email-masked]> wrote:

> Looks like it does the trick if we need it. Not sure if we'd have to
> update the optional checksum to use a new checksum too.
>
> Craig
>
> On Thu, Jun 4, 2020 at 1:33 PM Joe Touch <[email-masked]
> <touch@strayalpha..com>> wrote:
>

---

From: Craig Partridge <[email-masked]>
Date: Thu, 4 Jun 2020 15:26:17 -0600

The SSH spec says terminate on failure and that it requires a reliable
underpinning.

Termination on error is no good.  One of the studies shows huge failure
rates (over 50%) for large file transfers.  One guess is that's due to
security protocols terminating when TCP hands up something with an error.

Craig

---

From: Brian E Carpenter <[email-masked]>
Date: Fri, 5 Jun 2020 09:28:23 +1200

Richard,

> On the UDP side, there should be no issue.  Anything above UDP has to tolerate loss anyway, so there should be mechanisms that will require from corruption-treated-as-loss. 

That depends. If a single bit inversion can get through undetected, the application layer might never detect an anomalx. (See what I did there?)

Regards
   Brian Carpenter

On 05-Jun-20 09:18, Richard Barnes wrote:
> Yeah, secure transports already consider the data from the network untrusted, including TLS and DTLS, but also IPsec/ESP, SSH, SRTP and QUIC.  As long as those protocols are using reasonably modern, AEAD ciphers, the worst impact of corruption in the transport is DoS.
> 
> That said, there might be some pretty hard crashes in cases where the security protocol relies on the transport for something..  For example, TLS doesn't have a mechanism for requesting that data be re-sent in the event of loss/corruption, because it's assumed that TCP provides that.  If corruption is only detected at the TLS layer, there's not a way to recover.  That said, I wouldn't be surprised if many TLS stacks close the connection on bad data anyway.
> 
> On the UDP side, there should be no issue.  Anything above UDP has to tolerate loss anyway, so there should be mechanisms that will require from corruption-treated-as-loss.
> 
> So there may be some feedback loops to close, but that's a more limited, host-internal problem.
> 
> On Thu, Jun 4, 2020 at 4:17 PM Craig Partridge <[email-masked] <mailto:[email-masked]>> wrote:
> 
>     Looks like it does the trick if we need it. Not sure if we'd have to update the optional checksum to use a new checksum too.
> 
>     Craig
> 
>     On Thu, Jun 4, 2020 at 1:33 PM Joe Touch <[email-masked] <mailto:touch@strayalpha..com>> wrote:
> 
>         > On Jun 4, 2020, at 12:13 PM, Craig Partridge <[email-masked] <mailto:[email-masked]>> wrote:
>         >
>     Craig Partridge's email account for professional society activities and mailing lists.
>

---

From: Nico Williams <[email-masked]>
Date: Thu, 4 Jun 2020 16:34:14 -0500

> security protocols terminating when TCP hands up something with an error.

+1 (though, of course, one could have the protocol above TLS/SSH be
resumable in some cases).

Note too that recovery by retransmission requires more buffering unless
TLS/SSH and TCP could coordinate and share buffers.

My take is that better checksums are needed even for TLS/SSH, but 
not for TCP over IPsec, and maybe not for DTLS apps.

Nico
--

---

From: Nico Williams <[email-masked]>
Date: Thu, 4 Jun 2020 16:41:43 -0500

On Fri, Jun 05, 2020 at 09:28:23AM +1200, Brian E Carpenter wrote:
> > On the UDP side, there should be no issue.  Anything above UDP has
> > to tolerate loss anyway, so there should be mechanisms that will
> > require from corruption-treated-as-loss. 
> 
> That depends. If a single bit inversion can get through undetected,
> the application layer might never detect an anomalx. (See what I did
> there?)

A long time ago, back at Sun, we had a NIC on a server that would
introduce paired-bit errors at offsets such that all checksums in the
stack were defeated occasionally, which then corrupted Teamware back
when Sun used Teamware over plain NFS for its source repositories(!).
IIRC there was a nifty blog entry about this, but I'm not sure how I'll
ever find it.  Anyways, that one achieved legendaray status at Sun.  It
was very hard to find, much harder than your typo :)

Nico
--

---

From: Richard Barnes <[email-masked]>
Date: Thu, 4 Jun 2020 17:42:06 -0400

On Thu, Jun 4, 2020 at 5:28 PM Brian E Carpenter <
[email-masked]> wrote:

> That depends. If a single bit inversion can get through undetected, the
> application layer might never detect an anomalx. (See what I did there?)
>

My point was that you could rely on a security layer in between the
transport and the app, in particular one that applies authenticated
encryption to verify the integrity of incoming data.  And cryptographic
integrity checks are in fact bit-sensitive.

Such a layer is in place for ~92% of HTTP traffic nowadays (depending on
how you measure), all WebRTC traffic, etc.  So like I said before, this
kind of looks like an argument for encrypting everything.

--Richard

> security protocol relies on the transport for something..  For example, TLS
> <mailto:touch@strayalpha..com>> wrote:
> >
> >         > On Jun 4, 2020, at 12:13 PM, Craig Partridge <
> [email-masked] <mailto:[email-masked]>> wrote:
> >         >

---

From: Richard Barnes <[email-masked]>
Date: Thu, 4 Jun 2020 17:45:25 -0400

It seems like you're assuming that file transfer applications are sending a
large file all in one go, over a single TCP session.  All of the serious
file transfer applications that come to my mind protect against this by
handling large things in more manageable chunks -- think BitTorrent,
MPEG-DASH, even HTTP range requests.  That's prudent application design not
just because TCP passes corrupted data sometimes, but because transport
connections fail for all sorts of reasons.

--Richard

---

From: Christian Huitema <[email-masked]>
Date: Thu, 4 Jun 2020 14:54:24 -0700

> On Jun 4, 2020, at 2:26 PM, Craig Partridge <[email-masked]> wrote:
> 
> 
> The SSH spec says terminate on failure and that it requires a reliable underpinning.
> 
> Termination on error is no good.  One of the studies shows huge failure rates (over 50%) for large file transfers.  One guess is that's due to security protocols terminating when TCP hands up something with an error.

Good thing that the IETF is nearing completion of the work on QUIC version 1. I don't know whether you followed that, Craig, but QUIC plus UDP provides pretty much the same services as TLS plus TCP, including protection of every packet using AEAD encryption and checksum. Error not detected by the UDP checksum will most probably be detected by the 16 bits AEAD checksum.

Your note mentions that errors are so frequent that some might not be detected by the MD5 checksum. That's a tall claim. MD5 provides a 128 bit cryptographic checksum, so the chances of not detecting an error are something like 1/2^128, something like 1.0E-38.

There are attacks against MD5, which is why it is now deprecated. But these are cryptographic attacks, requiring significant computations to obtain two packets that would get the same checksum. We are speaking here of randomly flipping a few bits, which is completely different. Any robust 128 checksum will detect that.

-- Christian Huitema

---

From: Christian Huitema <[email-masked]>
Date: Thu, 4 Jun 2020 17:00:43 -0700

On 6/4/2020 2:54 PM, Christian Huitema wrote:
> Good thing that the IETF is nearing completion of the work on QUIC version 1.. I don't know whether you followed that, Craig, but QUIC plus UDP provides pretty much the same services as TLS plus TCP, including protection of every packet using AEAD encryption and checksum. Error not detected by the UDP checksum will most probably be detected by the 16 bits AEAD checksum.
Oops. 16 bytes AEAD checksum, not bits, of course.
> There are attacks against MD5, which is why it is now deprecated. But these are cryptographic attacks, requiring significant computations to obtain two packets that would get the same checksum. We…

I understand now that Craig was referring to environments in which the
TCP checksum is complemented by an MD5 checksum truncated to 32 bits.
That combination will let an error pass through for every 4 billion
errors not detected by the TCP checksum, which could very well be a
practical problem. When a full MD5 128 bit checksum used to check the
validity of a file, the residual TCP errors will be reliably detected.

-- Christian Huitema

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Thu, 4 Jun 2020 22:57:51 -0400

My first gut reaction to this was 'do the check at the application layer'.

And then I realized that was completely wrong. We have to know what error
rate we should expect at the TCP level so that we can write the application
in a sensible fashion.

Consider the case in which I am transfering a 60GB 4K movie over the net.
Say for the sake of argument there is a 1% chance of a one bit failure. If
we only have one checksum on that lot, we have a significant probability of
the effort being wasted.
There is another problem with the use of MACs which is that a MAC
verification failure is going to be reported (correctly) as an
authentication error and not as a communication error.

I don't think we can expect the transport layer to be 100% reliable. Using
MD5 makes no sense, SHA2 isn't that much slower if at all.

If we are using Merkle Damgard construction it is pretty easy to calculate
both a final value and interim values at (almost) no extra cost.

So lets imagine I am transferring that 60GB. I begin calculating the SHA2
value at the start. Every 10MB or so, I throw out the current output value.
If the receiver detects an error in the output value it calls back and
says, 'hey, that output value was wrong, resend that chunk'.

Can get fancier if people like and minimize the amount of data that needs
to be resent.

Alternatively, we could get creative and define a Merkle Tree construction
to replace Merkle-Damgard. Make it possible to parallelize digest
computation.

---

From: Joseph Touch <[email-masked]>
Date: Thu, 4 Jun 2020 21:01:15 -0700

> On Jun 4, 2020, at 7:57 PM, Phillip Hallam-Baker <[email-masked]> wrote:
> 
> Consider the case in which I am transfering a 60GB 4K movie over the net. Say for the sake of argument there is a 1% chance of a one bit failure.

There are a lot of statistical assumptions in that statement.

How about somebody showing an actual case where this has happened, please?

Before we solve a problem in theory rather than in practice.

Joe

---

From: Stewart Bryant <[email-masked]>
Date: Fri, 5 Jun 2020 07:50:46 +0100

Would the Fletcher checksum, which is also 16 bits but has CRC like properties and is easy for low powered h/w to calculate be sufficient?

I am curious as to where these errors are occurring. The MAC will be CRC protected in post case. Are the errors surviving the MAC or are they introduced in the h/w?

Stewart

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Fri, 5 Jun 2020 12:10:25 -0400

On Fri, Jun 5, 2020 at 12:01 AM Joseph Touch <[email-masked]> wrote:

> Say for the sake of argument there is a 1% chance of a one bit failure.
>
>

Has anyone been looking? The security area has always been interested in
theoretical attacks. They are by far the best kind.

I was throwing those numbers out to point out what is now a routine sort of
communication.

It is clear that as the size of bulk transfers increase, the probability of
a transmission failure that is not detected by the transport layer
approaches 1. Traditionally, we just ignored the risk. That was probably an
acceptable response in the days when we bought memory in 128K modules.
Those assumptions no longer hold.

SHA-2-256 is good for pretty much any feasible data transfer. But HTTP
certainly isn't and neither is QUIC. And it would be kinda silly to
conflate a protocol designed to support fast interactive response for Web
browsing with bulk data transfer.

The Mathematical Mesh has two separate messaging systems. There is a
control plane where the messages are limited to ~32KB (not certain that is
the sweet spot yet but less than 64K). And there is a data plane for bulk
messages that will be eventually engineered to support Terabyte and
Petabyte transfers.

It is pretty clear that the current transport checksum is sufficient for my
control plane. It is equally clear that the data plane can't rely on any
imaginable transport layer checksum to support Petabyte transfers. So I am
fine with the transport checksum either way.

Sure, I have taken some extreme points on the curve here and there might be
a point inbetween where it makes sense to upgrade the transport checksum
because it is causing issues at the application level. But the burden of
proof is on people suggesting we need to change the transport checksum that
such a position exists.

---

From: Joseph Touch <[email-masked]>
Date: Fri, 5 Jun 2020 09:30:09 -0700

Hi, Philip,

All these statistical analyses assume that errors are not correlated in many dimensions. If that were to happen on the wire, it would likely be caught by things like Ethernet’s CRC. The issue we’re dealing with are errors that are either not caught on the wire OR happen in memory or processing, neither of which are very likely to be uncorrelated.

The only thing that’s “pretty clear” to me is that if this were actually as uncorrelated as you’re assuming, we would have lots of examples of transfers that failed. I asked a simple question - are we seeing that? I’ll admit that doesn’t come up for me, but by everyone’s “analysis” they ought to be raining down everywhere, esp. in large data centers. But I haven’t heard anyone screaming yet.

So either the other protections we have (as others have mentioned) are sufficient or these errors are not correlated the way that’s been assumed (and thus they’re being caught by the existing checksum).

There’s no law against repeating a checksum with different data if all the data gets there OK.

Joe

---

From: Nico Williams <[email-masked]>
Date: Fri, 5 Jun 2020 11:39:12 -0500

> Has anyone been looking? The security area has always been interested in

No one looks for this.

> theoretical attacks. They are by far the best kind.

This is a real problem, not theoretical.

I described an accident that happened some... I don't remember, 12? 15?
years ago at Sun.  The NIC in question showed lots and lots of errors,
but nobody noticed.  And some of those errors went undetected until
eventually corruption was detected in an application that led to a bug
hunt that found the NIC to be busted.

Nobody looks at NIC error counts.

As MTU/MSS sizes go up, as bandwidth goes up, this becomes more of a
risk.

Now, we've talked about how some applications are or can easily be
impervious to this.  If you're transferring static data, this is not a
problem because you just use crypto that detects TCP checksum failures
and then make the application protocol recover.  But some applications
are more difficult to address than others.

I wonder how much TCP offload HW will complicate the upgrade path here.

Nico
--

---

From: Michael Richardson <[email-masked]>
Date: Fri, 05 Jun 2020 12:59:56 -0400

Stewart Bryant <[email-masked]> wrote:
    > Would the Fletcher checksum, which is also 16 bits but has CRC like
    > properties and is easy for low powered h/w to calculate be sufficient?

    > I am curious as to where these errors are occurring. The MAC will be
    > CRC protected in post case. Are the errors surviving the MAC or are
    > they introduced in the h/w?

I think that we must assume some amount of bit flips in memory between MACs
(meaning both MAC CRC and MAC devices).   I wonder if this has also gotten
worse now that ethernet controllers can calculate (and segment) traffic.

/me pines for ubiquitous deployment of AH.

--
Michael Richardson <[email-masked]>, Sandelman Software Works
 -= IPv6 IoT consulting =-

---

From: Nico Williams <[email-masked]>
Date: Fri, 5 Jun 2020 13:18:18 -0500

> /me pines for ubiquitous deployment of AH.

Isn't it deprecated?  It's just ESP w/ AEAD ciphers / cipher modes now.

---

From: Craig Partridge <[email-masked]>
Date: Sat, 6 Jun 2020 17:17:13 -0600

I got requests for citations/sources.  So here's what's known.

First, background reading from 20 years ago -- shows the distribution of
errors that we saw in TCP connections then.  Note while bit errors happen,
far more are bytes or larger.  This paper was also the origin of the idea
of putting 32-bit md5 sums on files...

@inproceedings{10.1145/347059.347561, author = {Stone, Jonathan and
Partridge, Craig}, title = {When the CRC and TCP Checksum Disagree}, year =
{2000}, isbn = {1581132239}, publisher = {Association for Computing
Machinery}, address = {New York, NY, USA}, url = {
https://doi.org/10.1145/347059.347561}, doi = {10.1145/347059.347561},
booktitle = {Proceedings of the Conference on Applications, Technologies,
Architectures, and Protocols for Computer Communication}, pages =
{309–319}, numpages = {11}, location = {Stockholm, Sweden}, series =
{SIGCOMM ’00} }

OK, on to what people are seeing today.  This shows that 1 in every 121
file transfers FTP delivers a file that, when you do the md5 sum, turns out
not to match the original (note there are multiple possible reasons, but
TCP checksum is a strong candidate).

@inproceedings{Liu:2018:CSD:3208040.3208053,

 author = {Liu, Zhengchun and Kettimuthu, Rajkumar and Foster, Ian and Rao,
Nageswara S. V.},

 title = {Cross-geography Scientific Data Transferring Trends and Behavior},

 booktitle = {Proceedings of the 27th International Symposium on
High-Performance Parallel and Distributed Computing},

 series = {HPDC '18},

 year = {2018},

 isbn = {978-1-4503-5785-2},

 location = {Tempe, Arizona},

 pages = {267--278},

 numpages = {12},

 url = {http://doi.acm.org/10.1145/3208040.3208053},

 doi = {10.1145/3208040.3208053},

 acmid = {3208053},

 publisher = {ACM},

 address = {New York, NY, USA},

 keywords = {GridFTP, file transfer, usage management, wide area network},

}

On a related point, about 60% of big file transfers in a major energy
network are failing (checksums are one of the suspects);

@inproceedings{shannigrahi2017request,

  title={Request aggregation, caching, and forwarding strategies for
improving large climate data distribution with NDN: a case study},

  author={Shannigrahi, Susmit and Fan, Chengyu and Papadopoulos, Christos},

  booktitle={Proceedings of the 4th ACM Conference on Information-Centric
Networking},

  pages={54--65},

  year={2017},

  organization={ACM}

}

This reference also sees high error rates:

Kettimuthu, Rajkumar, et al. "Transferring a Petabyte in a Day." Future
Generation Computer Systems 88 (2018): 191-198.

Anecdotally, folks are reporting some middlebox vendors are not updating
the TCP checksum but rather letting the outbound interface simply recompute
the entire checksum -- which means that if the TCP segment gets damaged
during middlebox handling, the middlebox will slap a valid checksum on bad
data.

As I noted -- the literature is slim, which is why a team I'm on is going
to seek more comprehensive error collection which actually captures the
errors in the data, so we can see what kinds of errors are causing trouble.

Craig

---

From: Masataka Ohta <[email-masked]>
Date: Sun, 7 Jun 2020 12:46:33 +0900

> TCP checksum is a strong candidate).

That's unreasonable because most errors are detected by datalink
layer checksum and almost all remaining errors are detected by
transport layer checksum, which should have been the reason
why transport checksum need not be so strong.

> during middlebox handling, the middlebox will slap a valid checksum on bad

That should be the real problem to make transport checksum not
to work end to end.

Thus, your proposal to have stronger checksum can not prevent
file corruptions.

So, we should make middlebox vendors to update checksum incrementally
or, check the original checksum just before sending a packet
with the original header (not applicable if payload is also modified).

						Masataka Ohta

PS

This is a old problem documented in the original paper on
the E2E principle.

https://dl.acm.org/doi/pdf/10.1145/357401.357402
2.2 A Too-Real Example
One gateway computer developed a transient
error: while copying data from an input to
an output buffer a byte pair was
interchanged, with a frequency of about one
such interchange in every million bytes passed.

---

From: "John Levine" <[email-masked]>
Date: 7 Jun 2020 12:26:24 -0400

In article <[email-masked]> you write:
>-=-=-=-=-=-

FYI, the ACM and some other publishers have currently disabled their
paywalls so anyone can download articles from their archive.

>High-Performance Parallel and Distributed Computing},

http://doi.acm.org/10.1145/3208040.3208053

>  author={Shannigrahi, Susmit and Fan, Chengyu and Papadopoulos, Christos},

https://www.researchgate.net/publication/319925429_Request_aggregation_caching_and_forwarding_strategies_for_improving_large_climate_data_distribution_with_NDN_a_case_study

>Generation Computer Systems 88 (2018): 191-198.

https://www.researchgate.net/publication/325405478_Transferring_a_Petabyte_in_a_Day

R's,
John

---

From: Phillip Hallam-Baker <[email-masked]>
Date: Sun, 7 Jun 2020 14:37:54 -0400

> > Has anyone been looking? The security area has always been interested in

Does anyone have infrastructure that works so well and so dependably that
they can afford the time to perform a full post mortem on every error?

How many people do post mortems versus turning it off and on again? There
are so many bugs in the application layer that hardware errors are rarely
considered.

Case in point, I have close to $10K worth of consumer level Internet gear
in my house. Every so often the network will go into some weird race
condition that is clearly a consequence of buffer bloat or some piece of
connected hardware spamming the network with garbage. The manufacturers
provide precisely zero support for debugging. None, nil, nadda.

The reason I try to use consumer grade equipment is that I want to
understand the system from the consumer's perspective and right now I don't
have any respect for any of the hardware suppliers involved. That said, 24
port PoE internet switches are probably not consumer grade.

> This is a real problem, not theoretical.

Of course it is real. The Internet is a sufficiently large network that
everything that can happen will happen.

> I wonder how much TCP offload HW will complicate the upgrade path here.

The argument I was making was actually somewhat different to the one Joe
responded to.

My point is actually that data transfers are becoming sufficiently large
that it is no longer sensible or useful to adopt the assumption that TCP is
a perfect, error-free transport at the application layer.

But the memory issue Michael raised is also rather important. The UNIX
assumption that 'everything is a stream of bits' was a viable assumption at
one point in time. But as we build larger and larger systems, that
assumption also becomes weak. We need to think about how we store large
quantities of data on SSD etc. as well. I already have RAIDs that are
approaching 100TB and soon we will be at the point where PetaByte SSD
stores are common.

If we want systems to work well, we cannot build systems that are hurling
terabytes of data about in the exact same way that we used to build systems
when a 20MB drive was the acme of luxury.

Rather than trying to make TCP/IP a flawless transport, we have to apply
the same principle of making a lossy channel robust at the higher level.

---

From: Joseph Touch <[email-masked]>
Date: Sun, 7 Jun 2020 12:08:16 -0700

+1

> On Jun 7, 2020, at 11:37 AM, Phillip Hallam-Baker <[email-masked]> wrote:
> 
> Rather than trying to make TCP/IP a flawless transport, we have to apply the same principle of making a lossy channel robust at the higher level.

My other concern is also with offload mechanisms and middleboxes. There’s the old adage of whether we’re corruption errors (line, memory) or manipulation errors (incorrect code or assumptions about the inputs to that code).

Overall, I’d feel a lot better about upending transport checksums if we had evidence that the checksum wasn’t catching errors. If the checksum is correct because it’s being constantly recomputed without being checked, a new alg won’t fix the issue.

And if this were truly a 1 in 100 problem, wouldn’t we be seeing a lot more of it?

If somethings fishy here, it’s not clear at all yet what it is.

Joe

---

From: Christian Huitema <[email-masked]>
Date: Sun, 7 Jun 2020 12:39:38 -0700

> On Jun 7, 2020, at 12:08 PM, Joseph Touch <[email-masked]> wrote:
> 

Or, use a keyed cryptographic checksum and do not give the key to middleboxes.

-- Christian Huitema

---

From: Michael Thomas <[email-masked]>
Date: Sun, 7 Jun 2020 14:16:18 -0700

> Or, use a keyed cryptographic checksum and do not give the key to middleboxes.
I've always had an unease about transport layer security vs transport 
IPsec. At least I now have something to hang my hat on.

Mike

---

From: Joseph Touch <[email-masked]>
Date: Sun, 7 Jun 2020 15:20:43 -0700

TLS isn’t “transport layer”. It’s app-layer.

TCP-AO is transport layer - and would similarly protect against middlebox modifications.

Joe

> On Jun 7, 2020, at 2:16 PM, Michael Thomas <[email-masked]> wrote:
> 
> 
> I've always had an unease about transport layer security vs transport IPsec. At least I now have something to hang my hat on.
> 
> Mike
> 
>

---

From: Michael Thomas <[email-masked]>
Date: Sun, 7 Jun 2020 15:28:34 -0700

> TCP-AO is transport layer - and would similarly protect against middlebox modifications.

That's my point: TLS leaves the transport layer unprotected.

Mike

---

From: "Salz, Rich" <[email-masked]>
Date: Sun, 7 Jun 2020 22:54:19 +0000

>    That's my point: TLS leaves the transport layer unprotected.

Why do you say that?

---

From: Michael Thomas <[email-masked]>
Date: Sun, 7 Jun 2020 16:16:46 -0700

>>     That's my point: TLS leaves the transport layer unprotected.
Because the TCP headers aren't part of the hmac digest? Am I missing 
something?

Mike

---

From: "Salz, Rich" <[email-masked]>
Date: Sun, 7 Jun 2020 23:52:36 +0000

something?

And how does that affect the application data?  What corruption of TCP headers would not end up being noticed at the application layer and therefore TLS?

---

From: Michael Thomas <[email-masked]>
Date: Sun, 7 Jun 2020 17:21:49 -0700

> And how does that affect the application data?  What corruption of TCP headers would not end up being noticed at the application layer and therefore TLS?

well, it could send it to the wrong port, but i'll guess that tls is on 
to that problem. i mean, it kind of sounds like you're saying the 
transport checksum failing isn't a big deal? creating a gigantic window 
would certainly not be a good thing in the face of congestion. transport 
mode ipsec wouldn't suffer those kinds of problems.

Mike

---

From: Michael Richardson <[email-masked]>
Date: Sun, 07 Jun 2020 20:59:32 -0400

Thank you for these references. 1 in 121. wow.

Craig Partridge <[email-masked]> wrote:
    > Anecdotally, folks are reporting some middlebox vendors are not
    > updating the TCP checksum but rather letting the outbound interface
    > simply recompute the entire checksum -- which means that if the TCP
    > segment gets damaged during middlebox handling, the middlebox will slap
    > a valid checksum on bad data.

Seems like an argument for anyone dealing with large data transfers to
consider IPv4 End of Life.  IPv6 is here... Use it.

(I'd wish that this also meant using IPsec AH, which we could probably
do with IPv6, but we never got useful APIs.  Insert Sad-Poo Emoji)

---

From: Benjamin Kaduk <[email-masked]>
Date: Sun, 7 Jun 2020 19:46:47 -0700

> And how does that affect the application data?  What corruption of TCP headers would not end up being noticed at the application layer and therefore TLS?

A sometimes significant issue is that if TLS is the thing that notices the
corruption, the stream shuts down and any recovery steps need to be done by
the application.  Some applications don't do this well, so their
application data can be badly affected.  But in general a well-designed
application protocol should have a way to recover fairly gracefully, as has
been noted a few times upthread.

-Ben

---

From: Nick Hilliard <[email-masked]>
Date: Mon, 8 Jun 2020 10:09:30 +0100

> mode ipsec wouldn't suffer those kinds of problems.

in their current incarnations, transport mode ipsec and tcp-ao aren't 
deployable at scale in the same way that tls is.

Regarding transport layer integrity, there are distant echoes of the old 
circuit-switched vs packet-switched arguments going on here.  tcp/ip 
made circuit switching redundant by loosening its assumptions about 
transport layer reliability.  I wonder are we now seeing something 
similar with TLS, which no longer depends on either underlying transport 
or ip header integrity by pushing data stream integrity management 
higher up the stack.

Nick

---

From: Carsten Bormann <[email-masked]>
Date: Mon, 8 Jun 2020 11:54:15 +0200

> And how does that affect the application data?  What corruption of TCP headers would not end up being noticed at the application layer and therefore TLS?

Noticed, sure.
It is still a performance bug if transfers need to be redone from scratch just because the TCP checksum failed to catch corruption.
As the transfers get longer, the probability of some corruption approaches one, so the number of transfer attempts needed before one goes through goes through the roof.

(Literally the first thing I did on the IP network of this university, when I came here in 1993, was to find the reason for random file corruption in the network.  Of course, it was a Sun NFS server with a failing Ethernet card.  At the time, UDP checksums were generally switched off for NFS, but even if they would have been switched on, we’d just seen the corruption ~ 2**-16 as often.  Security protocols used right would have protected us from this and would have made the bit errors a performance statistic…)

TL;DR: Don’t use the network without security.  
But if the secure protocols fail to recover, you still have a problem.
QUIC completely solves this particular problem; only TLS/TCP has it, DTLS doesn’t.

Grüße, Carsten

---

From: Nick Hilliard <[email-masked]>
Date: Mon, 8 Jun 2020 11:29:50 +0100

Carsten Bormann wrote on 08/06/2020 10:54:
> Security protocols used right would have protected us from this and
> would have made the bit errors a performance statistic…)
not at the time, no.  There was no cpu support for common crypto 
operations then, so enabling transport layer security would have trashed 
performance to the point that the server would have been unusable.  Even 
years later, you couldn't feasibly enable SSL on servers without 
offloading to hardware assistance cards.

We've really forgotten how computationally expensive cryptography is, 
now that it's been swept under the carpet by increasingly advanced 
hardware support.  E.g. even consumer grade CPUs have had dedicated 
native AES encryption support for the last 10 years.

But now that it's here, it's compelling to use pervasively.

Nick

---

From: Michael Thomas <[email-masked]>
Date: Mon, 8 Jun 2020 09:59:19 -0700

On 6/8/20 2:09 AM, Nick Hilliard wrote:
>> well, it could send it to the wrong port, but i'll guess that tls is 
>> on to that problem. i mean, it kind of sounds like you're saying the 
>> transport checksum failing isn't a big deal? creating a gigantic 
>> window would certainly not be a good thing in the face of congestion. 
>> transport mode ipsec wouldn't suffer those kinds of problems.
>
why would you say that? what layer the crypto is performed seems sort of 
irrelevant: rsa, aes and sha don't care who calls them. i assume that 
you can hack ipsec to emulate clients not having certs. what's left?
>
> Regarding transport layer integrity, there are distant echoes of the 
> old circuit-switched vs packet-switched arguments going on here.  
> tcp/ip made circuit switching redundant by loosening its assumptions 
> about transport layer reliability.  I wonder are we now seeing 
> something similar with TLS, which no longer depends on either 
> underlying transport or ip header integrity by pushing data stream 
> integrity management higher up the stack.
>
Quic seems to have done the opposite by moving it down. But do I trust 
higher levels to deal with congestion avoidance correctly? Not at all. 
That's a tragedy of the commons waiting to melt down.

Mike

---

From: Joe Touch <[email-masked]>
Date: Mon, 8 Jun 2020 10:11:09 -0700

> On Jun 8, 2020, at 10:00 AM, Michael Thomas <[email-masked]> wrote:
> 
> i assume that you can hack ipsec to emulate clients not having certs.

It is called BTNS.  See RFC 5387.

---

From: Nico Williams <[email-masked]>
Date: Mon, 8 Jun 2020 12:16:30 -0500

> It is called BTNS.  See RFC 5387. 

Yes, but you also need RFC5660 implementations to make it more
meaningful.  Still, if all you want is error detection, BTNS will do.

---

From: Nico Williams <[email-masked]>
Date: Mon, 8 Jun 2020 12:21:48 -0500

On Mon, Jun 08, 2020 at 09:59:19AM -0700, Michael Thomas wrote:
> irrelevant: rsa, aes and sha don't care who calls them. i assume that you
> can hack ipsec to emulate clients not having certs. what's left?

TLS is trivial to deploy: change your apps.

IPsec has no notion of "connections" -- well, there's RFC5660, but that
basically never got implemented (I suppose it's never too late) -- so
it's hard to apply.

TCP-AO requires an out of band key exchange mechanism which I guess
could be IKE, but is that implemented?

> That's a tragedy of the commons waiting to melt down.

Eh?  No, QUIC / HTTP/3 switches to UDP, but integrity protection happens
at the application layer, not at the UDP or IP (IPsec) layers.

I do expect UDP-based Internet protocols to handle congestion avoidance.

Nico
--

---

From: Nick Hilliard <[email-masked]>
Date: Mon, 8 Jun 2020 18:24:07 +0100

> you can hack ipsec to emulate clients not having certs. what's left?

Usability?  How about you put someone's granny in front of a computer 
and give her the simple task of transferring some data over tls, or 
tcp-ao, or ipsec.  Any data would do, e.g a http GET, or a one-line 
message to her grand-daughter to say happy birthday.

> That's a tragedy of the commons waiting to melt down.

The other aspect of these distant echoes related to how previous 
incumbencies gave pretty serious push-back due to their concerns about 
people not paying enough attention to lower-layer considerations.

Nick

---

From: Michael Thomas <[email-masked]>
Date: Mon, 8 Jun 2020 10:24:10 -0700

> meaningful.  Still, if all you want is error detection, BTNS will do.

this is undoubtedly a complete rehash, but who controls what the root 
CA's are with ipsec? is that something that the application layer has 
some say-so over? could my app say i don't care who the root CA is?

Mike

---

From: Michael Thomas <[email-masked]>
Date: Mon, 8 Jun 2020 10:37:21 -0700

On 6/8/20 10:24 AM, Nick Hilliard wrote:
>>> in their current incarnations, transport mode ipsec and tcp-ao 
>>> aren't deployable at scale in the same way that tls is.
>> why would you say that? what layer the crypto is performed seems sort 
>> of irrelevant: rsa, aes and sha don't care who calls them. i assume 
>> that you can hack ipsec to emulate clients not having certs. what's 
>> left?
>

Uh, why are you selling apps so short? An app is capable of making 
library calls for TLS but incapable of making the OS calls for IPsec? 
That's just silly.

The only reason, imo, that tls took hold is because it beat ipsec to the 
market. By the time ipsec was well supported, nobody cared any more.

Mike

---

From: Nick Hilliard <[email-masked]>
Date: Mon, 8 Jun 2020 19:12:10 +0100

> library calls for TLS but incapable of making the OS calls for IPsec? 

Difficult to see how this was inferred? :-)

> market. By the time ipsec was well supported, nobody cared any more.

ssl, then later tls, took hold because it was designed for, and 
therefore easily applied to tcp sessions and because there was a lot of 
effort put into creating ancillary frameworks, e.g. the pki, sensible 
APIs, etc.  This push towards application usability made it transparent 
to the protocol consumer, whether that be granny, or 14yo whizz-kid, or 
someone trying to do some online shopping.

Obviously you're technically correct that an app can call any library 
function and that it matters little to a CPU or the data, or the network 
layer whether it's rsa, aes or sha - or ipsec, or tls or whatever.  But 
the success of tls came down to usability, or more specifically 
use-transparency: security could be implemented without people even 
being aware of it and shifting people from unencrypted to encrypted data 
transfer was a easy as configuring a server-side redirect.  Conversely 
configuring and managing ipsec still creates thundering headaches even 
for experienced operators.

There are plenty of reasons that security protocols in particular end up 
being unusable in practice - usually for sound reasons, none of which 
were individually wrong, but the outcome is invariably byzantine.  It's 
often the usability of a system which determines success, not the 
systems' other technical merits.

Nick

---

From: Michael Thomas <[email-masked]>
Date: Mon, 8 Jun 2020 11:23:09 -0700

On 6/8/20 11:12 AM, Nick Hilliard wrote:
> therefore easily applied to tcp sessions and because there was a lot 
> of effort put into creating ancillary frameworks, e.g. the pki, 
> sensible APIs, etc.  This push towards application usability made it 
> transparent to the protocol consumer, whether that be granny, or 14yo 
> whizz-kid, or someone trying to do some online shopping.
>
> function and that it matters little to a CPU or the data, or the 
> network layer whether it's rsa, aes or sha - or ipsec, or tls or 
> whatever.  But the success of tls came down to usability, or more 
> specifically use-transparency: security could be implemented without 
> people even being aware of it and shifting people from unencrypted to 
> encrypted data transfer was a easy as configuring a server-side 
> redirect.  Conversely configuring and managing ipsec still creates 
> thundering headaches even for experienced operators.

ssl had the advantage that it 1) beat ipsec to market, and 2) wasn't 
subject to API differences from OS layer calls like IPsec was, and with 
quite a bit of churn as i recall too. it's really too bad, imo. we 
wouldn't have had to do the contortions of dtls, for example. and now 
there's this problem. none of them are earth shattering, but it would 
have been cleaner.

Mike

---

From: Carsten Bormann <[email-masked]>
Date: Mon, 8 Jun 2020 20:39:55 +0200

On 2020-06-08, at 12:29, Nick Hilliard <[email-masked]> wrote:
> not at the time, no. 

Of course not, it was practically illegal to do security.

We now have the opportunity to make pervasive use of security; nobody knows how long that window of opportunity will stay open.  Instead of working on changing checksums, we should go for it.

Grüße, Carsten

---

From: Nico Williams <[email-masked]>
Date: Mon, 8 Jun 2020 13:45:44 -0500

On Mon, Jun 08, 2020 at 10:24:10AM -0700, Michael Thomas wrote:
> this is undoubtedly a complete rehash, but who controls what the root CA's
> are with ipsec? is that something that the application layer has some say-so
> over? could my app say i don't care who the root CA is?

The idea with RFCs 5387 and 5660 is that there is no need for an IPsec
PKI for IPsec to be useful, and, indeed, that IPsec for authentication
is tricky because -after all- it deals in... IP addresses, which are not
useful for authentication.

Instead, use IPsec for session crypto and use channel binding to bind
IPsec channels to higher-layer protocols where authentication can and
does happen.

Nico
--

---

From: Michael Thomas <[email-masked]>
Date: Mon, 8 Jun 2020 11:52:04 -0700

> IPsec channels to higher-layer protocols where authentication can and

Sorry for being lazy and not skimming them, but does this imply sort of 
like a naked public key kind of auth like ssh? Or maybe a DNS based one 
like DKIM?

I mean, mitm is always a consideration so auth is always needed, right?

Mike

---

From: Nico Williams <[email-masked]>
Date: Mon, 8 Jun 2020 14:02:21 -0500

On Mon, Jun 08, 2020 at 11:23:09AM -0700, Michael Thomas wrote:
> ssl had the advantage that it 1) beat ipsec to market, and 2) wasn't subject
> to API differences from OS layer calls like IPsec was, and with quite a bit
> of churn as i recall too. it's really too bad, imo. we wouldn't have had to
> do the contortions of dtls, for example. and now there's this problem. none
> of them are earth shattering, but it would have been cleaner.

You can sprinkle TLS anywhere you have an octet stream.  You can
sprinkle DTLS anywhere you have datagram flows.  No need for OS support
-- it will just work.  IPsec?  IPsec requires OS support.

Also, IPsec got a lot of things wrong.  It's simply not usable at
Internet scale as originally intended because... it's IP-layer, so it
deals in discrete packets and IP addresses.  Well, discrete packets do
not define application connections/sessions[*], IP addresses are too
dynamic and useless for authentication and authorization[**], and
configuration is ETOOHARD.

As you can see, a dozen years ago was already too late, but our idea
then was to

 - construct protection guarantees for packet flows using IPsec,
 - to use anonymous or anonymous-like key exchanges to key IPsec,
 - and to use channel binding from application layer protocols that can
   authenticate more useful names than IP addresses.

Nico

[*] In the IPsec architecture, RFC 4301, there is no guarantee that the
    packets making up a TCP connection, say, will have anything like
    similar protection for the lifetime of the TCP connection.
    Everything about how a TCP conenction is protected by IPsec depends
    entirely on *configuration* with no standard interfaces _at all_ for
    applications to manipulate said configuration.

[**] Yes, in RFC 4301 you're supposed to specify things like trust
     anchors and policies like "any peer with a certificate from this CA
     can _claim_ IP addresses in the following prefixes/ranges".  This
     _cannot_ scale to Internet scale.

     Giving up on authentication and authorization in the RFC 4301
     scheme (BTNS, RFC 5387) and constructing logical packet flows with
     consistent protection during their lifetimes (RFC 5660), fixes the
     problem.  It didn't get implemented -- even if it had been, TLS was
     already king.  This would all have to have happened at least half a
     decade earlier, if not a whole decade earlier, to have had a
     chance.

     What was particularly appealing at one point was the possibility of
     having ESP offload in NIC HW.  The lower in the stack the crypto
     happens, the easier it is to offload it.

---

From: John C Klensin <[email-masked]>
Date: Mon, 08 Jun 2020 15:07:05 -0400

--On Monday, June 8, 2020 20:39 +0200 Carsten Bormann
<[email-masked]> wrote:

>...
> We now have the opportunity to make pervasive use of security;
> nobody knows how long that window of opportunity will stay
> open.  Instead of working on changing checksums, we should go
> for it.

<mini-rant>
While you are going for it just be sure that if the window
closes again, and closes sufficiently hard in some places to ban
the use of encrypted message flows entirely, the community is
not faced with a choice among no Internet, a highly fractionated
Internet with no communications between "crypto ok" and "crypto
prohibited" countries, or trying to limp along using protocols
that are known to be defective because we decided to ignore the
problems with them in favor of putting all of our proverbial
eggs in the pervasive security and encryption basket.
</mini-rant>

    john

---

From: Nico Williams <[email-masked]>
Date: Mon, 8 Jun 2020 14:08:26 -0500

On Mon, Jun 08, 2020 at 11:52:04AM -0700, Michael Thomas wrote:
> Sorry for being lazy and not skimming them, but does this imply sort of like
> a naked public key kind of auth like ssh? Or maybe a DNS based one like
> DKIM?

Bare public keys, certified public keys -- either way.

You just wouldn't bother with RFC 4301 style Peer Authentication
Database (PAD, which also deals in authorization), and instead construct
a guarantee of protection and SA peer continuity at the TCP (and even
UDP!, for "connected" UDP sockets) layer.  Then you'd let the app use
channel binding to IPsec, thus binding authentication at the app layer
to session crypto at the lowest end-to-end layer.

That was the idea.  A good idea, IMO, but it didn't happen.  It was too
late.

> I mean, mitm is always a consideration so auth is always needed, right?

See channel binding, RFC 5056.

Nico
--

---

From: Richard Barnes <[email-masked]>
Date: Mon, 8 Jun 2020 15:29:30 -0400

The upshot of the message that started this thread is that if you don't put
your eggs in that basket, then the Internet doesn't work.

---

From: Russ Housley <[email-masked]>
Date: Mon, 8 Jun 2020 15:33:42 -0400

> On Jun 8, 2020, at 3:07 PM, John C Klensin <[email-masked]> wrote:
> 
> 
> 

Meeting this requirement requires integrity protection, not confidentiality.

Russ

---

From: Brian E Carpenter <[email-masked]>
Date: Tue, 9 Jun 2020 08:57:00 +1200

On 09-Jun-20 07:02, Nico Williams wrote:
> sprinkle DTLS anywhere you have datagram flows. 

Unless someone says "multicast". 

   Brian

> No need for OS support

---

From: Nico Williams <[email-masked]>
Date: Mon, 8 Jun 2020 16:00:40 -0500

> Unless someone says "multicast". 

That's a whole different ball of wax.

---

From: Warren Kumari <[email-masked]>
Date: Mon, 8 Jun 2020 19:50:09 -0400

> why transport checksum need not be so strong.

I was trying to avoid this thread, but...

I'm somewhat surprised by some of the numbers (like the 1 in every 121
file transfers).
As a quick check, I looked on a personal webserver (connections from
random people on the Internets), and have received 201.1TB, and
207,183,907,570 (207 billion) packets.
Netstat shows a total of 1029 (detected[0]) CRC errors, or around one
every ~200M packets.

I *think* (but may be completely wrong!) that the chance of a 16bit
checksum giving a false-negative is just 2^16[1], so 200M*2^16 one in
around every 13 billion packets?
My average packet size is ~970byes, so that is ~one bad packet in ~12PB.

I believe the L2 quality and checksums has increased sufficiently that
it has made up for the increase in bandwidth and data volumes -- I'm
sure we all used to watch, and expect errors on PSTN modems, 100Meg
Ethernet, 56k leased lines and the like. I still graph FCS/CRC errors
on router interfaces, but they are basically just empty graphs these
days...

Are others seeing much much worse numbers from looking at their counters?

W
[0]: Yup, it's possible that there were some number of undetected ones
(the whole debate in this thread), but assuming that there are not
systemic issues in the checksum algorithm I believe that the chance of
an error occurring and the checksum calculation happening to match is
2^(length of checksum), and so there is a ~1.6% chance of it happening
(1029/2^16), but I bumped the numbers up to 1030 anyway.

[1]: Actually, I think I overestimate the chance of this happening --
there would need to be a corruption of a packet, *and* the checksum in
the same packet would need to be corrupted, and happen to be the
correct one for that corrupted packet (a 1 in 2^16 chance) - but, that
requires (at least) 2 corruptions in the same packet, or a corruption
which happens to calculate to the same value,  and I don't know how to
easily account for that, so I'll just use the worse case estimate.

-- 
I don't think the execution is relevant when it was obviously a bad
idea in the first place.
This is like putting rabid weasels in your pants, and later expressing
regret at having chosen those particular rabid weasels and that pair
of pants.
   ---maf

---

From: John C Klensin <[email-masked]>
Date: Mon, 08 Jun 2020 22:18:21 -0400

Richard,

I interpreted Craig's  message a little differently.  As Russ
put it, this is a message integrity problem, not a
confidentiality one.  Yet, unless I have completely
misunderstood, much of the conversation has been about methods
that were designed for privacy protection using encryption with
integrity protection being nothing but a necessary side-effect.
Carsten spoke of a window of opportunity.  While I can see lots
of reasons for addressing the issues that Craig raised and doing
so sooner rather than later (especially if they are real, which
several people seem to doubt), I don't see that window closing
on us in the near future.  I do see a risk, one that I think
others have mentioned in this thread but that is certainly under
discussion in other fora, of our losing the battle, at least in
some places, to keep encryption available across the Internet
and I assumed that was the window Carsten was referring to (and
that set me off).

thanks,
   john

--On Monday, June 8, 2020 15:29 -0400 Richard Barnes
<[email-masked]> wrote:

> The upshot of the message that started this thread is that if
> you don't put your eggs in that basket, then the Internet
> doesn't work.
> 
> 
>> > We now have the opportunity to make pervasive use of
>> > security; nobody knows how long that window of opportunity
>> > will stay open.  Instead of working on changing checksums,
>> > we should go for it.
>> 
>> <mini-rant>
>> closes again, and closes sufficiently hard in some places to
>> ban the use of encrypted message flows entirely, the
>> community is not faced with a choice among no Internet, a
>> highly fractionated Internet with no communications between
>> "crypto ok" and "crypto prohibited" countries, or trying to
>> limp along using protocols that are known to be defective
>> because we decided to ignore the problems with them in favor
>> of putting all of our proverbial eggs in the pervasive
>> security and encryption basket. </mini-rant>
>> 
>>     john
>> 
>>

---

From: Christian Huitema <[email-masked]>
Date: Mon, 8 Jun 2020 19:39:12 -0700

> On Jun 8, 2020, at 7:18 PM, John C Klensin <[email-masked]> wrote:
> 
>  Yet, unless I have completely

Encryption without authentication is subject to a variety of attacks such as truncating messages, splicing them, or changing the encrypted text. That's why most modern encryption systems provide authentication at the same time as encryption. So it is much more than "a side effect"...

-- Christian Huitema

---

From: John C Klensin <[email-masked]>
Date: Tue, 09 Jun 2020 01:04:41 -0400

--On Monday, June 8, 2020 19:39 -0700 Christian Huitema
<[email-masked]> wrote:

>> that were designed for privacy protection using encryption
>> with integrity protection being nothing but a necessary
>> side-effect.
> 
> Encryption without authentication is subject to a variety of
> attacks such as truncating messages, splicing them, or
> changing the encrypted text. That's why most modern encryption
> systems provide authentication at the same time as encryption.
> So it is much more than "a side effect"...

Christian, 

Perhaps a poor choice of words.  While it feels to me as if we
are quibbling about fine points of terminology, most likely I'm
missing something important, so let me try again, noting your
"most" and skipping over the question as to whether "integrity
protection" and "authentication" are the same thing and using
more words than many people seem to prefer.

Because integrity protection comes with most modern encryption
systems, there are two ways to approach integrity protection for
data transmission.  One is to encrypt the transmission, thereby
getting the benefits of the associated integrity protection as
well as privacy protection.  The other is to concentrate on
integrity protection methods that leave the transmission payload
in clear (at least as far as the data transmission system is
concerned [1]).

Historical experience going back centuries indicates that, from
time to time, some governments --often driven by law enforcement
or national security agencies and considerations -- have become
anxious about encryption methods strong enough to impede their
access to the encrypted data.  Their proposed mechanisms for
dealing with those concerns have including laws and regulations
requiring parties with access to the cleartext versions of those
data on request, government-accessible backdoors into encryption
technologies, government access to decryption keys, and outright
bans on the use of strong encryption at least under selected
circumstances or by selected parties.  Demands for access to
encrypted data or restrictions on the use of encryption have
occurred in several countries in recent years.  Even while
resisting such demands as hostile to, among other things,
personal privacy and secure commercial transactions over
communications networks, it is sensible for us to make
thoughtful and considered decisions about network design that
include, at least, fallback strategies against the possibility
of bans or several restrictions on cryptography that would
affect non-trivial portions of the Internet.

By contrast, mechanisms that merely provide integrity protection
without resort to encryption have never faced the same level of
government scrutiny as encryption.

Is that better?

thanks,
   john

[1] Information encrypted prior to transmission and decrypted
only subsequent to receipt are not discussed here.

---

From: Michael Thomas <[email-masked]>
Date: Tue, 9 Jun 2020 16:32:38 -0700

> -- it will just work.  IPsec?  IPsec requires OS support.

Yes, but that's also the reason a long on the tooth transport checksum 
becomes a concern. any cryptographic hash is good enough to detect errors.

> configuration is ETOOHARD.

Yes, but that's a feature not a bug in this particular case. I really 
don't find it believable app developers would have had a hard time with 
making the OS calls needed anymore than making the calls that the TLS 
libraries require. And as always: when something is widely used, it gets 
care and feeding. So if ipsec is still clunky, and says it's because 
it's not being widely used, and the niche cases of vpn's can suck it up.

>     authenticate more useful names than IP addresses.
Had SSL/TLS not come along first, there would been immense pressure to 
make this all work.
>       _cannot_ scale to Internet scale.

These are OS implementation details, right? There is nothing inherently 
impossible about filtering on 4-tuples where the app supplies the root 
CA it likes to IKE, right? I mean, that's what TLS is doing, but it's 
just doing it implicitly from the OS layer filtering.

Mike

---

From: Michael Thomas <[email-masked]>
Date: Tue, 9 Jun 2020 16:52:22 -0700

> Unless someone says "multicast".

Hasn't that word been obsoleted and reserved for future use? :)

Mike

---

From: Nico Williams <[email-masked]>
Date: Tue, 9 Jun 2020 19:12:28 -0500

On Tue, Jun 09, 2020 at 04:32:38PM -0700, Michael Thomas wrote:
> Yes, but that's a feature not a bug in this particular case. I really don't
> find it believable app developers would have had a hard time with making the
> OS calls needed anymore than making the calls that the TLS libraries
> require. And as always: when something is widely used, it gets care and
> feeding. So if ipsec is still clunky, and says it's because it's not being
> widely used, and the niche cases of vpn's can suck it up.

The IPsec model is busted even for VPNs.  To mitigate the issues that
arise in VPNs you really need to assign a unique internal address to
each user or accept that authorized users may mount certain attacks on
each other.

But there's nothing about ESP itself that relegates it to VPN usage
only!  Only the IPsec architecture (RFC 4301) causes that, not anything
about ESP.

ESP would have been _great_ for transport-mode IPsec and widespread
application use if RFC 5660 or something like it had been widely
implemented earlier (earlier even than it was written).  The relevant
idea was invented in the 90s, even, the Solaris/Illumos IP_SEC_OPT
socket option (invented by Dan McDonald, I believe).

So, I wouldn't say that IPsec being needlessly difficult to use is a
feature.  No, it's a bug.  TLS offload is necessarily harder to
implement than ESP offload, so we've missed a valuable opportunity to
increase the performance of network security, thus hindered its
adoption.

I would call that a small disaster, honestly.  Not a feature.  A bug.

> Had SSL/TLS not come along first, there would been immense pressure to make
> this all work.

That's likely true, but we can't really know.

> These are OS implementation details, right? There is nothing
> inherently impossible about filtering on 4-tuples where the app
> supplies the root CA it likes to IKE, right? I mean, that's what TLS
> is doing, but it's just doing it implicitly from the OS layer
> filtering.

RFC 5660 gives two implementation designs: tag packets as they move up
(or down) the stack with relevant metadata, or "edit" the RFC 4301 SPD
and PAD in very specific, simple, and narrow ways.  Both schemes allow
the host to guarantee continuity of protection to a logical packet flow
-- a "connection".

The primary scheme given (config "editing") is rather unintrusive,
requiring not changes to how packets move up and down the stack.  The
alternative scheme (packet tagging) is potentially very intrusive for
some operating systems, though it is the simpler scheme.

Nico
--

---

From: Michael Thomas <[email-masked]>
Date: Tue, 9 Jun 2020 17:31:32 -0700

> each user or accept that authorized users may mount certain attacks on
Ok, i'm confused. The demultiplexing of the 5-tuple deals with that, 
right? Just like with TLS which implicitly uses that demultiplexing the 
OS provides as well.
> socket option (invented by Dan McDonald, I believe).
Yeah, I certainly was a fan of the idea of transport mode ESP. But it 
lost and that is that... until suddenly... though i would be being 
overly dramatic to say this was some sort of killer problem.
> I would call that a small disaster, honestly.  Not a feature.  A bug.

But bugs can be fixed. And especially if enough love is thrown at them.

So the long and short of this entire issue seems to be is, is the 
uncaught error rate serious enough that warrant rethinking weak 
transport and frankly L2 layer error detection? Crypto hashes certainly 
are a gigantic cannon to make that problem go away if used correctly. 
considering they are becoming almost table stakes for most IP traffic, 
killing two birds with one stone my not be so bad.

Mike

---

From: "John Levine" <[email-masked]>
Date: 10 Jun 2020 01:08:09 -0000

In article <[email-masked]>,
Michael Thomas  <[email-masked]> wrote:
>transport and frankly L2 layer error detection? ...

Having read the papers that Craig referenced, that's my interpretation.

One of them is about a big physics application which sends multiple
terabytes of data over the net using what looks like a version of
FTP that transfers several files at once.  They send the data as a lot
of of 4 gig files. When they started verifying file checksums, they
found about 20% of the received files were corrrupted in transit.

In that application they resend the corrupt files and they obviously
need make the files smaller. But retransmitting a file at a time seems
a lot less efficient than improving the checksums and using the
existing TCP packet level retransmission.

-- 
Regards,
John Levine, [email-masked], Primary Perpetrator of "The Internet for Dummies",
Please consider the environment before reading this e-mail. https://jl.ly

---

From: Michael Thomas <[email-masked]>
Date: Tue, 9 Jun 2020 18:23:18 -0700

> existing TCP packet level retransmission.
Which pretty much makes a case for something like transport mode ipsec, 
right? it's a hell of a lot cheaper to drop a packet than retransmit an 
entire file, right?

I guess this gets chalked up to false efficiencies. Thankfully we're a 
point where a lot of these tradeoffs aren't quite as hand-wringing.

Mike

---

From: Brian E Carpenter <[email-masked]>
Date: Wed, 10 Jun 2020 14:25:32 +1200

> Hasn't that word been obsoleted and reserved for future use? :)

As a desirable function, certainly not. As a feature that is intrinsic
to your favourite L2 technology, and therefore a fairly simple add-on
to layer 3, its days are numbered. As an easy thing to secure, hmm.

   Brian

---

From: Warren Kumari <[email-masked]>
Date: Wed, 10 Jun 2020 10:24:36 -0400

> found about 20% of the received files were corrrupted in transit.

I'm assuming you are talking about "Cross-Geography Scientific Data
Transferring Trends and Behavior", which contains (Section 4.1
Checksum, encryption, and reliability, p.12):
"We note that if a user changes a file during a transfer, this action
can be reported as an integrity failure. We cannot distinguish this
from an actual failure."

Yes, I'm sure that checksum errors do exist, but from my quick checks
I haven't been seeing anything like the error rates discussed here --
and, as a quick sanity check, 4GB is on the same order as a DVD/many
OS distributions:
RedHat: 8.2.0 - 8GB, 7.9 Beta - 4GB --
https://developers.redhat.com/products/rhel/download
Ubuntu: 18.04 (Desktop): 2GB -- https://releases.ubuntu.com/18.04/
Pop!_OS: 20.04LTS: 2.36 GB (NVIDIA) -- https://pop.system76.com/
Fedora 32: Standard ISO image for x86_64: 1.9GB --
https://getfedora.org/en/server/download/
Kali Linux 64-Bit (Installer): 3.6GB -- https://www.kali.org/downloads/
Linux Mint 19.3 "Tricia" - Cinnamon (64-bit): 1.9GB --
https://www.linuxmint.com/edition.php?id=274
debian-10.4.0-amd64-DVD-3.iso: 4.4GB --
https://cdimage.debian.org/debian-cd/current/amd64/iso-dvd/

I'm assuming that a: almost all of us have downloaded multiple copies
of at least a few of these, and b: we check the hashes on the ISOs we
downloaded.
I certainly haven't been seeing anything like 1 in 5, or 1 in 10 ISO
downloads with a corrupt hash[0].

I also move significant amounts of data around - perhaps I'm just
blessed, but if I were getting corruption on anything approaching that
level, I'm sure I'd have noticed - 20% errors in 4GB files mean I
should be seeing a corruption once every ~20GB. I regularly move TBs
around (backups, DRBD, large containers, databases, etc) - ssh/scp
will log "2: Packet corrupt" (or "Corrupted MAC on input.
Disconnecting: Packet corrupt" on the server side). I stuff all of my
logs into a combination of Logstash and Loki, and querying this gives
no occurrences of this message:

Loki: "{job="syslog",type="server"} |~ "sshd.*Corrupt MAC" == 0

Again, I'm sure that there are checksum errors, but I think that a:
there is lots of data that can be easily looked at to estimate
occurrence (including from CDNs and large scale operators), b: we need
to prioritize what we work on.
I'd love to see people having a look at their systems and reporting
what sorts of errors they see....

W
[0]: actually I've only once seen the checksum not match, and that was
because of a NAT box which tried to ALG fixups in the payload and
replaced all occurrences of the external address bit-pattern with the
internal one...

---

From: "John R Levine" <[email-masked]>
Date: 10 Jun 2020 11:17:40 -0400

> Checksum, encryption, and reliability, p.12):

No, it's "Transferring a Petabyte in a Day".

"As mentioned, we split each 1.2 TiB snapshot into 256 files of 
approximately equal size. We determined that transferring 64 or 128 files 
concurrently, with a total of 128 or 256 TCP streams, yielded the maximum 
transfer rate. We achieved an average disk-to-disk transfer rate of 92.4 
Gb/s (or 1 PiB in 24 hours and 3 minutes): 99.8% of our goal of 1 PiB in 
24 hours, when the end-to-end verification of data integrity in Globus is 
disabled. In contrast, when the end-to-end verification of data integrity 
in Globus is enabled, we achieved an average transfer rate of only 72 Gb/s 
(or 1 PiB in 30 hours and 52 minutes).

The Globus approach to checksum verification is motivated by the 
observations that the 16-bit TCP checksum is inadequate for detecting data 
corruption during communication [16, 17] and that corruption can occur 
during file system operations [18]. Globus pipelines the transfer and 
checksum computation; that is, the checksum computation of the ith file 
happens in parallel with the transfer of the (i + 1)th file. Data are read 
twice at the source storage system (once for transfer and once for 
checksum) and written once (for transfer) and read once (for checksum) at 
the destination storage system. Therefore, in order to achieve the desired 
rate of 93 Gb/s for checksum-enabled transfers, in the absence of checksum 
failures, 186 Gb/s of read bandwidth from the source storage system and 93 
Gb/s write bandwidth and 93 Gb/s of read bandwidth concurrently from the 
destination storage system are required. If checksum verification failures 
occur (i.e., one or more files are corrupted during the transfer), even 
more storage I/O bandwidth, CPU resources, and network bandwidth are 
required in order to achieve the desired rate."

Globus is a file transfer service from U of Chicago

https://www.globus.org/data-transfer

Regards,
John Levine, [email-masked], Taughannock Networks, Trumansburg NY
Please consider the environment before reading this e-mail. https://jl.ly

---

From: "tom petch" <[email-masked]>
Date: Wed, 10 Jun 2020 17:10:19 +0100

----- Original Message -----
From: Warren Kumari <[email-masked]>
To: John Levine <[email-masked]>
Cc: IETF Discuss <[email-masked]>
Sent: 10/06/2020 15:24:36

<tp>
Warren

I have had two experiences of errors in downloads.

One was downloading pdf from an academic institution when about one in three was corrupt and unusable; repeating the download eventually succeeded.  Typical size would be 350kilobyte

The other was downloading large reference XML documents when almost every one was corrupt but being text, the corruption was apparent and I could edit the characters to make the tags valid and view the documents.  A typical size was 250 kilobyte.

I download a lot from other organisations, such as the IETF, FTP whenever possible, and have never detected corruption anywhere else.  My inference was, in both cases, that the corruption was taking place in the backend servers and not in the network ie that TCP was doing its job of reliably 

---
New Outlook Express and Windows Live Mail replacement - get it here:
https://www.oeclassic.com/

delivering corrupted data!

Tom Petch

W

---

From: "Salz, Rich" <[email-masked]>
Date: Wed, 10 Jun 2020 16:37:27 +0000

Your URL's are https and TLS will resend packets even if bit-errors slip past TCP.  Do you get the same results for http URL's?

---

From: Carsten Bormann <[email-masked]>
Date: Wed, 10 Jun 2020 18:40:06 +0200

On 2020-06-10, at 18:37, Salz, Rich <rsalz=[email-masked]> wrote:
> 
> TLS will resend packets even if bit-errors slip past TCP

Last time I looked, TLS would abort the connection.

Which is exactly what makes QUIC more useful here than TLS/TCP.

Grüße, Carsten

---

From: "Salz, Rich" <[email-masked]>
Date: Wed, 10 Jun 2020 16:48:57 +0000

>    Last time I looked, TLS would abort the connection.

Sigh.  Yeah.  Oops.