Subject: [tsvwg] Fall-back to DSCP 0 in draft-ietf-tsvwg-rtcweb-qos ?

From: Michael Welzl <[email-masked]>
Date: Fri, 24 Jun 2016 14:17:07 +0200

Dear all,

We recently did DSCP related measurements that lead us to propose a change to draft-ietf-tsvwg-rtcweb-qos.
These measurements are reported in this paper:
http://heim.ifi.uio.no/~michawe/research/publications/anrw16-measurement.pdf
... which will be presented at the ANRW workshop, the Saturday before the IETF meeting.

The paper reports on tests between people's homes and 3 servers that we ran - one in Oregon (amazon cloud) and two in Norway. This gave us 185 distinct paths (IP address pairs). The test was a TCP handshake, done with raw sockets, and we played around with values in the IP header. Meanwhile, Runa (main author) did some more tests from various public places during a trip from Norway to India, essentially confirming the finding we discuss here; we'll ask for a slot to report about both the data in the ANRW paper and the newer data in the MAPRG session.

The number of data points isn't exactly huge in both cases, but still relevant, we believe: it is about a persistent failure that we saw on different paths, and so we think it's probably reasonable to take it into account (given that a fix shouldn't be very complicated either).

We used, as input, DSCP values from the table in draft-ietf-tsvwg-rtcweb-qos.
Here are some sentences copy + pasted from our paper:

***
"To minimize the chance that congestion-based drops make us believe in a failure
to communicate using certain values in the IP header, we re-tried failed packet
exchanges up to three times, and we sent an ICMP packet just ahead of every
measurement packet. We only assumed a communication failure when the test
failed three times and the ICMP packet succeeded."

"Considering table 1 in [6], we assumed the flow type “Interactive Video with or
without Audio” with application priorities “Very low” (DSCP value CS1 (8)), “Low”
(DF (0)) or “Medium” (AF42 (36)), and flow type “Audio” with application prioritiy
“High” (EF PHB (46))."

"We did see consistent packet drops too: on 23 paths for DSCP value 8, 21 paths
for 36, 19 for 46."
***

We also saw DSCP value changes, but nothing very alarming there - in one case, AF42 was changed to AF41, which is even nice  :)     however, this consistent drop behavior that appears only when using a DSCP value != 0 isn't good at all. Now these things could in principle have happened close to our 3 servers, meaning there were only few devices that did it, but in fact this measurement was a part of a larger measurement campaign in which we also checked where such packet drops typically happened. The result was: most such drops seem to be close to the client, which may indicate that there were in fact multiple such boxes.

We believe that draft-ietf-tsvwg-rtcweb-qos should say: if setting the DSCP values as proposed here consistently produces packet drops, senders should fall back to DSCP 0. WebRTC shouldn't "fail" because of the DSCP.

Cheers,
Michael

---

From: Brian E Carpenter <[email-masked]>
Date: Sat, 25 Jun 2016 08:25:20 +1200

> We believe that draft-ietf-tsvwg-rtcweb-qos should say: if setting the DSCP values as proposed here consistently produces packet drops, senders should fall back to DSCP 0. WebRTC shouldn't "fail" b…

This is an interesting and worrying result indeed, but it's in no way specific
to WebRTC. I think a generic RFC ("Happy Eyeballs with Differentiated Services"?)
would be the way to go. But (as with a lot of recent discussion about IPv6
extension headers causing packet drops) we really need to understand why this
happens. Is it traffic policing that fails to reclassify and re-mark packets,
misguided firewalls, or what?

Regards
   Brian

---

From: Michael Welzl <[email-masked]>
Date: Fri, 24 Jun 2016 22:52:16 +0200

Hi,

> On 24. jun. 2016, at 22.25, Brian E Carpenter <[email-masked]> wrote:
> 
> would be the way to go.

Well, if you use the DSCP within your own administrative domain for DiffServ, you should be safe…. so I guess this problem mostly matters for usage across domains, or even end-to-end.
=> It seems to me that it’s broader than WebRTC in exactly the same way as RFC7657 is broader than WebRTC. 

As a side note, I think the term “Happy Eyeballs” is misleading here. This term, to me, indicates parallel usage of multiple packet types to see which one(s) succeed(s) and then try to make the best choice.
This isn’t what you would do here - we’re talking about a rare failure situation. It’s really only something I’d suggest as a fall-back, upon seeing communication failing entirely, over a longer period (perhaps in line with a circuit-breaker).

> But (as with a lot of recent discussion about IPv6

I have no clue… but as my previous email said, we have indications that this behavior occurred close to the senders (most of them being in people’s homes), hence it’s more likely that this was done by a middlebox or an edge router than by a core router.
That’s as much as we know...

Cheers,
Michael

---

From: <[email-masked]>
Date: Mon, 27 Jun 2016 07:34:22 +0000

Hi Michael,

Thanks, an interesting test set up.

The first IP layer policy point in a network may be the (fixed network) BRAS or another node terminating and authorizing network access. I can't tell you exactly how these are configured in the network of the company I work for, but I think it is rewrite with CS0 and transport. I know that the peering routers are configured that way.

Your doc says: 
Marking packets as AF42 (36) provoked another undefined value (4) on 14 paths, .... Value 46 (EF PHB), on the other hand, was turned into 6 – yet another undefined value – on 17 paths. 

This looks like a so called "IP precedence" rewrite: 100100 is rewritten to 000100 and 101110 to 000110. It's possible that 001000 was rewritten to 000000 that way too in a similar number of cases.
I favor standardizing this behavior, as it is a typical and simple MPLS edge node configuration.  

In Germany, layer 2 interconnection agreements between national providers resulted in a recommendation to drop packets with unexpected Ethernet priority markings at carrier interconnection. I dislike this idea and commented against it, but I wasn't directly involved in negotiations.

From experience in rolling out new services, there seem to be German network operators dropping packets with unexpected DSCPs at IP interconnections.

I couldn't read from your doc whether you differentiated between fixed, WiFi and LTE/3G access. Please add this information, as policies may differ depending on the type of access. I think to recall that e.g. DPI was detected more often in wireless networks when EU regulation probed for that (no, I don't think one needs DPI to detect an unexpected DSCP - my point is to indicate that there differences in policy in general).

Regards,

Ruediger

> But (as with a lot of recent discussion about IPv6 extension headers 
> causing packet drops) we really need to understand why this happens. 
> Is it traffic policing that fails to reclassify and re-mark packets, 

[MW] I have no clue… but as my previous email said, we have indications that this behavior occurred close to the senders (most of them being in people’s homes), hence it’s more likely that this was done by a middlebox or an edge router than by a core router.
That’s as much as we know...

---

From: Michael Welzl <[email-masked]>
Date: Mon, 27 Jun 2016 16:52:20 +0200

Dear Ruediger,

Thanks a lot for your interest!
More below:

> On 27 Jun 2016, at 09:34, [email-masked] wrote:
> 
> Hi Michael,
> 

Interesting, thanks!

> From experience in rolling out new services, there seem to be German network operators dropping packets with unexpected DSCPs at IP interconnections.

This is very interesting - and I agree it's the wrong thing to do  ("be liberal in what you accept" ...).

Runa Barik (the Ph.D. student whose work this is) is planning to give a presentation to MAPRG about this, where he'll also talk about a new set of measurements that he did during a trip from Norway to India. There, he saw the same problem, and he also carried out traceroute-type measurements with the changed packets to see where it happened - with the result that it often is *inside* the network, i.e. not just 2-3 hops or so away from the client  (which is where many other packet drops happened). Even though it's a very small datapoint, it would confirm what you say about the location of this problem.

> I couldn't read from your doc whether you differentiated between fixed, WiFi and LTE/3G access. Please add this information, as policies may differ depending on the type of access. I think to recal…

Sorry: we didn't - but it's good advice, we will incorporate this in the future!

Cheers,
Michael