During Netflix money fight, Cogent’s other big customers suffered too

Cogent prioritized small business traffic over large network users.

Read the whole story

 

[KyleM]

wicketr[/url]":1yd5tgfp]In a world of "net neutrality" Cogent was just as guilty. You can't divide packets into wholesale vs retail unless you are doing exactly what Verizon is proposing with different tiers of access speeds.

I totally disagree. Cogent was doing the best they could given the constraints imposed on them by the 4 ISPs. If cogent transmits the traffic through their network, just to get to the edge router and not have enough capacity with their peer, this is not cogent fault, assuming they were willing to upgrade their connection, which they were.

 

[entropist65]

wicketr[/url]":1an3f1eb]In a world of "net neutrality" Cogent was just as guilty. You can't divide packets into wholesale vs retail unless you are doing exactly what Verizon is proposing with different tiers of access speeds.

I don't entirely disagree with you, but there's more nuance to it than that. Cogent wasn't making fast lanes and slow lanes in order to extract higher fees from anyone. This was truly network management. And this isn't something that they stand to gain from. If their wholesale customers can't get data delivered through Cogent's network, they're free to find another provider to carry their data. Case in point is Netflix, who ultimately paid to deliver data directly to Verizon, et al instead of through Cogent.

Things aren't quite the same for ISP subscribers. If the ISP is throttling data it's usually a case of tough luck because you don't have any other option.

 

[Matty]

Those who believe Netflix and Cogent should have to pay ISPs to upgrade their networks say the data proves Netflix and Cogent deliberately saturated the networks to force the ISPs into offering a better deal.

Why do we keep hearing this?

They didn't dump traffic on the ISPs. ISP customers were requesting it.

 

[Drizzt321]

KyleM[/url]":1nh3exwt]

wicketr[/url]":1nh3exwt]In a world of "net neutrality" Cogent was just as guilty. You can't divide packets into wholesale vs retail unless you are doing exactly what Verizon is proposing with different tiers of access speeds.

I totally disagree. Cogent was doing the best they could given the constraints imposed on them by the 4 ISPs. If cogent transmits the traffic through their network, just to get to the edge router and not have enough capacity with their peer, this is not cogent fault, assuming they were willing to upgrade their connection, which they were.

Yea, Cogent had (has) the capacity to transit all of the data, and necessary interconnection points, just as traffic increased with some of it's peers they refused to upgrade connections. I imagine most previous connection upgrades were in response to ISP consumers requesting more traffic (e.g. sorta like what they were doing with Netflix), it's just with Netflix it ended up being much more data all at the same time, and so the ISPs really didn't like that. Since they are all in the content delivery business as well.

Now, I understand there are real (and large) costs with adding ports/routers to handle a load like that. But. ISPs demanded payments as if traffic coming from Cogent were _transiting_ their network, not _terminating_ on the network due to a request originating from that network. BIG difference.

As for the QOS to decide who's packets get dropped...this probably falls under reasonable network management. They dropped packets from customers who use Cogent primarily as a more standard ISP, so various small, medium, and large businesses who used it in the normal daily business, email, web, regular data transfer, B2B, etc. Customers like Netflix, who send large amounts of data in bulk were the first to have their packets dropped.

 

[KyleM]

nweaver[/url]":22hx8u6a]Three important notes:

The problems with Cogent started when they started carrying Netflix. The sudden collapse of Cogent's performance when they started providing transit for Netflix is as telling as the sudden improvement. Arguably, Cogent should never have sold their service to Netflix in the first place: Cogent wasn't capable of reliably doing the job.

The problems with other customers stopped when Cogent turned on quality-of-service markings, creating a slow lane for Netflix and a fast lane for everybody else. But this fast lane/slow lane only applies within Cogent's network, further suggesting this was primarily a Cogent problem.

Cogent knew they had a problem with Netflix traffic right from the start (this traffic congestion was undoubtedly blowing up Cogent's own internal monitoring), but it took months before they actually responded to the problem and implemented the quality-of-service solution.

You have no proof to back any of these statements up. Cogents network did have the capacity to handle the netflix traffic. It is only at the edge where the wholesale traffic was getting dropped due to congestion on the peering links. If cogents internal network was congested, performance with other networks would have been impacted.

I think people generally have a misunderstanding of the role of transit providers. In any sane world, no ISP would ever be able to demand money from transit providers or content providers. The ISPs customers are the ones who are requesting the data, and they are paying for the privilege to do that.

 

[Matty]

The Medium write up on this was good not just for Crawford's quote.

It had a bit of an illumination of the retail side Cogent customers. School districts not able to make payroll due to the packet loss caused by Comcast customers trying to watch Netflix. Remote workers not being able to VPN into work.

https://medium.com/backchannel/jammed-e474fc4925e4

It's a fine line to walk as a net neutrality proponent. We don't want the ISPs to degrade our traffic for payola. On the other hand watching a movie (or a bit of extra buffering) seems less important than say medical records being moved for a critical patient or even techers getting paid.

 

[akurczyn]

This is no news for me, I live outside of USA and my ISP's main link there is thru Cogent. Time and again page load times go up and a simple traceroute shows high latency times on Cogent hops.

 

[jlivingood]

"The system “took a lot of testing, took a lot of engineering time and resources,” Kilmer told Ars. Kilmer wrote that the system is “consistent with recommendations from BITAG (Broadband Internet Technical Advisory Group).”"

Except it is not really consistent with the BITAG document. I was one of the contributors to that BITAG report, and the editor was Scott Jordan (the new FCC CTO). It is worth people actually reading section 7, and in particular 7.1. Find it at http://www.bitag.org/documents/BITAG_-_ ... Report.pdf.

Section 7.1 says some important things, like this should be on a network management website (not an obscure M-Lab mailing list). This isn't proactive disclosure, it was basically forced by the release of the M-Lab report and the 10/31/14 M-Lab blog post pointing to 0x48 in the packets.

From 7.1:
The BITAG suggests that ISPs and ASPs should disclose information about their user- or application- based network management and congestion management practices for Internet services such that the information is readily accessible to the general public. This information should be made available on network operators’ public web sites and through other typically used communications channels, including mobile apps, contract language, or email. ISPs and ASPs may choose to use a layered notice approach, using a simple, concise disclosure that includes key details of interest to consumers complemented by a more thorough and detailed disclosure for use by more sophisticated users, application developers, and other interested parties. The detailed disclosure should include:
• descriptions of the practices;
• the purposes served by the practices;
• what types of traffic are subject to the practices, if not all traffic, and, if appropriate, a general explanation of how traffic types are identified on the network;
• the practices’ likely effects on end users’ experiences;
• the triggers that activate the use of the practices and whether those triggers are user- or application- based;
• the approximate times at which the practices are used, if they are limited to particular times of day; and
• which subset of users may be affected, if not all, e.g. the geographic boundaries in which the practices are used.
The disclosure should also include the predictable impact, if any, of a user’s other subscribed network services on the performance and capacity of that user’s broadband Internet access services during times of congestion, where applicable. These disclosures should complement ISPs’ other disclosures that describe their Internet service offerings, expected and actual access speeds, performance characteristics, and suitability for supporting particular kinds of applications.

All of these disclosures may be made without divulging competitively sensitive information and should be consistent with the requirements described in the FCC’s Open Internet Order [Open Internet Order].

 

[sonicmerlin]

Sigh, the attempts at blaming Cogent are ridiculous. Cogent gains nothing by harming their wholesale customers. They have plenty of competition so if their customers are annoyed by their practices they can switch to someone else. The problem is all the peering points were saturated b/c Verizon et. al weren't upgrading them the way they usually do.

 

[jlivingood]

After my post above with the actual details of the BITAG document (specifically 7.1) I notice the story here was updated to say "Cogent said it implemented the network management in a "visible" and "transparent" way, but the company didn't discuss it publicly when the system was implemented."

If it did it visibly and transparently, but did not discuss it publicly, how did they do it exactly?

IMHO any ISP that takes transparency seriously ought to be able to tell you immediately where their network management disclosures are. The site I maintain at work is http://networkmanagement.comcast.net. Many other ISPs have similar such sites.

And while IANAL, perhaps folks here missed the FCC notice in July at http://www.fcc.gov/document/fcc-isps-in ... rency-rule entitled "FCC Enforcement Advisory: Broadband Providers Must Disclose Accurate Information to Protect Consumers". See "What does the Transparency Rule Require" in that document. How do these disclosures measure up?

 

[Fearknot]

sonicmerlin[/url]":1g0z4hls]Sigh, the attempts at blaming Cogent are ridiculous. Cogent gains nothing by harming their wholesale customers. They have plenty of competition so if their customers are annoyed by their practices they can switch to someone else. The problem is all the peering points were saturated b/c Verizon et. al weren't upgrading them the way they usually do.

True. And I have absolutely no sympathy for Verizon et al.

On the other hand, Cogent is the one that has the peering agreement with the ISPs. If I'm a paying wholesale customer of Cogent, I expect that Cogent maintains effective peering agreements. When Cogent computes their resource requirements to support taking Netflix as customer, they can't just ignore the edge connectors. I think that the companies whose traffic was slowed down have a legitimate complaint that Cogent took on more traffic (from Netflix) than they could deliver, and they knew it.

 

[jbrodkin]

jlivingood[/url]":2fhfznhz]After my post above with the actual details of the BITAG document (specifically 7.1) I notice the story here was updated to say "Cogent said it implemented the network management in a "visible" and "transparent" way, but the company didn't discuss it publicly when the system was implemented."

If it did it visibly and transparently, but did not discuss it publicly, how did they do it exactly?

IMHO any ISP that takes transparency seriously ought to be able to tell you immediately where their network management disclosures are. The site I maintain at work is http://networkmanagement.comcast.net. Many other ISPs have similar such sites.

And while IANAL, perhaps folks here missed the FCC notice in July at http://www.fcc.gov/document/fcc-isps-in ... rency-rule entitled "FCC Enforcement Advisory: Broadband Providers Must Disclose Accurate Information to Protect Consumers". See "What does the Transparency Rule Require" in that document. How do these disclosures measure up?

Thanks for the info, glad to have a Comcast employee here. Dan Rayburn pointed that out to me, I didn't notice your post until after I added it in.

 

[ReductiMat]

jlivingood[/url]":37oo8efb]If it did it visibly and transparently, but did not discuss it publicly, how did they do it exactly?

Exactly my thought.

Queue Inigo Montoya meme.

 

[hopews]

jbrodkin[/url]":141nnlrk]

jlivingood[/url]":141nnlrk]After my post above with the actual details of the BITAG document (specifically 7.1) I notice the story here was updated to say "Cogent said it implemented the network management in a "visible" and "transparent" way, but the company didn't discuss it publicly when the system was implemented."

If it did it visibly and transparently, but did not discuss it publicly, how did they do it exactly?

IMHO any ISP that takes transparency seriously ought to be able to tell you immediately where their network management disclosures are. The site I maintain at work is http://networkmanagement.comcast.net. Many other ISPs have similar such sites.

And while IANAL, perhaps folks here missed the FCC notice in July at http://www.fcc.gov/document/fcc-isps-in ... rency-rule entitled "FCC Enforcement Advisory: Broadband Providers Must Disclose Accurate Information to Protect Consumers". See "What does the Transparency Rule Require" in that document. How do these disclosures measure up?

Thanks for the info, glad to have a Comcast employee here. Dan Rayburn pointed that out to me, I didn't notice your post until after I added it in.

It seems to me that the linked FCC advisory likely doesn't apply to cogent as it is not a provider of broadband internet acccess services, but a backbone provider. From the linked FCC Advisory:

"Who Is Subject to the Transparency Rule? The rule applies to every provider of broadband Internet access services in the United States. This includes fixed broadband Internet access providers (for example, cable companies, landline telephone companies, and fixed wireless or satellite service providers),as well as mobile broadband Internet access providers (for example, mobile wireless providers that offer data plans for Internet access for smartphones)."

 

[jlivingood]

hopews[/url]":34c562w9]

It seems to me that the linked FCC advisory likely doesn't apply to cogent as it is not a provider of broadband internet acccess services, but a backbone provider. From the linked FCC Advisory:

"Who Is Subject to the Transparency Rule? The rule applies to every provider of broadband Internet access services in the United States. This includes fixed broadband Internet access providers (for example, cable companies, landline telephone companies, and fixed wireless or satellite service providers),as well as mobile broadband Internet access providers (for example, mobile wireless providers that offer data plans for Internet access for smartphones)."

Interestingly I had the same initial reaction!

But I checked Cogent's website and found they offer a service called Dedicated Internet Access - see http://www.cogentco.com/en/products-and ... net-access. That page says "Our Internet access is dedicated, non-oversubscribed and backed by one of the best SLAs available. Cogent is a refreshing change from your traditional telephone company. We will impress you with our simple product, rapid service installation and proactive customer support. Our ability to provide reliable, yet affordable bandwidth has turned thousands of businesses into loyal customers."

I also checked their most recent SEC 10-K (http://google.brand.edgar-online.com/di ... m%253dCCOI). That SEC filing said "Cogent Communications Group, Inc. (the “Company”) is a Delaware corporation and is headquartered in Washington, DC. The Company is a facilities-based provider of low-cost, high-speed Internet access and Internet Protocol (“IP”) communications services. The Company’s network is specifically designed and optimized to transmit data using IP. The Company delivers its services to small and medium-sized businesses, communications service providers and other bandwidth-intensive organizations in North America and Europe. The Company recently began expansion into Japan.

The Company offers on-net Internet access services exclusively through its own facilities, which run from its network to its customers’ premises. Because of its integrated network architecture, the Company is not dependent on local telephone companies to serve its on-net customers. The Company’s on-net service consists of high-speed Internet access and IP connectivity ranging from 100 Megabits per second to 10 Gigabits per second of bandwidth."

 

[jlivingood]

jlivingood[/url]":7ommnet8]

hopews[/url]":7ommnet8]

It seems to me that the linked FCC advisory likely doesn't apply to cogent as it is not a provider of broadband internet acccess services, but a backbone provider. From the linked FCC Advisory:

"Who Is Subject to the Transparency Rule? The rule applies to every provider of broadband Internet access services in the United States. This includes fixed broadband Internet access providers (for example, cable companies, landline telephone companies, and fixed wireless or satellite service providers),as well as mobile broadband Internet access providers (for example, mobile wireless providers that offer data plans for Internet access for smartphones)."

Interestingly I had the same initial reaction!

But I checked Cogent's website and found they offer a service called Dedicated Internet Access - see http://www.cogentco.com/en/products-and ... net-access. That page says "Our Internet access is dedicated, non-oversubscribed and backed by one of the best SLAs available. Cogent is a refreshing change from your traditional telephone company. We will impress you with our simple product, rapid service installation and proactive customer support. Our ability to provide reliable, yet affordable bandwidth has turned thousands of businesses into loyal customers."

I also checked their most recent SEC 10-K (http://google.brand.edgar-online.com/di ... m%253dCCOI). That SEC filing said "Cogent Communications Group, Inc. (the “Company”) is a Delaware corporation and is headquartered in Washington, DC. The Company is a facilities-based provider of low-cost, high-speed Internet access and Internet Protocol (“IP”) communications services. The Company’s network is specifically designed and optimized to transmit data using IP. The Company delivers its services to small and medium-sized businesses, communications service providers and other bandwidth-intensive organizations in North America and Europe. The Company recently began expansion into Japan.

The Company offers on-net Internet access services exclusively through its own facilities, which run from its network to its customers’ premises. Because of its integrated network architecture, the Company is not dependent on local telephone companies to serve its on-net customers. The Company’s on-net service consists of high-speed Internet access and IP connectivity ranging from 100 Megabits per second to 10 Gigabits per second of bandwidth."

I suppose they could argue they are not a provider of broadband Internet access, but then that'd mean they provide narrowband and that's not accurate based on the speed cited above. At the end of the day, I think a company either provides Internet access or they don't. And if they do, well then they all ought to play by the same transparency rules.

 

[ChickenHawk]

So packets were going to be dropped because the link was saturated, and they made a judgement call about which packets it was gunna be.

Thats not fast lane / slow lane. Thats not artificiallly capping for profit, thats a judgement call for network management *Shrugs*.

If only the other side had agreed to the FOC upgrade instead of playing Mafioso....

 

[Abhi Beckert]

ChickenHawk[/url]":30q626zq]So packets were going to be dropped because the link was saturated, and they made a judgement call about which packets it was gunna be.

And shockingly, they dropped packets from the "peer" who doesn't pay anything but sends orders of magnitude more data than they receive, instead of about equal which is the definition of "peer".

How can anyone seriously expect them to drop packets from paying customers? This kind of network management is normal.

 

[Fatesrider]

Technical aspects aside, I have a slightly different take on this whole issue that may be overlooked in the minutiae:

The end user pays for a specific tier of download/upload speeds. The ISP is contracted to supply the end user's data at that speed.

Now before anyone blows fuses, I do realize that there are disclaimers that place limits on that, but still, if someone is paying for 25/10 or 25/5 or 100/50, they should be getting at LEAST 80% of that speed ALL THE TIME from ANYONE supplying it at or above that speed. The ISP is contracted to only throttle down to match the speed service the end user is paying for.

It sounds naive, but this is the whole crux of the matter to most people. They pay for a service and the ISP's play tricks to provide less than what the end user has paid to receive. Different issue than what's described here? No, not really. The ISP's control the content delivery through their networks. If they don't have the capacity to ALL of their data at the advertised speeds, they should make it far more clear to the end user that they're paying for a service that they're probably not going to see happen much, if at all. Failing that disclosure, they need to make good on delivering what they advertise.

This is where common carrier status comes in. Ensure enough throughput to meet demand and act as the dumb pipes they are treating all data, regardless of origin, the same.

The ONLY reason they need to throttle is that they are unwilling to meet the data load demands placed on their networks. The only reason they'd do that is to maximize profits and minimize costs.

And that's where any sympathy from me goes away and their rationalizations (justified or not) for throttling fall on MY deaf ears. A business that doesn't cater to the desires, or provide the promised service to, its customers, and can still stay in business at a profit is one that is either in dire need of regulation or a monopoly, or both. In this case, it's both.

It's odd to speak of "net neutrality" needing "regulation", but if corporate America, already an international embarrassment when it comes to Internet service, can't stop fleecing everyone connected to an ISP and live up to its advertisements 99.99999999% of the time, then it's time to MAKE THEM DO SO.

Sorry if this hits the returns from investors (well, not VERY sorry), but we need to make our Internet better, more reliable and far more impartial of what we get over it. Since the ISP's prioritize profits over service, and have an effective monopoly on their markets, regulation is the only way left to do that.

 

[d4Njv]

The ISPs make it sound like they are doing Cogent or Netflix a favor by allowing traffic through at the advertised speeds. But Netflix's income doesn't really hinge on how smoothly the ISPs end up delivering Netflix traffic to its subscribers, since by the traffic reaches the edge of the ISPs it has already been paid for. You don't decide whether to pay Netflix only after you finish watching the movie. ISPs aren't doing Netflix a favor; they would merely be upholding their end of their bargain with their subscribers.

 

[pcarter]

If I am paying Netflix to deliver me videos and they recently move their traffic over to a crappy delivery transit provider with a long history of peering disputes and carrier relationship abuse.... Can I blame my ISP for that?

Yes I am paying my ISP (and generally get good broadband), but I am also paying Netflix.

 

[realwarder]

Comcast: Thank you customer for paying to provide data. Now Cogent, pay us because our customer wants your data. Umm. Something sounds wrong here.

 

[Abhi Beckert]

Fatesrider[/url]":36wu2fmb]Technical aspects aside, I have a slightly different take on this whole issue that may be overlooked in the minutiae:

The end user pays for a specific tier of download/upload speeds. The ISP is contracted to supply the end user's data at that speed.

Now before anyone blows fuses, I do realize that there are disclaimers that place limits on that, but still, if someone is paying for 25/10 or 25/5 or 100/50, they should be getting at LEAST 80% of that speed ALL THE TIME from ANYONE supplying it at or above that speed. The ISP is contracted to only throttle down to match the speed service the end user is paying for.

It sounds naive, but this is the whole crux of the matter to most people.

At work we have a connection where our ISP guarantees to give us at least 50% of the speed we pay for.

It costs a couple thousand dollars a month to have that contract, and we had to negotiate for _months_ to get a price that low, pitting different providers against each other. One of them gave us such a cheap deal because they have dark fibre running past our building that isn't being used by anyone (until we connected to it).

Forget about 80%, nobody will sell that to you for any price.

Alao they still have an exception for downtime and disruptions that can drop us below 50% temporarily. The guarantee only applies when their network is operating without faults.

In practice we always get exactly 100% because it's almost unheard of for half the customers on an ISP to use the internet at the same time for more than a few microseconds.

 

[xnef1025]

While true that Cogent spins the story their way and Comcast another, let's recall that Level 3 reported the same issues with the ISPs as Cogent when they carried Netflix traffic. If I recall they offered to foot the bill for the new equipment and were turned down because Comcast and company wanted to hold out for regular money.

So are we to believe that Level 3 and Cogent are both terrible freeloaders out to bismirch the good names of Comcast, ATT, and Verizon? Isn't it more likely that these giant corporations with a history of anti-consumer shenanigans were abusing their positions closer to the truth?

 

[tzs]

Google "cogent peering issues" or "cogent peering dispute" or "cogent depeering". Over the last decade, they have had a lot of peering disputes with such companies as Level 3, Google, Hurricane Electric, China Telecom, Sprint, and Verizon. There is something not right going at Cogent.

 

[tim305]

entropist65[/url]":3qj571vf]

wicketr[/url]":3qj571vf]In a world of "net neutrality" Cogent was just as guilty. You can't divide packets into wholesale vs retail unless you are doing exactly what Verizon is proposing with different tiers of access speeds.

I don't entirely disagree with you, but there's more nuance to it than that. Cogent wasn't making fast lanes and slow lanes in order to extract higher fees from anyone. This was truly network management. And this isn't something that they stand to gain from. If their wholesale customers can't get data delivered through Cogent's network, they're free to find another provider to carry their data. Case in point is Netflix, who ultimately paid to deliver data directly to Verizon, et al instead of through Cogent.

Things aren't quite the same for ISP subscribers. If the ISP is throttling data it's usually a case of tough luck because you don't have any other option.

Sure they were. Higher paying retail customers got better service than lower paying wholesale customers like Netflix. I don't know what could more obvious.

On the other hand, I don't see anything wrong with it. Absent the fast lanes, Cogent could have achieved roughly the same effect by accepting less wholesale traffic into their network, just as Comcast was doing at the other end of the pipe. The internal prioritization lets them carry (a little) more traffic without jeopardizing the high paying retail traffic. Though you would expect the wholesale customers to start complaining about the loss rates at some point.

 

[LoneSnark]

entropist65[/url]":2xwjre8v]

I don't entirely disagree with you, but there's more nuance to it than that. Cogent wasn't making fast lanes and slow lanes in order to extract higher fees from anyone. This was truly network management. And this isn't something that they stand to gain from. If their wholesale customers can't get data delivered through Cogent's network, they're free to find another provider to carry their data. Case in point is Netflix, who ultimately paid to deliver data directly to Verizon, et al instead of through Cogent.

Things aren't quite the same for ISP subscribers. If the ISP is throttling data it's usually a case of tough luck because you don't have any other option.

Absolutely untrue. Had netflix been willing to pay to upgrade itself to "retail", or fast lane, status, they could have. I'm sure cogent would have loved having the extra money.

And the vast majority of ISP subscribers do have alternatives, and sometimes do avail themselves of them, just as Netflix did here. Using your definition, if this is not an instance of "fast lane service", then the only valid examples of fast lane service only could apply to that small fraction of ISP subscribers which lack any alternatives, be they Cellular, Satellite, etc.

 

[tim305]

KyleM[/url]":rfah7pco]

wicketr[/url]":rfah7pco]In a world of "net neutrality" Cogent was just as guilty. You can't divide packets into wholesale vs retail unless you are doing exactly what Verizon is proposing with different tiers of access speeds.

I totally disagree. Cogent was doing the best they could given the constraints imposed on them by the 4 ISPs. If cogent transmits the traffic through their network, just to get to the edge router and not have enough capacity with their peer, this is not cogent fault, assuming they were willing to upgrade their connection, which they were.

But, Cogent is not selling transport to edge routers of other networks. They are selling transport to destination IP addresses. It is up to them to make arrangements to get that traffic delivered. If they can't do it, then they should not accept the traffic. There has never been any rule or understanding that certain networks must carry traffic for free. A lot of networks engage in settlement free peering, but that is purely at their option, as a business decision.

When a residential ISP controls the only route to vast numbers of users, they have tremendous bargaining power in these negotiations and could set exorbitant rates to traverse their network. But, we just haven't seen that. Transit prices, which would include whatever connections fees the major ISP's have been charging, have fallen from $1200/Mbps/month in 1998 to less then a $1/Mbps/month in 2014. It's time we see more of those cost reductions on the content provider side passed on to residential ISP subscribers, or put into building out better residential networks.

 

[jlivingood]

LoneSnark[/url]":2dchkhy1]Absolutely untrue. Had netflix been willing to pay to upgrade itself to "retail", or fast lane, status, they could have. I'm sure cogent would have loved having the extra money.

That suggests that Netflix knew they were getting sub-optimal, slow lane service. If so, it would be disappointing that they'd also be running a parallel campaign to heap shame on ISPs for their declining quality. Certainly with their measurements they saw the decline once they pushed traffic to Cogent, so I guess they just decided it was worth the sacrifice in quality.

 

[jlivingood]

tim305[/url]":1tzobn9k]But, Cogent is not selling transport to edge routers of other networks. They are selling transport to destination IP addresses. It is up to them to make arrangements to get that traffic delivered. If they can't do it, then they should not accept the traffic. There has never been any rule or understanding that certain networks must carry traffic for free. A lot of networks engage in settlement free peering, but that is purely at their option, as a business decision.

Bingo! The M-Lab data seems to show that the the very moment that a new major source came onto the network that congestion appeared. Which seems to suggest quite plainly that they oversold what they could deliver.

 

[wiz420]

tim305[/url]":2ewtxv1n]

KyleM[/url]":2ewtxv1n]

wicketr[/url]":2ewtxv1n]In a world of "net neutrality" Cogent was just as guilty. You can't divide packets into wholesale vs retail unless you are doing exactly what Verizon is proposing with different tiers of access speeds.

I totally disagree. Cogent was doing the best they could given the constraints imposed on them by the 4 ISPs. If cogent transmits the traffic through their network, just to get to the edge router and not have enough capacity with their peer, this is not cogent fault, assuming they were willing to upgrade their connection, which they were.

But, Cogent is not selling transport to edge routers of other networks. They are selling transport to destination IP addresses. It is up to them to make arrangements to get that traffic delivered. If they can't do it, then they should not accept the traffic. There has never been any rule or understanding that certain networks must carry traffic for free. A lot of networks engage in settlement free peering, but that is purely at their option, as a business decision.

When a residential ISP controls the only route to vast numbers of users, they have tremendous bargaining power in these negotiations and could set exorbitant rates to traverse their network. But, we just haven't seen that. Transit prices, which would include whatever connections fees the major ISP's have been charging, have fallen from $1200/Mbps/month in 1998 to less then a $1/Mbps/month in 2014. It's time we see more of those cost reductions on the content provider side passed on to residential ISP subscribers, or put into building out better residential networks.

We could easily wind up in a place where the ISP toll could exceed the cost of transit. Do you really think that transit is the largest cost to the content providers? I doubt it: The transit market is actually competitive.

 

[killamike]

Abhi Beckert[/url]":1nypjeyx]
At work we have a connection where our ISP guarantees to give us at least 50% of the speed we pay for.

It costs a couple thousand dollars a month to have that contract, and we had to negotiate for _months_ to get a price that low, pitting different providers against each other. One of them gave us such a cheap deal because they have dark fibre running past our building that isn't being used by anyone (until we connected to it).

Forget about 80%, nobody will sell that to you for any price.

Alao they still have an exception for downtime and disruptions that can drop us below 50% temporarily. The guarantee only applies when their network is operating without faults.

In practice we always get exactly 100% because it's almost unheard of for half the customers on an ISP to use the internet at the same time for more than a few microseconds.

Wow, the US gets absolutely FLEECED on internet costs. In my country in Eastern Europe our ISP tiers (for domestic and corporate customers alike) are always advertised for their 'minimum' guaranteed speeds. So I pay for a 50mbit guarantee line (incidentally it costs me just a hair under $16p/m under current exchange rates), but I end up getting around 90mbits on speedtest.net very consistently (rarely does it drop to maybe 70mbit).

US Netflix via DNS proxy is always HD within the first 30secs of streaming and almost never drops down to SD (once every couple of weeks of daily streaming). I have several computers with torrent clients running continuously and can hit speeds of around 9-10MB/sec (megabytes not megabits) with a good swarm of local peers.

For the record we have very little regulation (more lately but basically none in the 90s up to mid 00s), but we have real hardcore competition. There are litterally 10s if not 100s of independant ISPs to choose from. Irocnically, creating a real free market - in eastern europe of all places. THe problem in the US is basically state-backed mono/duo-polies who get to do what they want and charge what they want.

the only solutions going forward for the US are either REAL heavy regulation (which the FCC does not appear to be capable or willing to do) or opening up the market for real competition (which is prevented by millions and millions of lobby dollars)... :/

 

[ChickenHawk]

LoneSnark[/url]":1t68wlxt]

entropist65[/url]":1t68wlxt]

I don't entirely disagree with you, but there's more nuance to it than that. Cogent wasn't making fast lanes and slow lanes in order to extract higher fees from anyone. This was truly network management. And this isn't something that they stand to gain from. If their wholesale customers can't get data delivered through Cogent's network, they're free to find another provider to carry their data. Case in point is Netflix, who ultimately paid to deliver data directly to Verizon, et al instead of through Cogent.

Things aren't quite the same for ISP subscribers. If the ISP is throttling data it's usually a case of tough luck because you don't have any other option.

Absolutely untrue. Had netflix been willing to pay to upgrade itself to "retail", or fast lane, status, they could have. I'm sure cogent would have loved having the extra money.
.

I think that statement you've made is untrue. The article makes it clear Cogent made the distinction based on the usage type of the customer - those using things like VPN that were time sensitive were classed as "retail"

 

[superchkn]

KyleM[/url]":1eaj2u3j]

wicketr[/url]":1eaj2u3j]In a world of "net neutrality" Cogent was just as guilty. You can't divide packets into wholesale vs retail unless you are doing exactly what Verizon is proposing with different tiers of access speeds.

I totally disagree. Cogent was doing the best they could given the constraints imposed on them by the 4 ISPs. If cogent transmits the traffic through their network, just to get to the edge router and not have enough capacity with their peer, this is not cogent fault, assuming they were willing to upgrade their connection, which they were.

Indeed. It pays to understand how QoS functions, and how Cogent was specifically configuring their network.

A slow and fast lane would be to setup a queue limited to some arbitrary bandwidth lower than that of your pipe for the slow lane, and unrestricted (and prioritized) for the fast lane. This would result in the slow always being relatively slow regardless of the bandwidth available and in times of congestion, the packets in the fast pipe would be prioritized over the slow.

Instead, what it sounds like Cogent did is to prioritize one set of traffic over another, which would only make any significant difference when the pipe was congested.

Those configurations listed above are not equivalent despite the way some ISPs are trying to spin it.

 

[scpATL4now]

I would love to see (just for a day, an hour even) Cogent refuse to carry Comcast's traffic. Let people know that their internet traffic is made up of a lot more than Comcast's network

 

[KyleM]

Abhi Beckert[/url]"tv7zf87]

ChickenHawk[/url]"tv7zf87]So packets were going to be dropped because the link was saturated, and they made a judgement call about which packets it was gunna be.

And shockingly, they dropped packets from the "peer" who doesn't pay anything but sends orders of magnitude more data than they receive, instead of about equal which is the definition of "peer".

I don't think you know what you are talking about at all. When Tier 1 providers peer they usually expect similar levels of traffic in each direction. When an ISP peers with a Tier 1 provider, there really is no expectation of balanced traffic. Retail ISP customers always download a lot more than they upload. Requests for a resource are usually many orders of magnitude smaller than the resource themselves, especially when it comes to video. The thing is though, that the ISP customer actually requested that data. It's not cogent just sending a firehose of data, but it is ISP customers who are requesting that data and the ISP refusing to upgrade the connections to handle those requests. This is 100% the ISPs fault and not the transit providers or netflixs fault.

 

[WaveRunner]

KyleM[/url]":evqrxoaa]

Abhi Beckert[/url]":evqrxoaa]

ChickenHawk[/url]":evqrxoaa]So packets were going to be dropped because the link was saturated, and they made a judgement call about which packets it was gunna be.

And shockingly, they dropped packets from the "peer" who doesn't pay anything but sends orders of magnitude more data than they receive, instead of about equal which is the definition of "peer".

I don't think you know what you are talking about at all. When Tier 1 providers peer they usually expect similar levels of traffic in each direction. When an ISP peers with a Tier 1 provider, there really is no expectation of balanced traffic. Retail ISP customers always download a lot more than they upload. Requests for a resource are usually many orders of magnitude smaller than the resource themselves, especially when it comes to video. The thing is though, that the ISP customer actually requested that data. It's not cogent just sending a firehose of data, but it is ISP customers who are requesting that data and the ISP refusing to upgrade the connections to handle those requests. This is 100% the ISPs fault and not the transit providers or netflixs fault.

Peering wasn't designed to handle redundant amounts of video data. We've had CDNs for the past 15 years to allow for video and other large files. When I download a steam game, it's not coming from transit, when I watch a youtube or CNN video it's not coming from transit, when I download any ISO from MSDN. Seems to me Netflix and Cogent want to redefine the way the internet functions.

What I do expect though is once my ISP is paid to handle the CDN content that they give it to me at line speed. And that's exactly the offer on the table to Netflix, the same deal every other content provider has agreed to.