This article originally appeared in the May 2011 issue of NGN.
Speed is the primary way people describe their Internet connection, but speed tests are often misleading or completely wrong. The FCC (News - Alert) provides two tests at broadband.gov, one by Ookla (makers of speedtest.net) and one by M-Labs. But large differences are common (with Ookla giving larger numbers). What’s up?
A recent paper, “Understanding Broadband Speed Measurements” by Steve Bauer, David Clark and William Lehr, reports on their review of 400,000 tests made using the M-Labs network diagnostic tool. Fully 38 percent of those tests never managed to fill the access link. This means 38 percent of the tests never measured the available speed!
Most measurement complexity comes from the behavior of transmission control protocol (the TCP in TCP/IP). TCP varies the rate at which a source transmits data in response to the network’s ability to carry that data and the destination’s ability to absorb it. Delaying the signals that TCP responds to reduces throughput, but the receiver’s configuration can also be a limit. Remember, the first TCP/IP stacks were configured for 300bps modems and a 1.5mbps backbone.
For maximum speed, TCP needs both a large receive window at the destination and short round trip times. As speeds have increased, default windows sizes have grown, but many Windows XP systems remain in service and not all routers and firewalls implement TCP window scaling correctly, so small receive windows remain an issue. Meanwhile to deal with real world round trip times, the newest browsers open multiple TCP sessions in parallel.
One simple difference between the M-Labs and Ookla (News - Alert) (Speedtest.net) tests is that M-Labs has servers in just 14 cities globally while Speedtest.net has servers in 680 cities across 135 countries. The nearest M-Labs server to Boston is in Atlanta, whereas Speedtest.net (News - Alert) has servers right here in Boston. The difference is 98ms vs. 24ms, causing a significant impact on results.There’s also a structural difference between Speedtest.net and M-Labs. The M-Labs test uses a single TCP session while Ookla's Speedtest.net opens between one and eight TCP sessions in parallel in an attempt to, at least partially, get around TCP issues.Today, Ookla’s speedtest.net is more likely to reflect actual access link performance, but the M-Labs platform is more useful for researchers. M-Labs collects and stores more detailed information per test and makes all its data available to qualified researchers. And M-Labs is based on open source software, so how it works is visible and significant improvements may be contributed in the future.
Meanwhile, everyone talks about Internet speeds but few, in the ISP community or at the FCC, understand speed tests or understand which data is credible. Be careful when a politician quotes broadband data – it’s quite possibly wrong.
Edited by Stefania Viscusi