TMCnet News
RootMetrics EE's 4G network performance in Cardiff, Liverpool and Sheffield(M2 PressWIRE Via Acquire Media NewsEdge) Background In late November and early December 2012, RootMetrics conducted thousands of mobile internet, call, and text tests throughout the Cardiff, Liverpool and Sheffield areas to provide consumers with information on which operator provided: 1. The fastest and most reliable mobile internet network 2. The most reliable call service (fewest dropped and blocked calls, etc.) 3. The best text service (fastest delivery, etc.). Measurements included EE's 4G-enabled service (only available with a buy-up plan) as well as EE's legacy T-Mobile and Orange networks (which can still be measured separately, though for EE consumers who do not buy-up to 4G, their phone may say "EE" and they may not be able to tell which network they are on). In testing, it has been possible to show how EE's new 4G network is performing in each area - in addition to providing wider findings about all the network operators and how they are faring as a whole. This report provides a comprehensive analysis of EE's 4G network in Cardiff, Liverpool and Sheffield as defined by Eurostat's Large Urban Zones. These areas - in contrast to the Ofcom reports - can be quite extensive and include the many suburban and rural areas outside a given city that are economically or functionally linked to the core city. For this reason, testing follows population distribution, ensuring more tests in the cities, fewer in outlying towns, etc. For the three areas specifically measured, the Large Urban Zone incorporates: 1. Market name: Cardiff Actual area measured: Cardiff, Caerphilly, Rhondda Cynon Taf, Vale of Glamorgan Dates scouted: 26-30 November 2012 Miles driven: 1,168 KM, 726 Miles 2. Market name: Liverpool Actual area measured: Merseyside (county) Dates scouted: 2-6 December 2012 Miles driven: 1,141 KM, 709 Miles 3. Market name: Sheffield Actual area measured: Sheffield (City), South Yorkshire (Barnsley), Derbyshire (High Peak, North East Derbyshire and Chesterfield districts) Dates scouted: 2-6 December 2012 Miles driven: 1,336 KM, 830 Miles Note that findings that make reference to the 'city' in the following are based on UK Ordnance Survey boundaries of the metropolitan areas or appropriate district. To simulate a typical consumer's mobile experience in the area, RootMetrics conducted tests based on where, when, and how people use their devices most frequently. A full methodology can be found at the end of this document. Findings Key Findings Rumblings versus realities: There has been much speculation and many competitive claims implying EE's 4G service will under-deliver. In contrast, we find the EE 4G roll-out is largely delivering on its public promises (with room for some improvement) and is a measured success when compared to similar roll-outs in the United States. Reliability is high and speeds meet those EE has pledged to achieve, though the speeds are lower than the full potential of 4G. Counter to competitive marketing that we have seen, we observed fewer differences between indoor and outdoor service than other networks have implied may occur. We found significant limitations in geographic availability, so it's very important for individuals to know if their towns are covered before signing up for the extra expense. We also found a mixed picture of how often we were on 4G in our testing (from a high of 71.3% of download tests on 4G in Cardiff [city] to a low of 48.5% of download tests on 4G in Sheffield [city]). While the low-end of this range may sound very low, it is typical of other new roll-outs we have seen in new 4G cities in the US (at the high-end, the percentage of tests on 4G is quite good). Average speed: EE claims that the average speed of its 4G network would be 8-12 Mbps (source: EE data). Looking just within the city boundaries a few weeks after the launch, EE's 4G network delivered on these promises: Liverpool and Cardiff met the bar with download speeds of 9.4 Mbps and 11.8 Mbps, respectively. Sheffield just barely missed the mark with an average download speed of 7.9 Mbps, though within the margin of error at the low end of EE's commitment. In each case, the speed is more than twice as fast as the next fastest network (Vodafone in Liverpool; Three in Cardiff and Sheffield), and fast enough that a consumer would see a true difference in performance with things like HD video (YouTube), posting photos to social media, etc. However, 4G launches in other countries can be much faster than these speeds. For example, AT&T when launching its 4G service in Dallas jumped from average pre-4G download speeds of 4.3 Mbps to average download speeds of 17.2 Mbps. The speeds seen in UK cities are good, but not necessarily taking full advantage of 4G's capabilities. 4G-only speed: The gap between 4G's potential and EE's promised speed is also revealed when looking at the speeds achieved when only on 4G (versus the averages above). In Cardiff (city), for instance, the 4G-only average download speed was 16.2 Mbps - nearly 40% faster than the average download speed of 4G and non-4G (i.e. the typical experience). In Liverpool (city), the 4G-only average download speed was 14.4 Mbps - 34.7% faster than the average download speed of 4G and non-4G. Given that EE's network seems capable of delivering faster speeds (on account of 4G only tests), we believe the performance gap occurs because of the percentage of tests not on the 4G network (rather than a network design limitation). Frequency of 4G connections: The real effect of jumping on or off of a 4G connection will be felt in terms of speed (and what a user can or cannot do because of it), so we don't tend to focus on how often 4G is available. In the UK, however, networks charge a premium to access the service so it is fair to look at the percentage of tests over 4G. In our tests, we found significant regional variations regarding availability of the service between the cities: our phones made and kept a 4G connection 71.3% of the time on download tests in Cardiff [city only] compared to Liverpool [city only] at 58.7% and Sheffield [city only] at 48.5%. For context, the numbers even at the low end are in line of what we have seen from US networks when they have launched new 4G markets. 4G geographic availability: While the percentage of 4G connections in Cardiff, Liverpool and Sheffield city centres is good (in comparison to similar global roll-outs and with some variations), mapping of the individual tests locations indicates 4G had not been deployed widely (or it was not available at all) outside the city boundaries. This is to be expected given EE's promise to launch in "cities" rather than the larger "LUZ" area tested by RootMetrics. A question, however, is whether someone who lives in St Helens, for instance - a main suburb just 12 miles from James Street Station in Liverpool centre and in which we were not able to make any true 4G connection - thinks of himself as outside the city. o Near Cardiff, 4G availability outside the city was only found in the near-in towns in the Vale of Glamorgan, and the availability of 4G dropped to 35.1% of upload tests and 33.6% of download tests. o Outside Liverpool, we found 4G in Birkenhead, Wallasey, Kirkby and Bootle; the availability of 4G dropped to 45.8% of upload tests and 43.1% of download tests. o Outside Sheffield, 4G was only seen in very close-in villages and the availability of 4G dropped to 30.4% of download tests and 32.1% of upload tests. Network reliability: Reliability assesses whether a data test can be started and completed successfully. In each city, EE's network proved reliable, with 94.0% of tests in Cardiff, 92.9% of tests in Liverpool and 93.9% of tests in Sheffield finishing successfully. These numbers were on par or better than competitors, though in Liverpool EE provided the least reliable network (by a small margin: it was 3.7% behind the most reliable network, O2). Indoor vs. outdoor: There are variations between the three cities, and while we do see a bit of a drop-off in speed, it is not a rule nor does it clearly place indoor speeds outside EE's promised thresholds. For instance, in Liverpool, indoor download speeds were at parity with outdoor speeds (and within EE's 8-12 Mbps download speed range). In Cardiff, indoor download speeds were 30.4% slower than outdoor speeds - but still statistically overlap with the low end of EE's claimed 4G speeds. In Sheffield, indoor download speeds were 20.8% slower than outdoor speeds and fell below EE's promised threshold (we found a 6.5 Mbps average indoor download speed). Competitors have implied that EE's indoor service will be poor, but we do not see broad evidence or a pattern of this yet (statistical analysis is being undertaken to understand if this is more or less than typical in other markets and will be available shortly). Call service: While the 4G conversation is focused on mobile internet, call success rates (signal) remain important for consumers. EE delivered superior results in this regard, besting competitors in Cardiff with very low blocked and dropped calls, tying for best in Sheffield and drawing with a number of others in Liverpool. Interestingly, EE's results often stood out from those of the T-Mobile and Orange networks, suggesting something about EE's network engineering that is providing fast speed and better call service. Comment from RootMetrics Bill Moore, CEO and President at RootMetrics said: "The launch of 4G in the UK will bring a massive change, and while the mobile industry has been wrapped up in claims and counter-claims, consumers have been left wondering whether 4G is really all that exciting. What they should find on a good 4G network is a mobile internet experience like never before, with activities that were previously choppy and frustrating - like downloading movies and HD videos or uploading large photos to social media - suddenly become part of the everyday. For it's part, EE's 4G network has lived up to its own promises: speeds are meeting EE's pledges and indoor performance is reasonably good. At the same time, the technology is more powerful than the bar that has been set, and there should be room for further improvement." "When it comes to availability of EE 4G in city centres, the picture is positive if not universally excellent. We see the percentage of 4G improving in line with other launches, but there are some notable variations like people in Cardiff getting a better service than those in Sheffield, despite paying the same. And in nearby suburbs, the picture is mixed as to whether 4G is available yet. Anyone wanting to upgrade to 4G should first check how good their coverage would be before they buy - we offer an independent source for this information on the RootMetrics website." Detailed results for each city Below is the mobile internet section from each of the studies that will be released. The link and password that follow the city name are provided should you wish to access the Call and Text sections of the report. Cardiff (report link: http://bit.ly/RMCardiff2013; password: rmtuk2) Mobile internet speeds across the entire Cardiff area EE's 4G-enabled network delivered the fastest average download speed in Cardiff at 6.8 Mbps, as well as the fastest average upload speed at 4.0 Mbps. o EE's 4G download speed was nearly twice as fast as that of 2nd place finisher Three and more than twice as fast as those of the other networks. Three's 3G network delivered 2nd place average download and upload speeds of 3.5 Mbps and 1.8 Mbps, respectively. Vodafone placed last in both average speed measures. Consistency of fast mobile internet speeds We're firm believers that sheer speed alone isn't the best indication of mobile internet performance. A more practical measure is how often users actually experience fast speeds. We've found that mobile performance improves notably at speeds above 5 Mbps. o EE and Three recorded download speeds above 5 Mbps more often than the other operators, doing so in 40.2% and 26.2% of our tests, respectively. o EE was the only operator to record upload speeds above 5 Mbps, doing so in 31.3% of our tests. Potential slowdowns We've found that mobile internet speeds below 1.5 Mbps often result in poor experiences when downloading or uploading files. o Vodafone recorded the greatest volume of download and upload speeds below 1.5 Mbps during testing, doing so 57.7% and 80.5% of the time, respectively. Network reliability Three provided the most reliable network in Cardiff, with an overall connection success rate of 95.9%. EE's 4G network The EE speeds included in this report are the average speeds we found across the entire area we tested. However, at the time of our testing, EE had not launched 4G service in many towns within the area, such as Pontypridd. o In our tests, we found that 4G was widely available within the city of Cardiff. We connected to the 4G network in 70.4% of our upload tests and 71.3% of our download tests. The remainder of the time, our test phones connected to EE's 3G network. o Within the city, EE recorded an average download speed of 11.8 Mbps and an average upload speed of 7.0 Mbps - these are the fastest average speeds we've found in the UK to date. o Outside the city, the availability of 4G dropped to 35.1% during our upload tests and 33.6% during our download tests, indicating a lack of 4G roll-out in these areas at the time of testing. Indeed, we found 4G service outside the city only in the nearby towns of the Vale of Glamorgan. o EE's network was reliable, with successful connections made 94.0% of the time. In terms of pure 4G (counting tests conducted only on 4G), EE had an average download speed of 16.9 Mbps and their average upload speed clocked in at 9.1 Mbps, speeds well over four times those of the average download and upload speeds we recorded on the fastest 3G network. The speeds recorded during indoor testing were somewhat slower than those found outdoors: o EE's average indoor download speed hit 7.8 Mbps compared to 11.2 Mbps outdoors. o Likewise, their average indoor upload speed hit 3.9 Mbps compared to 6.3 Mbps outdoors. Among the 3G networks Three's average download and upload speeds of 3.5 Mbps and 1.8 Mbps, respectively, placed second overall, but were the fastest speeds among the 3G networks. No other 3G network recorded an average download speed above 3.0 Mbps. Vodafone delivered last place average download and upload speeds of 1.8 Mbps and 0.9 Mbps, respectively. o At these speeds, it could take someone more than four times as long to upload the same photo to Facebook on Vodafone compared to EE 4G. Of the surrounding areas we tested, we found the slowest speeds in the borough of Caerphilly, where we recorded an average download speed of 2.0 Mbps across all operators. o Three provided the fastest average download speed in Caerphilly at 3.8 Mbps. In contrast, Vodafone's average download speed of 0.7 Mbps was more than three times slower than Vodafone's speed in Cardiff itself. Liverpool (report link: http://bit.ly/RMLiverpool2013; password: rmtuk1) Mobile internet speeds across the entire Liverpool area EE's 4G-enabled network delivered the fastest average download speed in Liverpool at 7.2 Mbps, as well as the fastest average upload speed at 4.6 Mbps. o At 7.2 Mbps, EE's average download speed was more than twice as fast as that of any other operator. o In addition, EE's average upload speed was faster than the average download speed recorded by any other operator. o This is noteworthy because we've generally found that average download speeds are faster than average upload speeds. Vodafone's 3G network placed 2nd, with an average download speed of 3.7 Mbps. O2 and Three both delivered a 3rd place average download speed of 3.3 Mbps. Orange and T-Mobile shared last place in our average download speed measure. Consistency of fast mobile internet speeds We're firm believers that sheer speed alone isn't the best indication of mobile internet performance. A more practical measure is how often users actually experience fast speeds. We've found that mobile performance improves notably at speeds above 5.0 Mbps. o EE recorded download and upload speeds above 5 Mbps more often than the other operators, doing so in 50.7% and 36.8% of our tests, respectively. o Moreover, EE was the only operator to record upload speeds above 5 Mbps. Potential slowdowns We've found that mobile internet speeds below 1.5 Mbps often result in markedly poor experiences when downloading or uploading files. o Orange, T-Mobile, and Three recorded download speeds below 1.5 Mbps more often than the other operators, as each did so at least 33.9% of the time. o Vodafone recorded the greatest percentage of upload speeds below 1.5 Mbps, doing so in 68.3% of our tests. Network reliability While all operators provided strong network connection success rates above 92.9%, O2 registered the strongest overall network connection success rate at 96.6%. EE's 4G network The EE speeds included in this report are the average speeds we found across the entire area we tested. However, at the time of our testing, EE had not launched 4G service in many towns within the area, such as St Helens and Southport. When we were unable to connect to 4G, our test phones connected to EE's 3G network. o We found 4G widely available within the city of Liverpool and the nearby towns of Birkenhead, Wallasey, Kirkby, and Bootle. Our testing showed that speeds within these areas exceeded those we found across the rest of Merseyside. For instance, within the city of Liverpool, EE's average download speed reached 9.4 Mbps and their average upload speed was 6.2 Mbps. o Within the city of Liverpool, we connected to the 4G network in 65.1% of our upload tests and 58.7% of our download tests. o Outside the city, meanwhile, these rates dropped: we connected to the 4G network in 45.8% of our download tests and 43.1% of our upload tests. In terms of pure 4G (counting tests conducted only on 4G), EE had an average download speed of 12.7 Mbps and an average upload speed of 9.0 Mbps. The speeds we recorded indoors and outdoors were similar (and statistically equal): o EE's average download speed indoors was 8.5 Mbps compared to 8.7 Mbps outdoors. o EE's average upload speed indoors was 4.9 Mbps compared to 5.8 Mbps outdoors. Among the 3G networks Vodafone delivered a 2nd place average download speed of 3.7 Mbps and a 3rd place average upload speed of 1.4 Mbps. EE's legacy 3G networks, Orange and T-Mobile, registered the slowest average download speeds. Orange also had the slowest average upload speed. o It could take nearly 3.5 times as long to upload a photo to Facebook from EE's legacy Orange network compared with its new 4G network (assuming 4G was available). St Helens had the slowest average speeds in Merseyside across all networks, with an average download speed of 1.9 Mbps and an average upload speed of 1.2 Mbps. o The slower average speeds we found here were likely due to the lack of 4G service in the area at the time of our testing. o Vodafone had the fastest average download speed in St Helens at 2.7 Mbps, but even at this speed, downloading a file in St. Helens could take nearly twice as long as it would in Knowsley or Liverpool. Sheffield (report link: http://bit.ly/RMSheffield2013; password: rmtuk3) Mobile internet speeds across the entire Sheffield area EE's 4G-enabled network delivered the fastest average download speed at 5.8 Mbps, as well as the fastest average upload speed at 3.8 Mbps. Three registered second place average download and upload speeds of 3.0 Mbps and 1.8 Mbps, respectively. Orange delivered a last place average download speed of 2.1 Mbps. In terms of upload speed, Vodafone placed last, with an average upload speed of 0.9 Mbps. Consistency of fast mobile internet speeds We're firm believers that sheer speed alone isn't the best indication of mobile internet performance. A more practical measure is how often users actually experience fast speeds. We've found that mobile performance improves notably at speeds above 5.0 Mbps. o EE recorded the greatest volume of download and upload speeds above 5 Mbps during testing, doing so 38.0% and 27.2% of the time, respectively. o Moreover, EE was the only operator to record upload speeds above 5 Mbps. Potential slowdowns We've found that mobile internet speeds below 1.5 Mbps often result in markedly poor experiences when downloading or uploading files. o Vodafone recorded the greatest volume of download and upload speeds below 1.5 Mbps during testing, doing so 45.6% and 77.2% of the time, respectively. Network reliability Three provided the most reliable network in Sheffield, recording an overall connection success rate of 96.0%. In contrast, Vodafone placed last with a rate of 88.0%. EE's 4G network The EE 4G speeds included in this report are the average speeds we found across the entire area we tested. However, at the time of our testing, EE had not launched 4G service in many towns within the area, such as Barnsley and Chapeltown. o Within the city of Sheffield, we connected to the 4G network in 48.5% of our download tests and 51.4% of our upload tests. The remainder of the time, our test phones connected to EE's 3G network. o Within the city, EE's average download speed clocked in at 7.9 Mbps and their average upload speed was 5.2 Mbps. o Outside the city, the availability of 4G dropped to 30.4% during our download tests and 32.1% during our upload tests, suggesting a lack of 4G roll-out in these areas at the time of testing. Indeed, we did not find evidence of 4G beyond the nearby villages of Oughtibridge and Dronfield. o The network was reliable across the entire Sheffield area, with successful connections made 93.9% of the time. In terms of pure 4G (counting tests conducted only on 4G), EE had an average download speed of 12.9 Mbps and an average upload speed of 9.1 Mbps. o At 12.9 Mbps, EE's average download speed was over four times faster than that of the fastest 3G network across the area. Indoor speeds were slower than outdoor speeds, but similar to the overall averages we found: o EE's average download speed indoors was 6.5 Mbps compared to 8.2 Mbps outdoors. o EE's average upload speed indoors was 3.7 Mbps compared to 5.4 Mbps outdoors. Among the 3G networks Three's average download and upload speeds of 3.0 Mbps and 1.8 Mbps, respectively, placed 2nd overall, but were the fastest among the 3G networks. O2 trailed closely behind Three, with average download and upload speeds of 2.9 Mbps of 1.3 Mbps, respectively. o At these speeds, it could take nearly twice as long to download a file on O2's network compared to EE's 4G network, and almost three times as long to upload a photo to Facebook. Outside the city, speeds were similar across the various districts we measured. o Three stood out in the High Peak district, with an average download speed of 4.3 Mbps and an average upload speed of 2.2 Mbps. o At the other end of the spectrum, the Barnsley area was the slowest district we measured, with average download speeds of 2.0 Mbps and upload speeds of 1.2 Mbps (across all networks). o Within Barnsley, Vodafone provided the slowest speeds, with an average download speed of 1.2 Mbps and an average upload speed of only 0.4 Mbps. Methodology This data in this report incorporates findings from The RootMetrics RootScore Reports for Cardiff, Liverpool and Sheffield. These reports provide an assessment of mobile performance that reflects consumer experience in a specific geographical area. To evaluate the three areas (as defined by the Eurostat's Large Urban Zone - LUZ) we performed 95,762 (Liverpool: 29,667, Sheffield: 34,458, Cardiff: 31,637) call, mobile internet, and text tests, covering all hours of the day and night. Tests were conducted using our RootScout app running from off-the-shelf, Android-based smartphones purchased from mobile phone company stores. The phones were used as a typical consumer would use them and were not modified with any external antennas or other non-standard equipment. The data provided in this report reflect our findings in the Cardiff market during testing conducted from 26-30 November 2012, The Liverpool market during testing conducted from 2-6 December 2012, and the Sheffield market during testing conducted from 2-6 December 2012 To ensure that testing aligns with the latest consumer experience and to provide each operator with the chance to rate as highly as possible, RootMetrics utilises the most advanced off-the-shelf smartphones available to the public at the time of our testing. Currently, we utilise the Samsung Galaxy S III LTE for EE 4G and the HTC One X for the 3G networks of O2, Orange, Three, TMobile, and Vodafone. The RootMetrics testing methodology centres on activities that consumers perform most often: making phone calls, uploading and downloading files from the internet, and sending and receiving text messages (SMS). To reflect how consumers use their phones on a daily basis, we conducted data collection tests in cities and towns within the LUZ, testing both indoors (at 127 locations - 40 in Cardiff; 40 in Liverpool; 47 in Sheffield), outdoors (conducted in the scouter's parked vehicle during indoor testing), and during drive tests (covering 2,265 miles -726 miles in Cardiff; 709 miles in Liverpool; 830 miles in Sheffield). To prevent bias, RootMetrics utilises a sampling methodology that randomly selects the indoor locations used for testing; drive testing takes place during travel between these random indoor locations. To measure call performance, RootMetrics places a call from each operator's phone and, if it is successful, attempts to hold that call open for two minutes. Both blocked and dropped call rates are determined from this test. To analyse texting, RootMetrics measures the amount of time it takes each operator to send a text to or receive a text from: 1) a phone within its own network; and 2) phones within the other operator's networks. During the text test, each phone in a kit sends an SMS message to a randomly selected phone in another kit. During mobile internet performance tests, RootMetrics opens an HTTP connection, then measures the mobile internet transfer rate to determine average and maximum speeds. Because we remain committed to evaluating service from the perspective of consumers, we analyse goodput rather than throughput during our tests. Throughput includes the protocol information that accompanies mobile internet packets, whereas goodput measures only the 'good' stuff -- the actual data that you are interested in. For an analogy, think of goodput and throughput in terms of sending/receiving a package: you are interested in the item inside the package (the goodput), but it is delivered inside a box that includes packing materials (throughput includes this extra 'overhead'). RootScores are calculated using a proprietary algorithm factoring in call, mobile internet, and text results. Within each category, based on various factors reflecting performance, a unique RootScore is calculated for each operator. Our RootScore Awards system incorporates a standardised scoring system for comparison across all markets and calculates operators' mobile internet speed scores relative to a standardised range. If an operator's mobile internet speed is on the slow side of this range, it will score low. If, on the other hand, an operator actually exceeds the fast end of the range, it can achieve a score over 100 for that market. Think of this as a bonus: the operator is doing more than expected. Because operators are compared directly against this standardised range no matter what market we are testing, you can easily see how performance in one city or town compares with performance in another city or town. Operator rankings are determined through direct statistical comparison of performance within each market. Calculations for RootScores have variability associated with them. RootMetrics analyses the interaction between score and variability to determine when RootScores statistically differ between operators. We then use a standard competition ranking algorithm to assign each operator a final rank. Through this process, operators with divergent scores are at times determined to be statistically indistinguishable and therefore receive the same rank; conversely, operators with very close scores can receive different final rankings depending on how they compare with other operators in the market. Any use of the information contained in this report must be accompanied by a statement identifying RootMetrics as the source. Results are based on testing at dates noted above. Mobile network performance is subject to a number of factors which may vary over time. Individual experience may vary. RootScores are derived using data sampling and analysis methods that are subject to statistical variation. No advertising or other promotional or commercial use can be made of the information contained in this report without the express prior consent of RootMetrics. Further information For further information, please contact Patrick Southwell at the Red Consultancy: T: 0207 025 6507 M: 0781 315 1994 E: [email protected] ((M2 Communications disclaims all liability for information provided within M2 PressWIRE. Data supplied by named party/parties. Further information on M2 PressWIRE can be obtained at http://www.presswire.net on the world wide web. Inquiries to [email protected])). (c) 2013 M2 COMMUNICATIONS |