The P3 Mobile
Benchmark UK
EE emerged the clear winner and took the best in test accolade
for 2014. Three ranked second overall and had the best voice
performance. O2 and Vodafone followed almost level-pegging
with a slightly better score for O2.
The first ever P3 Mobile
Benchmark UK ended in a
clear ranking of the countrys
operators. The best in
test seal was awarded to
EE, thanks to the best data
performance and a good
voice service.
Three ranked second with
the best voice performance
in the test, while O2 and
Vodafone followed almost
head-to-head.
The P3 Mobile Benchmark
used high-tech measurement equipment to test all
four UK mobile networks in
big cities, smaller cities and
on connecting routes. The
test cases were designed to
push the networks as hard as
possible, and were all based
on proven P3 measurement
methodologies.
The results show, the UK
networks provide much
better services in larger
metropolitan areas than
in smaller cities and
on
connecting
routes.
Compared
with
other
Mobile
11/201
Benchm
ark UK
Introduction
The UK is one of the largest and most interesting mobile communications
markets in Europe since it is moving very quickly at the forefront of trialling
and deploying some of the newest LTE technologies.
Thus, the UK is a natural area of
focus for P3 communications, the
international leader in mobile
benchmarking. The P3 Mobile
Benchmark UK is designed to
provide an independent overview of
the current network situation in the
UK; it highlights the leading-edge
innovations in the countrys mobile
networks, but also reveals potential
weak spots in mobile service quality
as perceived by customers.
Background
The network operators are in the
process of deploying 4G technology,
also known as LTE, across large
portions of their networks; LTE-
Timeline 2014
Jun
Start of scoping
and planning
work.
Jul
EE, O2, Three
and Vodafone
were notified
of the planned
benchmark by
P3, including
preliminary
information
around scope,
and were invited
to provide feedback.
Sep
P3 finally defined all test
parameters.
The decision was
made to publish
the benchmark
results in
November 2014.
Oct
Testing began.
All operators
were notified
and, in turn,
were asked to
provide notice
of any outages
which might
impact results.
Nov
Testing concluded. P3 began
post-processing
on the results.
The P3 Mobile
Benchmark UK
was published.
About P3 communications
Methodology
The basis for the benchmark was
the proven P3 measurement
methodology, used by the company
for similar testing around the
world. The methodology ensures
fair comparability of results between all operators under tests, and
yields reliable outcomes.
Smartphones
Measurement System
Voice Measurements
Data Measurements
Wikipedia
LinkedIn
Wordpress
The Telegraph
Gumtree
data
Srling
Glasgow
Paisley
East Kilbride
Edinburgh
Big Cities
Smaller Cities
Penrith
Preston Burnley
Leeds
Oldham
Manchester
Waypoints on Testing Route
London
Marlow
Witney
Cirencester
Stroud
Worcester
Droitwich Spa
Bromsgrove
Birmingham
Redditch
Evesham
Stratford-upon-Avon
Royal Leamington Spa
Nuneaton
Tamworth
Stafford
Congleton
Macclesfield
Wilmslow
Manchester
Chorley
Preston
Burnley
Bury
Oldham
Leeds
Harrogate
Penrith
Edinburgh
Dunfemline
Stirling
Falkirk
Livingston
Glasgow
Paisley
East Kilbride
Stafford
Birmingham
Evesham
Stroud
Marlow
London
Results
The results of the P3 Mobile
Benchmark UK are represented by
means of a scoring, that is used
to map the technical performance
metrics to a common scale and to
aggregate the achievements in the
different test categories to one final
count.
Operator
Voice Total
max.
190
140
153
84
94
Big Cities
125
73%
75%
32%
42%
Small Cities/Transfers
65
76%
91%
68%
65%
In cities EE and Three were on par in terms of voice scoring. While EE provided a slightly better call success ratio,
Three impressed with very short call set up times and the best speech quality.
The Vodafone network delivered good speech quality but could improve on call success ratio and call set up
time. The challenges for O2 were call stability and call set up times as well as speech quality, which suffered from
missing HD voice features. All networks performed well in terms of call sustainability.
Operator
EE
Three
O2
Vodafone
97.1
96.8
91.1
92.4
7.9
8.1
8.2
3.3
5.5
3.6
2.6
3.0
99.0
98.1
99.0
98.6
Three provided the best voice service in the tested smaller cities and on the transfer routes, leading in all Key
Performance Indicators (KPIs). Test winner EE provided a stable voice network, followed by O2 and Vodafone.
Call success ratio on transfer routes was an area of interest for almost all operators but Three, while call sustainability was managed well.
Operator
EE
Three
O2
Vodafone
97.4
4.5
3.6
99.5
93.5
90.9
8.0
7.3
2.5
3.0
98.5
99.0
94.3
7.0
3.3
98.7
Results: Data
The data score was separated into
three major sections. The first
section contained the results of
the metropolitan areas (big cities),
while the other sections included
separately smaller cities (Small
Cities) and connecting routes
(Transfers). The percentages in the
Operator
226
163
199
186
200
81%
50%
74%
72%
Web Browsing
70.0
79%
75%
90%
88%
File Download
40.0
71%
32%
58%
60%
File Upload
40.0
83%
47%
64%
66%
YouTube
50.0
90%
31%
72%
63%
Small Cities
40
45%
45%
45%
32%
Web Browsing
14.0
47%
67%
67%
47%
File Download
8.0
39%
33%
30%
23%
File Upload
8.0
49%
37%
39%
20%
YouTube
10.0
43%
29%
30%
28%
Transfers
70
66%
65%
47%
41%
Web Browsing
24.5
70%
78%
65%
54%
File Download
14.0
70%
62%
27%
28%
File Upload
14.0
82%
68%
56%
50%
YouTube
17.5
45%
45%
29%
28%
Data Total
Big Cities
max.
310
All networks showed good data stability in most of the tested services, with different strengths and weaknesses.
Vodafone and O2 delivered the best browsing results, while EE showed the best YouTube performance with
impressive success ratios.
EE achieved the highest data speeds in upload and download, followed by Vodafone, O2 and Three.
Operator
EE
Three
O2
Vodafone
Success Ratio (% / %)
97.0 / 99.0
97.5 / 98.8
99.0 / 99.3
4.1 / 1.1
4.5 / 2.7
99.0 / 99.7
3.5 / 1.2
99.7 / 2.9
5271
98.8 / 9.9
99.4 / 3.9
99.3 / 3.4
1160
3640
4478
99.9 / 1.6
3134
99.2 / 5.2
99.7 / 2.2
99.7 / 2.6
686
2228
2225
17361
6131
6355
13232
13973
1095
3662
3925
21032
3088
6577
10343
11257
625
1316
1826
99.9 / 1.6
99.6
95.7 / 2.2
98.8 / 1.3
99.5 / 1.7
93.9
97.6
99.3
98.8 / 1.4
98.0
79.0 / 3.7
96.7 / 2.0
93.5 / 2.0
68.1
92.7
90.7
3.8 / 1.4
10
The results in the tested small cities showed where all operators could find a lot of potential to improve their data
services. Smaller cities suffered from lower success ratios, longer session times and slower data speeds.
In small cities EE, Three and O2 performed on-par, followed by Vodafone.
Operator
EE
Three
O2
Vodafone
Success Ratio (% / %)
92.3 / 97.1
96.9 / 98.5
96.6 / 97.7
94.7 / 88.5
5.5 / 3.3
5.3 / 3.8
5.1 / 3.4
5.5 / 4.3
98.5 / 9.0
99.3 / 10.2
98.0 / 13.5
95.5 / 14.3
1206
1142
965
682
99.3 / 7.5
99.3 / 8.6
98.9 / 7.2
94.8 / 11.8
497
563
576
290
12158
6823
8355
8730
1226
1326
924
577
7585
2492
4530
4658
486
511
349
141
97.8 / 2.4
95.5
96.6 / 2.4
97.0 / 2.2
97.1 / 2.6
94.1
92.7
94.7
90.8 / 2.6
85.1
85.6 / 4.1
78.4 / 4.1
78.0 / 4.4
77.0
70.0
71.5
Youtube SD
Success Ratio / Start Time (% / s)
Video playouts without interruptions (%)
Youtube HD
Success Ratio / Start Time (% / s)
Video playouts without interruptions (%)
11
The results on transfer routes showed even more improvement potential than those in smaller cities. Biggest
challenges were the success ratios und the very slow throughputs. All services were suffering from those
performances.
EE and Three achieved the best scores followed by O2 and Vodafone.
Operator
EE
Three
O2
Vodafone
Success Ratio (% / %)
90.9 / 95.2
94.9 / 97.9
90.8 / 85.8
87.8 / 81.4
5.7 / 3.0
5.5 / 4.1
5.1 / 3.6
5.5 / 4.3
99.3 / 8.1
97.9 / 9.6
84.6 / 17.9
83.3 / 16.0
1250
1299
147
138
98.8 / 7.6
471
98.7 / 9.1
89.5 / 11.2
89.0 / 16.5
450
292
179
10379
1397
7560
7059
8046
1260
69
36
8699
2103
4622
4615
370
489
45
20
97.7 / 2.5
94.3 / 2.2
94.6 / 3.2
95.9
98.2 / 2.3
97.5
91.8
91.1
83.8 / 3.3
84.8 / 3.8
69.5 / 4.7
76.5 / 4.3
78.2
77.5
60.6
67.8
Youtube SD
Success Ratio / Start Time (% / s)
Video playouts without interruptions (%)
Youtube HD
Success Ratio / Start Time (% / s)
Video playouts without interruptions (%)
12
Results London
The major metropolitan area of the UK is worth having a detailed view at in terms of network performances.
If the London area were to be scored separately, with the same voice and data scoring used for big cities, the
results would be as follows:
Operator
TOTAL
max.
325
255
187
185
200
Telephony
max.
125
88
88
42
57
71%
70%
33%
46%
167
99
143
143
83%
50%
71%
72%
London ONLY
Data Services
London ONLY
125
max.
200
200
It is very interesting to see Vodafone, which performed in London much better in voice. EE achieved a better data
score, while O2 and Three delivered voice and data scores in London at the same level as in the total Big City
category.
13
366
316
280
VOICE
VOICE
VOICE
DATA
DATA
DATA
DATA
283
VOICE
Overall
Scores
MAX.
500
11/2014
Mobile Benchmark UK
Operator
TOTAL
max.
500
366
316
283
280
Telephony
max.
190
140
153
84
94
Big Cities
125
73%
75%
32%
42%
Small Cities/Transfers
65
76%
91%
68%
65%
226
163
199
186
Data Services
max.
310
Big Cities
200
81%
50%
74%
72%
Small Cities
40
45%
45%
45%
32%
Transfers
70
66%
65%
47%
41%
14
Conclusions .
& Highlights
The P3 Mobile Benchmark was specifically designed to push the limits of the most advanced mobile network
technology currently in service.
It revealed some very well performing services and some very
well served areas, but also plenty of
potential for improvement in all UK
networks.
11/2014
Mobile Benchmark UK
Three finished in second place, with the best voice performance in the test. Had it not trailed
the field in data, though, it could have put a lot more pressure on EE. Once its data challenges are solved, we can expect much more from Three.
O2 delivered well in data with the second best result, but has a lot of scope to improve its
scores in voice. It was not far behind EE on data, however, with some improvements to success rates and throughput, the data story next year could be different. Higher voice success
rates and better speech quality could further strengthen O2s position.
Vodafone ended up in third place for both voice and data. But O2 only pulled ahead in the
total scoring via a slight advantage in data performance; it is a small gap to close for Vodafone to draw level with O2, and a little more ground to cover to threaten Three. With good
data results and an increased stability in voice, Vodafone could catch up in next years P3
Mobile Benchmark UK.
15
International
Benchmark
While the best UK operator achieved
a score of 366, the top performers
in Germany, Austria and Switzerland
scored in a range above 440; and
even a score above 470 was achie-
ved in the last years. The lowest values in those countries were above
325, which would still be enough for
the second place in UK.
In other words, compared with
The Future
The P3 Mobile Benchmark 2014 is intended to be the first in a regular
series of such reports.
The mobile industry in the UK moves very quickly and is at the forefront
of trialling and deploying some of the newest LTE technologies, such as VoLTE and LTEAdvanced. Right now, each of the operators has a number of major cutting-edge initiatives
under development to boost their networks in different ways.
We expect to see significant improvements by the time the next testing cycle begins.
Headquarters:
Aachen (GER), Belgrade (SRB), Morristown (NJ, USA), Sydney (AUS)
All rights reserved. The information and statements provided in this document are for informational purposes only. Any measurements, recommendations, opinions or findings
provided are only true and accurate for the conditions encountered at the time of any survey and P3 communications accepts no liability for any errors or omissions. In no event will
P3 communications be liable for any consequential, indirect or incidental damages whatsoever, including lost profits, business, contracts, revenue or anticipated savings arising out of
the use of any P3 communications products or documentation (or from errors or deficiencies therein), reliance on the data or measurements produced or information contained within.
16