Anda di halaman 1dari 25

Network Quality & Performance Division

Monitoring the End User Quality of Experience using the derivative Mean Opinion Score (MOS) KPI

Paper Authors: Konstantinos Vlahodimitropoulos PhD, Evangelos Katsaros

Presentation: Filippos Kyriazidis

Effect of transmission performance on Multimedia Quality of Service

17 - 19 June 2008 - Prague, Czech Republic

Network Quality & Performance Division

Objectives:
Indicate the importance of MOS to the end user perceived speech quality. Introduce the MOS derivative Key Performance Indicator (KPI) which comes from Drive Tests Data and Network Statistics. Implement a mathematical formalism in order systemize the MOS calculation procedure. Compare different speech codecs implementations in BSC areas using MOS distribution patterns.
Effect of transmission performance on Multimedia Quality of Service 17 - 19 June 2008 - Prague, Czech Republic

Network Quality & Performance Division

Contents:
MOS concept & dependencies. Average MOS calculation in Cell & BSC level combining drive tests data & OSS measurements. Cell MOS distribution patterns for different speech codecs. EFR vs AMR codec comparison. Effect of MS technology on different speech codecs share. Evaluating speech codec effectiveness using quality and speech codec distribution patterns
Effect of transmission performance on Multimedia Quality of Service 17 - 19 June 2008 - Prague, Czech Republic

Network Quality & Performance Division

Mean Opinion Score (MOS) concept description: A common benchmark used to determine sound quality. In voice telephony, represents a measure of human speech quality, at the destination end of the circuit. It is an important factor in determining the QoS. MOS Measurement: 1. A pool of listeners rate a series of audio files using a five-grade scale ranging from 1 to 5. 2. The average or Mean Opinion Score (MOS) for each audio file is calculated. Conditions: 16 or more listeners (adequate sample). Quiet environment.

To reduce variability, tests commonly include reference files that have "industry accepted" MOS scores.

Effect of transmission performance on Multimedia Quality of Service

17 - 19 June 2008 - Prague, Czech Republic

Network Quality & Performance Division

MOS dependencies in a mobile network:

Speech Codec

Radio Interface Conditions

Full Rate (FR) Half Rate (HR) Enhanced Full Rate (EFR) Adaptive Multi Rate (AMR)

Signal Quality i.e.: BER, FER

MOS classification according to ASCOM scale


0 Bad 1.5 2.1 2.7 Fair 3.5 Good Excellent
17 - 19 June 2008 - Prague, Czech Republic

Poor

Effect of transmission performance on Multimedia Quality of Service

Network Quality & Performance Division

MOS concept & dependencies. Average MOS calculation in Cell & BSC level combining drive tests data & OSS measurements. Cell MOS distribution patterns for different speech codecs. EFR vs AMR codec comparison. Effect of MS technology on different speech codecs share. Evaluating speech codec effectiveness using quality and speech codec distribution patterns
Effect of transmission performance on Multimedia Quality of Service 17 - 19 June 2008 - Prague, Czech Republic

Network Quality & Performance Division

Basic principle of extracting a MOS derivative KPI:


Perform Drive Tests Query OSS

Process results & create MOS Quality & Speech Codec Matrix

Export Speech Codec usage Matrix Export Quality class statistics Matrix

Data Merge

Perform Matrix Operations to calculate Cell/BSC MOS Effect of transmission performance on Multimedia Quality of Service value

17 - 19 June 2008 - Prague, Czech Republic

Network Quality & Performance Division

Drive Tests Reference Table:


Quality degradation direction (8 quality classes)
Speech Codec / DL Quality class AMR FR MOS AMR HR MOS EFR MOS FR MOS HR MOS 0 1 2 3 4 5 6 7

3,71 3,36 3,76 3,19 2,92

3,56 3,24 3,68 3,14 2,83

3,31 3,04 3,43 3,03 2,68

3,05 2,88 3,08 2,77 2,59

2,72 2,71 2,72 2,58 2,43

2,49 2,47 2,34 2,22 2,13

2,12 2,11 2,01 1,82 1,66

1,70 1,40 1,44 1,45 1,41

Speech Codec degradation direction (5 speech codecs)

OSS Derived Tables Quality Class Share of BSC Speech Share of BSC Table
Speech Codec Share AMR FR AMR HR EFR FR HR
DL Quality QUAL_0 QUAL_1 QUAL_2 QUAL_3 QUAL_4 QUAL_5 QUAL_6 QUAL_7 Samples Share

33.95% 13.96% 39.46% 0.39% 12.23%

80.17% 3.73% 3.68% 3.67% 3.22% 2.72% 1.89% 0.91%

Effect of transmission performance on Multimedia Quality of Service

17 - 19 June 2008 - Prague, Czech Republic

Network Quality & Performance Division

BSC MOS calculation example:


Spech Codec / DL Quality class AMR FR MOS AMR HR MOS EFR MOS FR MOS HR MOS 0 1 2 3 4 5 6 7

3.71 3.36 3.76 3.19 2.92

3.56 3.24 3.68 3.14 2.83

3.31 3.04 3.43 3.03 2.68

3.05 2.88 3.08 2.77 2.59

2.72 2.71 2.72 2.58 2.43

2.49 2.47 2.34 2.22 2.13

2.12 2.11 2.01 1.82 1.66

1.70 1.40 1.44 1.45 1.41

Dimension: 5X8

DL Quality QUAL_0 QUAL_1 QUAL_2 QUAL_3 QUAL_4 QUAL_5 QUAL_6 QUAL_7

Samples Share

80.17% 3.73% 3.68% 3.67% 3.22% 2.72% 1.89% 0.91%

Speech Codec Weighted MOS Value 3.55 AMR FR MOS AMR HR MOS 3.24 EFR MOS 3.59 FR MOS 3.08 HR MOS 2.82

Dimension: 5X1

Dimension: 8X1

BSC MOS =

Speech Codec Weighted MOS Value AMR FR MOS 3.55 AMR HR MOS 3.24 EFR MOS 3.59 FR MOS 3.08 HR MOS 2.82

Speech Codec Share AMR FR AMR HR EFR FR

HR

33.95% 13.96% 39.46% 0.39% 12.23%

= 3.43

Dimension: 1X5

Dimension: 5X1

Effect of transmission performance on Multimedia Quality of Service

17 - 19 June 2008 - Prague, Czech Republic

Network Quality & Performance Division

Generalized calculation procedure:


The (m x n) (m codecs, n quality classes) matrix is multiplied with the (n x1) quality classes matrix as in (1):
M 00 ... ... M m0 ... ... ... ... ... ... ... ... Q0 SC0 M 0n Q1 SC1 ... ... = ... ... ... ... M mn Qm SCm

(1)

The product of (1), (m x 1) is further multiplied with the (1 x m) speech codecs matrix to obtain a single average MOS value, as in (2):
SC 0 ... ... [C 0 ... SC m

... ... ... C m ] = Average _ MOS

(2)

Effect of transmission performance on Multimedia Quality of Service

17 - 19 June 2008 - Prague, Czech Republic

Network Quality & Performance Division

Key issues:
Process can be easily implemented in a performance monitoring system. Can be applied in Cell, BSC or any other network structure. Cells with Poor or Bad MOS can be isolated and be further investigated. A single average MOS value of a cell group (i.e. BSC area) may not be essential. The comparison of different network areas provides QoS indicative results.

Effect of transmission performance on Multimedia Quality of Service

17 - 19 June 2008 - Prague, Czech Republic

Network Quality & Performance Division

MOS concept & dependencies. Average MOS calculation in Cell & BSC level combining drive tests data & OSS measurements. Cell MOS distribution patterns for different speech codecs. EFR vs AMR codec comparison. Effect of MS technology on different speech codecs share. Evaluating speech codec effectiveness using quality and speech codec distribution patterns
Effect of transmission performance on Multimedia Quality of Service 17 - 19 June 2008 - Prague, Czech Republic

Network Quality & Performance Division

MOS distribution:
Derives from cell level MOS calculation of a group of cells Gives a brief description of the end user perceived speech quality in a specific network area.

70
0 1.5 Bad Poor 2.1 2.7 Fair 3.5 Good Excellent 5

60 Number of cells 50 40 30 20 10 0
58 74 06 5 3 46 98 22

62

66

14

38

78

82

54

2,

2,

3,

2,

2,

3,

3,

2,

2,

2,

3,

3,

3,

3,

3,

3,

3,

3,

86

MOS classification
Effect of transmission performance on Multimedia Quality of Service 17 - 19 June 2008 - Prague, Czech Republic

3,

94

Network Quality & Performance Division

MOS improvement pattern:


The distance between distributions shows the MOS improvement between two different states of a group of cells.
70 FR/HR codec 60 50 Number of cells
0 Bad 1.5 2.1 2.7 Fair 3.5 Good Excellent 5 Poor

AMR codec (together with EFR & FR/HR)

40 30 20 10 0

2, 5

2, 58

2, 66

2, 74

2, 82

2, 9

2, 98

3, 06

3, 14

3, 22

3, 3

3, 38

3, 46

3, 54

3, 62

3, 94

3, 7

3, 78

3, 86

MOS classification
Effect of transmission performance on Multimedia Quality of Service 17 - 19 June 2008 - Prague, Czech Republic

>4

Network Quality & Performance Division

EFR versus AMR:


EFR achieves highest but also lowest MOS values in comparison to AMR. EFR distribution has a larger standard deviation than AMR. AMR is subsequently, a preferable speech codec.
160 EFR codec (together with FR/HR) 140 120 Number of cells 100 80 60 40 20 0
5 98 58 46 66 06 14 54 74 38 9 3 82 62 22 78 86 94 3, 7 2, 2, 3, 2, 2, 3, 3, 2, 3, 3, 3, 2, 3, 3, 2, 3, 3, 3, >4
0 Bad

AMR codec (together with EFR/FR/HR)


1.5 2.1 2.7 Fair 3.5 Good Excellent 5

Poor

MOS classification
Effect of transmission performance on Multimedia Quality of Service 17 - 19 June 2008 - Prague, Czech Republic

Network Quality & Performance Division

MOS concept & dependencies. Average MOS calculation in Cell & BSC level combining drive tests data & OSS measurements. Cell MOS distribution patterns for different speech codecs. EFR vs AMR codec comparison. Effect of MS technology on different speech codecs share. Evaluating speech codec effectiveness using quality and speech codec distribution patterns
Effect of transmission performance on Multimedia Quality of Service 17 - 19 June 2008 - Prague, Czech Republic

Network Quality & Performance Division

The role of Mobile Station technology


The following speech codecs share matrix is involved in all MOS calculations:

SC = [C0 ... .... ... Cm ]


According to the previous example:
AMRFR AMRHR EFR FR HR

SC = [33.95% 13.96% 39.46% 0.39% 12.23%]


The existence of old-technology mobiles in a network area, which do not support modern speech codecs, will affect codecs shares against EFR & AMR percentages. This will degrade average MOS of BTSs.

Promotion of latest technology MSs to existing subscribers could be a key issue of marketing strategy in order to optimize speech quality.
Effect of transmission performance on Multimedia Quality of Service 17 - 19 June 2008 - Prague, Czech Republic

Network Quality & Performance Division

MOS concept & dependencies. Average MOS calculation in Cell & BSC level combining drive tests data & OSS measurements. Cell MOS distribution patterns for different speech codecs. EFR vs AMR codec comparison. Effect of MS technology on different speech codecs share. Evaluating speech codec effectiveness using quality and speech codec distribution patterns

Effect of transmission performance on Multimedia Quality of Service

17 - 19 June 2008 - Prague, Czech Republic

Network Quality & Performance Division

Quality & Speech codec distribution patterns:


An equivalent way of deciding speech codecs effectiveness is to calculate the percentage of samples in each MOS class, i.e. create a graph with the following information:
45% 40%

35%

30% Samples share

25%

20%

15%

10%

5%

0% Excellent Good Fair Poor Bad

To achieve this aim drive tests data should be post-processed differently with regard to the original data.
Effect of transmission performance on Multimedia Quality of Service 17 - 19 June 2008 - Prague, Czech Republic

Network Quality & Performance Division

Reference Table Quality & Speech codec distribution patterns:


Post-Processed Drive Tests Data
SC MOS Class AMR FR E AMR FR G AMR FR F AMR FR P AMR FR B AMR HR E AMR HR G AMR HR F AMR HR P AMR HR B EFR E EFR G EFR F EFR P EFR B FR E FR G FR F FR P FR B HR E HR G HR F HR P HR B 0 87.79% 7.33% 4.31% 0.57% 0.00% 38.71% 53.12% 7.10% 1.08% 0.00% 91.48% 7.05% 1.25% 0.00% 0.23% 0.00% 96.14% 3.34% 0.39% 0.13% 0.16% 92.54% 5.40% 1.90% 0.00% 1 75.78% 13.66% 7.45% 3.11% 0.00% 20.81% 68.79% 7.72% 2.68% 0.00% 87.06% 9.90% 2.08% 0.80% 0.16% 0.21% 92.31% 6.44% 1.04% 0.00% 0.42% 81.89% 13.63% 3.95% 0.10% 2 55.38% 22.58% 16.94% 4.30% 0.81% 15.63% 58.75% 18.13% 6.88% 0.63% 68.26% 19.90% 8.82% 2.77% 0.25% 0.00% 84.43% 12.11% 3.11% 0.35% 1.31% 60.93% 26.24% 11.08% 0.44% 3 35.41% 31.91% 21.40% 9.73% 1.56% 10.59% 54.12% 23.53% 11.76% 0.00% 41.13% 30.65% 21.77% 5.24% 1.21% 0.00% 63.30% 25.00% 9.57% 2.13% 1.72% 49.68% 32.47% 14.41% 1.72% 4 15.51% 33.16% 32.09% 17.11% 2.14% 12.50% 35.00% 35.00% 12.50% 5.00% 15.69% 39.87% 29.41% 7.84% 7.19% 0.67% 43.33% 38.00% 16.67% 1.33% 1.32% 32.34% 39.27% 24.09% 2.97% 5 7.77% 30.10% 33.01% 24.27% 4.85% 10.81% 18.92% 29.73% 37.84% 2.70% 6.15% 18.46% 43.08% 23.85% 8.46% 0.00% 18.92% 36.94% 34.23% 9.91% 0.00% 9.83% 43.35% 37.57% 9.25% 6 0.00% 19.28% 30.12% 31.33% 19.28% 0.00% 12.50% 31.25% 43.75% 12.50% 0.00% 16.67% 21.43% 38.10% 23.81% 0.00% 4.00% 24.00% 45.33% 26.67% 0.00% 1.82% 13.64% 44.55% 40.00% 7 0.00% 0.00% 22.86% 34.29% 42.86% 0.00% 0.00% 11.11% 11.11% 77.78% 0.00% 0.00% 15.00% 15.00% 70.00% 0.00% 0.00% 10.00% 25.00% 65.00% 0.00% 0.00% 8.51% 19.15% 72.34%

Effect of transmission performance on Multimedia Quality of Service

17 - 19 June 2008 - Prague, Czech Republic

Network Quality & Performance Division

Calculation example in a BSC area with AMR /EFR/FR/HR enabled:


Calculation Procedure: Drive Tests data are weighted according to quality class share, that is the following table:
QUAL_0 QUAL_1 QUAL_2 QUAL_3 QUAL_4 QUAL_5 QUAL_6 QUAL_7 83.58% 3.12% 2.97% 2.93% 2.52% 2.23% 1.71% 0.94%

The resulting table is further transformed according to speech encoding share:


AMRFR AMRHR EFR FR HR 37.43% 8.73% 46.12% 0.44% 7.28%

Effect of transmission performance on Multimedia Quality of Service

17 - 19 June 2008 - Prague, Czech Republic

Network Quality & Performance Division

Calculation example in a BSC area with AMR /EFR/FR/HR enabled:


After all transformations (multiplications) the Drive tests table becomes:
SC MOS Class AMR FR E AMR FR G AMR FR F AMR FR P AMR FR B AMR HR E AMR HR G AMR HR F AMR HR P AMR HR B EFR E EFR G EFR F EFR P EFR B FR E FR G FR F FR P FR B HR E HR G HR F HR P HR B 0 27.213% 2.292% 1.348% 0.180% 0.000% 2.825% 3.876% 0.518% 0.078% 0.000% 35.261% 2.716% 0.482% 0.000% 0.088% 0.000% 0.354% 0.012% 0.001% 0.000% 0.010% 5.634% 0.329% 0.116% 0.000% 1 0.897% 0.159% 0.087% 0.036% 0.000% 0.057% 0.187% 0.021% 0.007% 0.000% 1.251% 0.142% 0.030% 0.011% 0.002% 0.000% 0.013% 0.001% 0.000% 0.000% 0.001% 0.186% 0.031% 0.009% 0.000% 2 0.663% 0.251% 0.188% 0.048% 0.009% 0.041% 0.152% 0.047% 0.018% 0.002% 0.936% 0.273% 0.121% 0.038% 0.003% 0.000% 0.011% 0.002% 0.000% 0.000% 0.003% 0.132% 0.057% 0.024% 0.001% 3 0.425% 0.350% 0.235% 0.107% 0.017% 0.027% 0.138% 0.060% 0.030% 0.000% 0.555% 0.414% 0.294% 0.071% 0.016% 0.000% 0.008% 0.003% 0.001% 0.000% 0.004% 0.106% 0.069% 0.031% 0.004% 4 0.160% 0.312% 0.302% 0.161% 0.020% 0.027% 0.077% 0.077% 0.027% 0.011% 0.182% 0.463% 0.342% 0.091% 0.083% 0.000% 0.005% 0.004% 0.002% 0.000% 0.002% 0.059% 0.072% 0.044% 0.005% 5 0.068% 0.251% 0.275% 0.202% 0.040% 0.021% 0.037% 0.058% 0.074% 0.005% 0.063% 0.190% 0.443% 0.245% 0.087% 0.000% 0.002% 0.004% 0.003% 0.001% 0.000% 0.016% 0.070% 0.061% 0.015% 6 0.000% 0.123% 0.193% 0.201% 0.123% 0.000% 0.019% 0.047% 0.065% 0.019% 0.000% 0.132% 0.169% 0.301% 0.188% 0.000% 0.000% 0.002% 0.003% 0.002% 0.000% 0.002% 0.017% 0.056% 0.050% 7 0.000% 0.000% 0.081% 0.121% 0.151% 0.000% 0.000% 0.009% 0.009% 0.064% 0.000% 0.000% 0.065% 0.065% 0.304% 0.000% 0.000% 0.000% 0.001% 0.003% 0.000% 0.000% 0.006% 0.013% 0.050% Total 29.427% 3.740% 2.710% 1.056% 0.361% 2.998% 4.487% 0.837% 0.309% 0.101% 38.248% 4.329% 1.945% 0.822% 0.772% 0.000% 0.392% 0.028% 0.013% 0.007% 0.020% 6.136% 0.651% 0.354% 0.125%

Effect of transmission performance on Multimedia Quality of Service

17 - 19 June 2008 - Prague, Czech Republic

Network Quality & Performance Division

Calculation example in a BSC area with AMR /EFR/FR/HR enabled:


A grouping by the MOS class yields the MOS class distribution of samples within the BSC area:
MOS Class Samples Share Excellent 70.69% Good 19.08% Fair 6.17% Poor 2.55% Bad 1.37%

80%

Samples Share
70% 60% 50% 40% 30% 20% 10% 0% Excellent Good Fair Poor Bad

Samples percentage

Effect of transmission performance on Multimedia Quality of Service

MOS Class

17 - 19 June 2008 - Prague, Czech Republic

Network Quality & Performance Division

Comparative evaluation of three different network states:


MOS Class R/EFR/FR/HR EFR/FR/HR FR/HR Excellent 70.69% 56.98% 0.11% Good 19.08% 33.65% 87.36% Fair 6.17% 5.32% 7.23% Poor 2.55% 2.47% 3.61% Bad 1.37% 1.58% 1.59%

100%

R/EFR/FR/HR
90% 80%

EFR/FR/HR

FR/HR

Samples percentage

70% 60% 50% 40% 30% 20% 10% 0% Excellent Good Fair Poor Bad

MOS Class

Effect of transmission performance on Multimedia Quality of Service

17 - 19 June 2008 - Prague, Czech Republic

Network Quality & Performance Division

Thank you for your attention

Effect of transmission performance on Multimedia Quality of Service

17 - 19 June 2008 - Prague, Czech Republic