activation - Non-regression test plan - Anca Beca NE/ Voice Quality Epert Alcatel-!ucent "omania Alcatel-Lucent File Date Edition Page B11 VQ measurement 19/3/09 1.0 1 Table of Contents 1 SCOPE.............................................................................................................................................................................. 2 NEEDED EQ!P"EN#................................................................................................................................................. 2.1 PREREQUISITES........................................................................................................................................................... 2.2 GENERAL OVERVIEW ON QVOICE............................................................................................................................... 2.3 SPEEC SA!PLE CLASSI"ICATION TRESOL#S......................................................................................................... 3 P$OPOSED #ES#S........................................................................................................................................................ % E&PEC#ED $EPO$#ED $ESL#S............................................................................................................................ Alcatel-Lucent File Date Edition Page B11 VQ measurement 19/3/09 1.0 2 1 Scope #$e scope of t$is document is to define t$e non-regression test cases% needed equipment and time sc$edule for voice quality assessment at t$e introduction of ne& soft&are release% ne& optional features or $ard&are en$ancements t$at mig$t impact t$e voice quality' (t$er tests% t$at mig$t )e required to asses t$e improvement in voice quality% do not ma*e t$e scope of t$is document' #$e tests s$all perform +S to +S and +S to ,S-N calls for E."% ."% H"% A+"-." and A+"-H"' ,f one or more of t$is codecs is not availa)le in t$e net&or*% t$e correspondent tests &ill )e removed' .or a fair comparison on field of t$e codec performance )efore and after t$e c$anges in t$e net&or* /e0 feature activation1% t$e tests $ave to )e done as muc$ as possi)le in t$e same radio conditions and at t$e same net&or* load' .or t$is% static tests are recommended to )e performed' #$e static tests $ave to )e performed )ot$ in good and )ad radio conditions to evaluate t$e )est and &orst case scenario' As t$e voice quality in t$e BSS is affected not only )y t$e radio conditions )ut also )y $andovers it is recommended to perform as &ell mo)ility tests' .or t$e mo)ility tests% t$e same route s$ould )e drove% in t$e same conditions and in t$e same direction% for t$e measurements )efore and after feature activation' Alcatel-Lucent File Date Edition Page B11 VQ measurement 19/3/09 1.0 3 2 Needed equipment 2.1 Prerequisites ,n order to carry on &it$ t$e measurements t$e follo&ing points s$ould )e ta*en into consideration #$e QVoice tool /from AS2(+1 s$ould )e used% equipped &it$ at least 3 test mo)iles &it$ forcing functions /speec$ version% )and1' QV+ /for mo)ile to mo)ile calls1 and QVS /for mo)ile to ,S-N calls1 are required QV4 for post-processing can )e remotely #$e frequency )and $as to )e t$e same during t$e tests' Availa)le ,S-N line at t$e +S2 is required Significant num)er of samples is necessary for relia)le statistical distri)utions /5677 samples/mo)ility test case and 377 samples/static test case1' #$e net&or* s$ould allo& ena)ling E."% H"% A+" 8 ."% A+" 8 H" #$e A+" parameters s$ould )e t$e A!9 optimi:ed0 2odec su)set0 53'3% 57'3% ;'< and <';6 #$res$olds0 #5=<dB% #3=;dB% #>=57dB% Hyst=3dB .or t$e stationary tests% at least t&o points &it$ good radio conditions and t&o &it$ )ad radio conditions $ave to )e considered' .or t$is% 3 different cells &it$ normal load $ave to )e c$osen' #$e parameter settings in t$e cells c$osen for t$ese tests $ave to )e t$e ones used in t$e most part of t$e net&or*' #$e route for t$e mo)ility tests $as to )e defined /same route for )efore/after tests1 including good and )ad radio conditions% &it$ reduced num)er of $andovers /avoid ping-pong1' #$e route $as to de drove in t$e same direction &it$ ma' 67*m/$' -rive test car needed' N4( access needed% for t$e monitoring of t$e net&or* ?4,s in t$e tested area% during t$e tests% to see any a)normal )e$avior of t$e net&or* t$at mig$t impact t$e results of t$e tests' #$e reference tests and t$e tests after feature activation or s&/$& migration $ave to )e performed in t$e same traffic conditions% same time of t$e day /load conditions1 and eactly on t$e same route /for t$e mo)ility tests1' 2.2 General overview on Qvoice #$e QVoice consists of t&o modules /QV+ @ QVS1' #$e QV+ /QVoice +o)ile unit1 is t$e mo)ile measurement tool installed in t$e car and QVS /QVoice Server unit1 is t$e stationary one eisting on t$e ,S-N side' #$e principle of operation is t$e follo&ing0 t$e AQV+A /or AQVSA according to t$e direction% 9!/-!% &e &ant to measure1 sends an audio pattern in a c$osen language /Englis$% .renc$% Berman or 2antonese1' #$e AQVSA /or AQV+A1 receives t$e pattern% compares it to t$e reference and calculates a note% under t$e 4ESQ metric' 6 seconds speec$ samples in different languages are transmitted in )ot$ directions' After t$e measurements are terminated% log files /C'Q+-1 from )ot$ modules are imported in t$e QVoice 4resentation station in order to perform analysis and produces reports and maps' .or +S-+S measurements% t$e QV+ is )ot$ t$e transmitter and t$e receiver of t$e audio pattern% as t$e ,S-N/4S#N net&or* is not involved in t$e call' #$e call is esta)lis$ )et&een t&o mo)iles located at t$e same QV+' Alcatel-Lucent File Date Edition Page B11 VQ measurement 19/3/09 1.0 $ .igure 50 -ifferent modules of QVoice tool for +S-,S-N measurement #$e 4rinciple of all end-to-end Voice Quality tools is t$e follo&ing0 5' A reference speec$ sample ,N' 3' A degraded speec$ sample (9#' >' #$e comparison )et&een t$em giving a score' "%&ure 2' Pr%n(%)*es +, V+%(e -ua*%t. t++*s 2.3 Speech sample classification thresholds ,n t$e follo&ing ta)le t$e ranges of t$e 4ESQ and corresponding classification are presented' Speech Quality PESQ Speech Sample From To Ecellent 6 >'6 Excellent Sample.wav Bood >'6 3'; Good Sample.wav .air 3'; 3'5 Fair Sample.wav 4oor 3'5 5'6 Poor Sample.wav Bad 5'6 5 Bad Sample.wav Alcatel-Lucent File Date Edition Page B11 VQ measurement 19/3/09 1.0 / #a'le 1( S)eec* +a,)le cla++i-ication Alcatel-Lucent File Date Edition Page B11 VQ measurement 19/3/09 1.0 0 3 Proposed tests #$e measurement test cases s$ould )e t$e ones in t$e follo&ing ta)le' Eac$ of t$e tests proposed )ello& s$ould )e made )efore and after t$e s&/$& migration or feature activation' Also% t$e results of t$e tests $ave to )e sent )y t$e measurement team after eac$ day% so t$at &e can find in time some pro)lems t$at mig$t appear and correct t$em' Time Tas 5 day Setting up equipment /QV+ D QVS1% testing equipment% validating proposed drive route% administrative issues 5 day /377 samples E 37 min1 (~ 4h with 4 mobiles) Static measurements in t&o selected cells in good radio conditions /"!ev F= -G7dBm% "Qual 710 +o)ile to ,S-N0 ." 8 ,S-N /377 samples/cells1 E." 8 ,S-N /377 samples/cell1 H" 8 ,S-N /377 samples/cell1 A+"-." 8 ,S-N /377 samples/cell1 A+"-H" 8 ,S-N /377 samples/cell1 +o)ile to mo)ile0 ." 8 ." /377 samples/cell1 E." 8 E." /377 samples/cell1 H" 8 H" /377 samples/cell1 A+"-." 8 A+"-." /377 samples/cell1 A+"-H" 8 A+"-H" /377 samples/cell1 5 day (~ 4h with 4 mobiles) Static measurements in t&o selected cells in )ad radio conditions /-H7 I= "!ev I= -577dBm% "Qual 8 no constraints10 +o)ile to ,S-N0 ." 8 ,S-N /377 samples/cell1 E." 8 ,S-N /377 samples/cell1 H" 8 ,S-N /377 samples/cell1 A+"-." 8 ,S-N /377 samples/cell1 A+"-H" 8 ,S-N /377 samples/cell1 +o)ile to mo)ile0 ." 8 ." /377 samples/cell1 E." 8 E." /377 samples/cell1 H" 8 H" /377 samples/cell1 A+"-." 8 A+"-." /377 samples/cell1 A+"-H" 8 A+"-H" /377 samples/cell1 3'6 days (1.5 days with 4 mobiles) +o)ility test measurements in t$e selected area0 +o)ile to ,S-N0 ." 8 ,S-N /5677 samples1 E." 8 ,S-N /5677 samples1 H" 8 ,S-N /5677 samples1 A+"-." 8 ,S-N /5677 samples1 A+"-H" 8 ,S-N /5677 samples1 Alcatel-Lucent File Date Edition Page B11 VQ measurement 19/3/09 1.0 1 +o)ile to +o)ile0 ." 8 ." /5677 samples1 E." 8 E." /5677 samples1 H" 8 H" /5677 samples1 A+"-." 8 A+"-." /5677 samples1 A+"-H" 8 A+"-H" /5677 samples1 > days 4ost processing and report creation Note 1! 377 samples are measured in a)out 37 minutes and 5677 samples are measured in a)out 3 $ours' Note 2! #$e duration of t$e tests is estimated in t$e supposition t$at t&o mo)iles are used' ,n t$is case% for t$e +S-,S-N test cases% on eac$ mo)ile is forced a different codec% and t&o test cases are performed at t$e same time' .or t$e +S-+S test cases% )ot$ mo)iles are used to perform one test' ,f more mo)iles are used /< or G1 t$e duration of t$e tests decreases' Here are summari:ed t$e foreseen duration% resources for all t$e tests descri)ed a)ove' 2odec #ype of test Human resources #ools -uration ." 8 ,S-N Static good radio conditions 5car and driver D 5 person AS2(+ QVoice 3C37 min Static )ad radio conditions 3C37 min +o)ility 3$ ." - ." Static good radio conditions 3C37 min Static )ad radio conditions 3C37 min +o)ility 3$ ." codec All test cases G$ @ <7 min E." 8 ,S-N Static good radio conditions 5car and driver D 5 person AS2(+ QVoice 37 min Static )ad radio conditions 37 min +o)ility 3$ E." - E." Static good radio conditions 37 min Static )ad radio conditions 37 min +o)ility 3$ E." codec All test cases G$ @ <7 min H" 8 ,S-N Static good radio conditions 5car and driver D 5 person AS2(+ QVoice 37 min Static )ad radio conditions 37 min +o)ility 3$ H" - H" Static good radio conditions 37 min Static )ad radio conditions 37 min +o)ility 3$ H" codec All test cases G$ @ <7 min A+"-." 8 ,S-N Static good radio conditions 5car and driver D 5 person AS2(+ QVoice 37 min Static )ad radio conditions 37 min +o)ility 3$ A+"-." - A+"-." Static good radio conditions 37 min Static )ad radio conditions 37 min +o)ility 3$ A+"-." codec All test cases G$ @ <7 min A+"-H" 8 ,S-N Static good radio conditions 5car and driver D 5 person AS2(+ QVoice 37 min Static )ad radio conditions 37 min +o)ility 3$ A+"-H" - A+"-H" Static good radio conditions 37 min Static )ad radio conditions 37 min +o)ility 3$ A+"-H" codec All test cases G$ @ <7 min All codecs All test cases E6 days Alcatel-Lucent File Date Edition Page B11 VQ measurement 19/3/09 1.0 2 Note! #$e a)ove ta)le doesnJt ta*e into account t$e possi)ility of performing more +S-,S-N tests at a time using more mo)iles' #$e ta)le is intended to )e $elpful in t$e assessment of t$e tests duration in case t$at t$e net&or* doesnJt support some of t$e codecs and some proposed tests $ave to )e removed' #$e estimation of t$e total test duration% ta*ing into account t$e a)ove considerations0 A(t%3%t. #urat%+n #eta%*s #est preparation - 5 day "eference tests ." codec 5 day E." codec 5 day H" codec 5 day A+"-." codec 5 day A+"-H" codec 5 day "eference tests All codecs 6 days .ollo&-up tests ." codec 5 day E." codec 5 day H" codec 5 day A+"-." codec 5 day A+"-H" codec 5 day .ollo&-up tests All codecs 6 days 4ost processing and report creation - > days #est trial - 3 &ee*s #$e follo&ing ?4,s is recommended to )e monitored during t$e period t$at t$e tests are performed and in t$e designated test area /for mo)ility tests1 or in t$e designated test cells /for static tests1' CS Summary "# monitorin$ K S-22H Assign 2ongestion K H( ,nc ,ntra .ail K S-22H Assign .ail K H( ,nc ,ntra 2ong K S-22H -rop K H( ,nc ,ntra Success S-22H #raffic Erl #otal K H( (ut ,ntra "(2 "#2H #raffic Erl #otal K H( (ut ,ntra -rop K "#2H Assign 2ongestion K H( (ut ,ntra Success K "#2H Assign Success K H( ,nc ,nter .ail K2all Setup Suc K H( ,nc ,nter 2ong K "#2H fail "adio K H( ,nc ,nter Success K 2all -rop K H( (ut ,nter "(2 K "#2H -rop K H( (ut ,nter -rop K H( (ut ,nter Success KH( Better 2ell Alcatel-Lucent File Date Edition Page B11 VQ measurement 19/3/09 1.0 4 % E&pected reported results After t$e measurements campaign of eac$ day% t$e results s$ould )e given )y t$e su)contractor in Ecel format /or 2SV1 containing t$e follo&ing fields0 #$e file containing t$e 4ESQ/4A2E results0 C:\Work\Issues\ oice!"ualit#\Exemple\$e%uested &ormat\EF$!Pace!C'(!)o*!+,(-+--..xls #$e file containing t$e H( information0 /:\work\Issues\ oice!"ualit#\Exemple\$e%uested &ormat\EF$!01!)o*!C'(!)o*!+,(-+--..xls #$e file containing t$e "!ev/ "Qual information0 C:\Work\Issues\ oice!"ualit#\Exemple\$e%uested &ormat\EF$!$x"!$x2!C'(!)o*!+,(-+--..xls
#$e final report s$all include t$e comparison of Voice Quality per codec and in general% )efore and after soft&are migration or feature activation in static and drive test conditions' #$e comparison &ill include average voice quality /&it$ and &it$out $andovers1 sample distri)ution /according to sample classification1% comparison on map /"!ev% "Qual% Voice Quality distri)ution1 #$e outcome of t$e test &ill reveal if t$e activation of t$e feature )roug$t any c$anges in t$e Voice Quality' EN' #F '#C()ENT Alcatel-Lucent File Date Edition Page B11 VQ measurement 19/3/09 1.0 10