SwissQual License AG Allmendweg 8 CH-4528 Zuchwil Switzerland t +41 32 686 65 65 f +41 32 686 65 66 e info@swissqual.com www.swissqual.com
SwissQual has made every effort to ensure that eventual instructions contained in the document are adequate and free of errors and omissions. SwissQual will, if necessary, explain issues which may not be covered by the documents. SwissQuals liability for any errors in the documents is limited to the correction of errors and the aforementioned advisory services. Copyright 2000 - 2013 SwissQual AG. All rights reserved. No part of this publication may be copied, distributed, transmitted, transcribed, stored in a retrieval system, or translated into any human or computer language without the prior written permission of SwissQual AG. Confidential materials. All information in this document is regarded as commercial valuable, protected and privileged intellectual property, and is provided under the terms of existing Non-Disclosure Agreements or as commercial-in-confidence material. When you refer to a SwissQual technology or product, you must acknowledge the respective text or logo trademark somewhere in your text. SwissQual, Seven.Five, SQuad, QualiPoc, NetQual, VQuad, Diversity as well as the following logos are registered trademarks of SwissQual AG.
Diversity Explorer, Diversity Ranger, Diversity Unattended, NiNA+, NiNA, NQAgent, NQComm, NQDI, NQTM, NQView, NQWeb, QPControl, QPView, QualiPoc Freerider, QualiPoc iQ, QualiPoc Mobile, QualiPoc Static, QualiWatch-M, QualiWatch-S, SystemInspector, TestManager, VMon, VQuad-HD are trademarks of SwissQual AG. SwissQual acknowledges the following trademarks for company names and products: Adobe, Adobe Acrobat, and Adobe Postscript are trademarks of Adobe Systems Incorporated. Apple is a trademark of Apple Computer, Inc. DIMENSION, LATITUDE, and OPTIPLEX are registered trademarks of Dell Inc. ELEKTROBIT is a registered trademark of Elektrobit Group Plc. Google is a registered trademark of Google Inc. Intel, Intel Itanium, Intel Pentium, and Intel Xeon are trademarks or registered trademarks of Intel Corporation. INTERNET EXPLORER, SMARTPHONE, TABLET are registered trademarks of Microsoft Corporation. Java is a U.S. trademark of Sun Microsystems, Inc. Linux is a registered trademark of Linus Torvalds. Microsoft, Microsoft Windows, Microsoft Windows NT, and Windows Vista are either registered trademarks or trademarks of Microsoft Corporation in the United States and/or other countries U.S. NOKIA is a registered trademark of Nokia Corporation. Oracle is a registered US trademark of Oracle Corporation, Redwood City, California. SAMSUNG is a registered trademark of Samsung Corporation. SIERRA WIRELESS is a registered trademark of Sierra Wireless, Inc. TRIMBLE is a registered trademark of Trimble Navigation Limited. U-BLOX is a registered trademark of u-blox Holding AG. UNIX is a registered trademark of The Open Group.
Contents
1 Introduction .......................................................................................................................................... 1 Overview of a Measurement Cycle........................................................................................................ 1 Reference Documentation ..................................................................................................................... 3 2 Ping Test ............................................................................................................................................... 4 Objective and Purpose .......................................................................................................................... 4 Basic Function ....................................................................................................................................... 4 IPv6 Support .......................................................................................................................................... 4 Forcing Diversity to Use a Specific IP Version for a Test ................................................................ 5 IPv6 Address Format........................................................................................................................ 6 IPv4 and IPv6 Traffic in NQDI .......................................................................................................... 6 Result Analysis ...................................................................................................................................... 7 3 Ping Trace Test .................................................................................................................................... 9 IPv6 Support .......................................................................................................................................... 9 Objective and Purpose .......................................................................................................................... 9 Basic Function ....................................................................................................................................... 9 Results Analysis .................................................................................................................................... 9 4 FTP Test .............................................................................................................................................. 11 Objective and Purpose ........................................................................................................................ 11 Basic Function ..................................................................................................................................... 11 Impact of Send Buffer Size parameter value ................................................................................ 11 Results Analysis .................................................................................................................................. 12 5 HTTP Browser Test............................................................................................................................ 14 Objective and Purpose ........................................................................................................................ 14 Basic Function ..................................................................................................................................... 14 Results Analysis .................................................................................................................................. 14 6 HTTP Transfer Test............................................................................................................................ 16 Objective and Purpose ........................................................................................................................ 16 Basic Function ..................................................................................................................................... 16 Results Analysis .................................................................................................................................. 16 7 Capacity Test ..................................................................................................................................... 18 Objective and Purpose ........................................................................................................................ 18 Basic Function ..................................................................................................................................... 18 Result Analysis .................................................................................................................................... 19 8 UDP Downlink Test ............................................................................................................................ 21 Objective and Purpose ........................................................................................................................ 21
Contents | CONFIDENTIAL MATERIALS ii
Basic Function ..................................................................................................................................... 21 Results Analysis .................................................................................................................................. 21 9 UDP Plus Test .................................................................................................................................... 23 Objective and Purpose ........................................................................................................................ 23 Basic Function ..................................................................................................................................... 23 Results Analysis UDPPlus ................................................................................................................... 23 10 Email Send Test ................................................................................................................................. 26 Objective and Purpose ........................................................................................................................ 26 Basic Function ..................................................................................................................................... 26 Results Analysis .................................................................................................................................. 26 11 Email Receive Test ............................................................................................................................ 28 Objective and Purpose ........................................................................................................................ 28 Basic Function ..................................................................................................................................... 28 Results Analysis .................................................................................................................................. 28 12 GPRS Attach Test .............................................................................................................................. 30 Objective and Purpose ........................................................................................................................ 30 Basic Function ..................................................................................................................................... 30 Results Analysis .................................................................................................................................. 30 13 GPRS Detach Test ............................................................................................................................. 32 Objective and Purpose ........................................................................................................................ 32 Basic Function ..................................................................................................................................... 32 Results Analysis .................................................................................................................................. 32 14 PDP Activation Test........................................................................................................................... 34 Objective and Purpose ........................................................................................................................ 34 Basic Function ..................................................................................................................................... 34 Results Analysis .................................................................................................................................. 34 15 PDP Deactivation Test ...................................................................................................................... 36 Objective and Purpose ........................................................................................................................ 36 Basic Function ..................................................................................................................................... 36 Results Analysis .................................................................................................................................. 36 16 WAP Test ............................................................................................................................................ 37 Objective and Purpose ........................................................................................................................ 37 Basic Function ..................................................................................................................................... 37 Results Analysis .................................................................................................................................. 37 17 SMS Send Test ................................................................................................................................... 39 Objective and Purpose ........................................................................................................................ 39
Contents | CONFIDENTIAL MATERIALS iii
Basic Function ..................................................................................................................................... 39 Results Analysis .................................................................................................................................. 39 18 SMS Receive Test .............................................................................................................................. 41 Objective and Purpose ........................................................................................................................ 41 Basic Function ..................................................................................................................................... 41 Results Analysis .................................................................................................................................. 41 19 MMS Send Test .................................................................................................................................. 43 Phone Configuration ............................................................................................................................ 43 Objective and Purpose ........................................................................................................................ 43 Basic Function ..................................................................................................................................... 43 Results Analysis .................................................................................................................................. 43 Trigger Points MMS Sender ................................................................................................................ 48 20 MMS Receive Test ............................................................................................................................. 49 Objective and Purpose ........................................................................................................................ 49 Basic Function ..................................................................................................................................... 49 Results Analysis .................................................................................................................................. 49 Trigger Points MMS Receiver .............................................................................................................. 53 A Port Usage and Firewall Settings..................................................................................................... 54 Dialup Network Connections ............................................................................................................... 54 B Reducing the Effective Pause Time ................................................................................................. 59
Figures
Figure 1-1 Overview of the idle times and pauses in a measurement cycle ..................................................... 1 Figure 1-2 NDIS post connect delay in NQDI. ................................................................................................... 2 Figure 2-1 Ping Test - Basic Function ............................................................................................................... 4 Figure 2-2 Network adapter properties .............................................................................................................. 5 Figure 2-3 IPv4 and IPv6 traffic throughput in NQDI ......................................................................................... 7 Figure 2-4 Ping Test - NQDI Result Analysis Window ...................................................................................... 8 Figure 3-1 Ping Trace Test - Basic Function ..................................................................................................... 9 Figure 3-2 Ping Trace Test - NQDI Result Analysis Window .......................................................................... 10 Figure 4-1 FTP Test - Basic Function .............................................................................................................. 11 Figure 4-2 FTP Test - NQDI Result Analysis Window..................................................................................... 13 Figure 5-1 Test HTTP Browser Test - Basic Function.................................................................................. 14 Figure 5-2 HTTP Browser Test - NQDI Result Analysis Window .................................................................... 15 Figure 6-1 Test HTTP Transfer Test - Basic Function ................................................................................. 16 Figure 6-2 HTTP Transfer Test - NQDI Result Analysis Window.................................................................... 17 Figure 7-1 Overview of a Capacity test ........................................................................................................... 18 Figure 7-2 Results of Capacity test ................................................................................................................. 20
Contents | CONFIDENTIAL MATERIALS iv
Figure 8-1 UDP Downlink Test Basic Function ............................................................................................ 21 Figure 8-2 UDP Downlink Test - NQDI Result Analysis Window .................................................................... 22 Figure 9-1 UDP Plus Test Basic Function .................................................................................................... 23 Figure 9-2 UDP Plus Test - NQDI Result Analysis Window ............................................................................ 25 Figure 10-1 Email Send Test Basic Function ............................................................................................... 26 Figure 10-2 Email Send Test - NQDI Result Analysis Window ....................................................................... 27 Figure 11-1 Email Receive Test Basic Function .......................................................................................... 28 Figure 11-2 Email Receive Test - NQDI Result Analysis Window .................................................................. 29 Figure 12-1 GPRS Attach Test Basic Function ............................................................................................ 30 Figure 12-2 GPRS Attach Analysis window (NQDI) ........................................................................................ 31 Figure 13-1 GPRS Detach Test Basic Function ........................................................................................... 32 Figure 13-2 GPRSDetach Analysis Window (NQDI) ....................................................................................... 33 Figure 14-1 PDP Activation Test Basic Function ......................................................................................... 34 Figure 14-2 PDP Activation Test - NQDI Result Analysis Window ................................................................. 35 Figure 15-1 PDP Deactivation Test Basic Function ..................................................................................... 36 Figure 15-2 PDP Deactivation Test - NQDI Result Analysis Window ............................................................. 36 Figure 16-1 WAP Test - NQDI Result Analysis Window ................................................................................. 38 Figure 17-1 SMSSend Test Diagram .............................................................................................................. 39 Figure 17-2 SMS Send test - Result Analysis Window ................................................................................... 39 Figure 18-1 SMS Receive Test - Basic Function ........................................................................................... 41 Figure 18-2 SMS Receive Test - NQDI Result Analysis Window.................................................................... 42 Figure 18-3 SMS Receive Test - NQDI Result Analysis Window.................................................................... 42 Figure 19-1 MMS Send Test Basic Function ................................................................................................ 43 Figure 19-2 MMS Send Test - NQDI Result Analysis Window........................................................................ 45 Figure 19-3 MMS Send Test - NQDI Result Analysis Window........................................................................ 46 Figure 19-4 MMS Send Test - NQDI Result Analysis Window........................................................................ 47 Figure 20-1 MMS Receive Test Basic Function ........................................................................................... 49 Figure 20-2 MMS Receive Test - NQDI Result Analysis Window ................................................................... 50 Figure 20-3 MMS Receive Test - NQDI Result Analysis Window ................................................................... 51 Figure 20-4 MMS Receive Test - NQDI Result Analysis Window ................................................................... 52 Figure A-1 GPRS markers in the GSM Analysis Window ............................................................................... 55 Figure A-2 GPRS markers in the Test Specific Analysis Window ................................................................... 56 Figure A-3 GPRS Markers in the Map Window ............................................................................................... 56 Figure A-4 PDP Context Activation after GPRS Dial ....................................................................................... 57 Figure A-5 GPRS PDP Context Activation ...................................................................................................... 57 Figure A-6 Data selection for GPRS dial-up connect failures ......................................................................... 58
Tables
Table 2-1 Timing of actions during a Ping Test ................................................................................................. 4 Table 2-2 Ping Test - Result Analysis ............................................................................................................... 7
Contents | CONFIDENTIAL MATERIALS v
Table 3-1 Ping Trace Test - Result Analysis ..................................................................................................... 9 Table 4-1 Timing of actions during an FTP Test ............................................................................................. 11 Table 4-2 FTP Test - Result Analysis .............................................................................................................. 12 Table 5-1 HTTP Browser Test - Result Analysis ............................................................................................. 14 Table 6-1 Timing of actions during an HTTP Transfer Test ............................................................................ 16 Table 6-2 HTTP Transfer Test - Result Analysis ............................................................................................. 16 Table 7-1 Description of the Capacity test data .............................................................................................. 19 Table 8-1 UDP Downlink test - Result Analysis .............................................................................................. 21 Table 9-1 UDP Plus Test - Result Analysis ..................................................................................................... 24 Table 9-2 UDP Plus Test - Values in Statistics Register ................................................................................. 24 Table 9-3 UDP Plus test - Statistics in Repetition Table ................................................................................. 25 Table 10-1 Email Send Test - Result Analysis ................................................................................................ 26 Table 11-1 Email Receive Test - Result Analysis ........................................................................................... 28 Table 12-1 Timing of actions during a GPRS Attach Test............................................................................... 30 Table 13-1 Timing of actions during a GPRS Detach Test ............................................................................. 32 Table 14-1 Timing of actions during a PDP Activation Test ............................................................................ 34 Table 15-1 Timing of actions during a PDP Deactivation Test ........................................................................ 36 Table 16-1 WAP Test - Result Analysis .......................................................................................................... 37 Table 17-1 SMS Send Test - Result Analysis ................................................................................................. 40 Table 18-1 SMS Receive Test - Result Analysis ............................................................................................. 41 Table 19-1 MMS Send Test - Result Analysis ................................................................................................. 43 Table 20-1 MMS Receive Test - Result Analysis ............................................................................................ 49 Table A-1 Port Number Listing ........................................................................................................................ 54 Table A-2 UDP Ports ....................................................................................................................................... 54
vi
Introduction
This document describes the Data and Messaging tests available with the Diversity Measurement Units, SwissQuals Subscriber Experience Management system for audio, data, messaging and video quality. Each test is handled in a separate chapter, consisting of a description of the test flow and the result analysis. Analysis is presented with NQDI, SwissQuals post processing application. Were necessary and appropriate, in depth detail information is provided as well. The appendix outlines general information as well other kind of nice to know descriptions. Note: Please note that this document does not describe how to setup Diversity nor does it deal with configuring the data and messaging tests. These topics are provided in document [1].
IDLE
DATA SESSION
IDLE
DATA SESSION
IDLE
GPRS Attach
HTTP
Get
FTP Get
A data session contains a series of one or more tests that are separated by pauses.
pause
pause
pause
Preparation
Test
Clean Up
Test session includes initial preparation (if required), the test, and a post-test clean-up phase.
The preparation phase typically lasts approximately 100 ms and is therefore not significant. Important: This phase might take longer for the GPRS attach test if the test requires a preliminary detach operation. Furthermore, Windows 7 can introduce instabilities into the test. The duration of the clean-up phase can vary based on the type of test. The clean-up phase of the following test types lasts is typically 1.5 seconds: Ping
1 CONFIDENTIAL MATERIALS
Chapter 1 | Introduction
GPRS Attach GPRS Detach PDP Activation PDP Deactivation SMS Send SMS Receive
The clean-up phase for the following test types typically lasts 3 seconds: HTTP Transfer HTTP Browser FTP, Capacity IPERF Ping Trace UDP Email MMS send MMS receive
Note: In the worst case, the duration of the clean-up phase might double. The NDIS option for a task introduces an additional pause after the connection has been established. This pause is the NDIS post connection delay and has a duration of 500 ms.
Note: Although you can change the duration of the clean-up phase and the NDIS post connect delay with Windows registry keys, SwissQual strongly recommends that you use the default values to avoid issues with trace recording or test success rate.
Reference Documentation
Reference
[1] [2]
Document
Manual NetQual - NQView Manual NQDI - KPI Users Guide
Ping Test
Basic Function
The test sends the ping (ICMP Echo Request) to the target host with the configured ICMP packet size and waits for the configured timeout duration for a successful response. If the ICMP Echo Reply does not arrive to the client within timeout, the ping response is considered failed.
Client Host
st
RTT1
ICMP Echo
Reply
Interval
ICMP Echo Re
quest
RTT2
ICMP Echo
Reply
The procedure is being repeated until the amount of configured ping attempts are sent to the host. Between two attempts the pause duration (Interval time) is applied.
Table 2-1 Timing of actions during a Ping Test
Phase
Preparation Test Clean Up
Action
Start test Ping request Waiting for trace/clean up
Duration
< 100 ms 21 s max 1.5* s
MsgLogTrace
Ping Test Started Ping reply - RTT
IPv6 Support
Diversity ping and ping trace tests support Internet Control Message Protocol version 6 (ICMPv6). IPv6 is enabled by default in Windows on factory PCM reference images starting from Diversity release 12.1 onward. The protocol is enabled with the following Windows registry key: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\services\TCPIP6\Parameters\DisabledComponents When the key is set to 0x00000000 or is not present, IPV6 is enabled.
Chapter 2 | Ping Test CONFIDENTIAL MATERIALS 4
Note: In older versions of PCM factory reference images, this key is set to fxffffffff to disable IPV6.
Important: Do not disable Internet Protocol Version 4 as this protocol is required for internal traffic between the controlling PC and the PCMs in a Diversity system as well as DNS traffic.
3. On the Edit menu, point to New, and then click String Value. 4. Rename the key to ForceIPVersion. 5. Right-click the key, click Modify, and in the Edit String dialog box enter the value for the Internet protocol that you want Diversity to use. For IPv4, type 0 For IPv6, type 1 For automatic protocol selection type 2
Note: In automatic mode, Diversity attempts to connect with IPv6 before IPv4. 6. Click OK.
Result Analysis
The post processing application NQDI shows the following data on the result analysis tab sheet.
Table 2-2 Ping Test - Result Analysis
Value
Host Avg. RTT [ms] No. of successful Pings Packet Size [Bytes] Time State RTT [ms] Sequence Number Msg Error Code
Description
Name or IP address of the host Average Round Trip Time of all ICMP Echo request/response RTTs The number of successful Pings ICMP packet size Current timestamp of message Not used/Reserved Round Trip Time of current ping Sequence number of ping for this test; starting with value 1 State message: OK if successful, otherwise the appropriate error text corresponding to the error code. 0 if successful; otherwise the error code
IPv6 Support
Diversity ping trace and ping tests support Internet Control Message Protocol version 6 (ICMPv6). For more information, see the "IPv6 Support" section on page 4.
Basic Function
The test sends the ping (ICMP Echo Request) to the target host with the configured ICMP packet size and a modified Time to Live (TTL) count value of 1 to the target destination. The TTL value indicates how many Hosts (aka Hop) are being passed through, before a ICMP Time to live exceeded message occurs. If the response message arrived within configured timeout the first Hop is considered found. The RTT value between client and that Hop is being calculated. Now, the TTL value is increased by one and the ICMP Echo Request is sent again. These steps are being repeated until the target destination could be reached.
Client
ICMP Echo Re quest (TTL=1)
RTT1
Hop
Hop
Host
ICMP TTL
Exceeded
ICMP TTL Ex
Results Analysis
The post processing application NQDI shows the following data on the result analysis tab sheet:
Table 3-1 Ping Trace Test - Result Analysis
Value
Time Hop
Description
Current timestamp of the message Number of the router passed.
9 CONFIDENTIAL MATERIALS
Value
RTT Host DNS Name Status
Description
Round Trip Time between Client and Hop IP address of either the hop where the TTL exceeded or the target host responding with the Echo Reply message. Not used / Reserved State message: OK if successful, otherwise the appropriate (ICMP) error text.
10
FTP Test
Basic Function
Before a file can be transferred over FTP, a control channel connection has to be established to the FTP server. This control channel is used to exchange FTP commands between client and server. Commands such as user account login, directory changing and choosing the FTP operation are applied. When it comes to a FTP GET (file download from server) or FTP PUT (file upload to the server) operation, the data channel is being established, which then is used to transfer the raw file data.
Client FTP Server
Control Connection
Data Connection
Important: When encountering connection problems (no or closed connections) with the selected Active/Passive mode, try using the other mode. Firewalls and NAPT may block or close connections. The control channel remains open during file transfer in order to exchange other commands (such as aborting the file transfer, and so on). Note: Active mode: The server connects to the client on a port provided by the client. Passive mode: The client connects to the server on a port provided by the server.
Table 4-1 Timing of actions during an FTP Test
Phase
Preparation Test Clean Up
Action
Start test FTP GET or FTP PUT Waiting for trace/clean up
Duration
< 100 ms 120 s max 3* s
MsgLogTrace
FTP Get/Put Test Started
then, starts to send the info. Meanwhile, the software thinks that it just sent 4096 bytes in a fraction of a second. We have mitigated the impact of this phenomenon by averaging over several seconds, but this does not fully eliminate the impact of buffering. Besides, what we are measuring in the ftp monitor is application throughput, and this is what we are reporting. Note: It is possible to have application throughputs exceeding the theoretical physical limitations due to enhanced phone and PC buffering techniques.
Results Analysis
The post processing application NQDI shows the following data on the result analysis tab sheet:
Table 4-2 FTP Test - Result Analysis
Value
Time Operation Filename Throughput
Description
Current timestamp of the message Operation, either PUT or GET Name of the file transferred or received As long as the state is listed as In Progress, this value represents the intermediate throughput in bytes/s, calculated as (number of bytes transferred since last message) / (time needed). At the end of a FTP session, the state will change to Success or Failed. In case of Success, the throughput value contains the overall throughput calculated as: (total number of bytes transferred) / (total time needed). In case of Failed, the overall value not calculated Name or IP address of the FTP host Intermediate number of bytes transferred In Progress, Success, Failed State message: OK if successful, otherwise the appropriate error text corresponding to the error code.
12
13
Basic Function
The test implementation uses Microsofts Internet Explorer (IE) for navigating to and downloading the web page. On test initialization the IEs temporary file folder (cache files) is being emptied in order to prevent the loading the page from the local file system. Also, any existing IE browser instances are being closed to avoid noninvolved HTTP traffic. Afterwards the web page is being downloaded. Once the IE instance reports the Navigation complete event the test is considered finished. Now, application throughput is calculated and number of images is counted. Request for HTML page is sent.
Time and HTTP throughput is calculated and recorded.
Web Server
Results Analysis
The post processing application NQDI shows the following data on the result analysis tab sheet:
Table 5-1 HTTP Browser Test - Result Analysis
Value
Msg Time Url Number of images Number of frames Total Size Throughput Error
Client
Port 80
HTML files and other related files, such as pictures are received.
Description
Current timestamp when result message has been written. Uniform Resource Locator Number of images within the web page. Number of frames within the web page Total size of web page. Throughput = totalSize / (time between navigate and documentComplete) 0 if successful; otherwise the error code
14
15
Basic Function
Request for HTML page is sent.
Time and HTTP throughput is calculated and recorded
Web Server
Note: By default, you cannot upload any type of file to the HTTP server. Instead, you need to enable this feature for the directory on the HTTP server. You can either grant universal access to the directory or require user name and password authentication.
Table 6-1 Timing of actions during an HTTP Transfer Test
Results Analysis
The post processing application NQDI shows the data in the following table on the result analysis tab sheet.
Table 6-2 HTTP Transfer Test - Result Analysis
Value
Msg Time Host Operation Buffer Size
Client
Port 80
Phase
Preparation Test Clean Up
Action
Start test HTTP GET Waiting for trace/clean up
Duration
< 100 ms 45 s max 3* s
MsgLogTrace
HTTPTransfer Get/Put Test Started
Description
Time when result message is written URL or IP address of the HTTP server GET (Download) or PUT (Upload) operation PUT operation uses local buffer (recommended size 32 kB)
16
Value
Local file name Remote file name Bytes Transferred State Error Throughput
Description
PUT operation only Path and file name on the HTTP server Intermediate number of bytes that have been transferred (Last Block = 0) Final file size (Last Block = 1) Possible states are: In Progress, Success, Failed State message is OK if the test is successful, otherwise the error text corresponding to the error code is displayed. As long as the state is listed as In Progress, this value represents the intermediate throughput in bytes/s, which is calculated as (number of bytes transferred since last message) / (time needed). At the end of the test, the state changes to Success or Failed. If the state is Success, the throughput value contains the overall throughput that is calculated as: (total number of bytes transferred) / (total time needed). If the state is Failed, the overall value is not calculated.
17
Capacity Test
Basic Function
The test configuration requires you to define a set of HTTP URLs. The target files, which are transferred simultaneously during the test, can reside on different servers. Important: To avoid incomplete and aborted transfers during the duration of the test, use files that are larger than 1 GB in size.
Start Data Transfer 1...n Test Duration Stop Data Transfer 1...n
Time
An error with one or more of the simultaneous transfers results in a test failure. Similarly, if one or more of these transfers successfully completes before the test duration elapses, the test fails.
18
Result Analysis
The following table describes the Capacity test data that you can analyse in NQDI.
Table 7-1 Description of the Capacity test data
Value
Msg Time Host Operation Buffer Size Local file name Remote file name Bytes Transferred State Error Throughput
Description
Time when results arrive List of the URLs for simultaneous transfer GET (Download) or PUT (Upload) operation Protocol for the data transfer PUT operation uses local buffer (recommended size 32 kB) PUT operation only Intermediate number of bytes that have been transferred (Last Block = 0) Final size (Last Block = 1) Possible states are: In Progress, Success, Failed Provides information on errors that occur during the test. If the test state is "In Progress", this value represents the intermediate throughput in bytes/s, which is calculated as (number of bytes transferred since last message) / (time needed). At the end of the test, the state changes to "Success" or "Failed". If the state is "Success", the throughput value contains the overall throughput that has been calculated as: (total number of bytes transferred) / (total time needed). If the state is Failed, the overall throughput cannot be considered.
19
20
Basic Function
Firstly, the server application is being configured. The configuration consists of application identifier (SQUDPDL), test number, number of packets to be sent by the server, interval, packet size and bandwidth. Once the server acknowledged the configuration, the TCP connection is terminated. Then, after a delay of one second the server begins sending the UDP packets towards the client with a calculated time gap of [nPackets * packetSize / bandwidth]. Note, that the time between the packets is limited to a 10-ms timeframe. Therefore, it is not possible to specify the bandwidth for each packet size. Request for UDP packets is sent. Single ACK packet is received.
The client logs each packet and calculates average throughput, jitter and packet loss for each bandwidth. As there can be delays on the network or even packet loss, the client can wait an additional buffer time after the last packet should have arrived. This simulates the buffer time used by several streaming applications.
Results Analysis
The post processing application NQDI shows the following data on the result analysis tab sheet:
Table 8-1 UDP Downlink test - Result Analysis
Value
Msg Time Host PacketSize Bandwidth
Description
Current timestamp of the message Name or IP address of the UDP server Size of each packet Bandwidth used to send packets
21 CONFIDENTIAL MATERIALS
Value
Throughput Packets Sent Packets Received Jitter Error
Description
Throughput = ((packetsRX-1) * packetSize) / (tLastPacket tFirstPacket) Number of packets sent Number of packets received in time 2 x standard deviation of time between packets State message: OK if successful, otherwise the appropriate error text corresponding to the error code.
22
Basic Function
The UDPPlus test requires a server application (squdpplus) which can communicate with the client (client side starts test). When a test is started, the client connects to the server and configures a test session (TCP). After successful server configuration, the client starts to send packets to the server using the pattern defined in the chirp file. The client sends only packets up to the size of the buffer (buffer size is defined with the Maximum capacity for send buffer parameter). If the capacity of the buffer is reached, client packet sending is suspended until the corresponding reply packet is received (on match, packet is removed from buffer) or if the packet in the buffer reached the timeout it is being deleted from the buffer and marked as lost. The server receives each packet and sends it back. Packet size can be adjusted by the server if the appropriate parameters have been set. The client logs each arriving packet and calculates the roundtrip time (per packet) and throughput (per chirp file repetition). If the server receives no packets for a user defined time, the server starts to send server generated packets to the client. If option is set, the client can handle these sever generated packets as regular downlink response packets. Connection request Configuration Info (TCP) Configuration (TCP)
The send/receive process is repeated for all UDP packets. Throughput and roundtrip time are recorded.
UDP Packet is
Server
Client
UDP Packet is
represents the RTT (H, U, B) or delay (S) of the packet. Lost packets are displayed as red dots at the sent time by the client. Lost server originated packets are not displayed.
Table 9-1 UDP Plus Test - Result Analysis
Value
ReceiveTime Type Number SendTime ServerTime Host RTT [ms] Delay [ms] Repetition Error
Description
The client receive timestamp Packet type: H=RTT, U=Uplink bandwidth, B=Downlink bandwidth, S=Server originated Packet sequence number Client send timestamp Server send timestamp UDPPlus server host address Roundtrip time for type H, U and B Transmission delay from server to client for type S Chirp file repetition number of this packet OK, if no error has occurred; error code if an error has occurred. Lost - if packet did not reach the client within the test time
In the Statistics window, the following values are calculated from each repetition or from the whole test:
Table 9-2 UDP Plus Test - Values in Statistics Register
Value
Lost Packets Packet Loss [%] RTT Mean [ms] RTT Standard Deviation [ms] RTT x% Value [ms] Uplink Bandwidth Mean [kbit/s] Uplink Bandwidth Standard Deviation [kbit/s] Uplink Bandwidth x% Value [kbit/s] Downlink Bandwidth Mean [kbit/s] Downlink Bandwidth Standard Deviation [kbit/s] Downlink Bandwidth x% Value [kbit/s]
Description
Number of lost packets Percentage of lost packets Mean RTT value for all type H packets Standard deviation value for all type H packets RTT value which x% of all H packets do not exceed Uplink bandwidth mean value for all chirp file repetitions Uplink bandwidth standard deviation value for all type chirp file repetitions
Uplink bandwidth value x% of all chirp file repetitions do not fall below Downlink bandwidth mean value for all chirp file repetitions Downlink bandwidth standard deviation value for all type chirp file repetitions
Downlink bandwidth value x% of all chirp file repetitions do not fall below
The Repetition Table shows statistics values per chirp file repetition within a test:
24
Repetition
RTT Mean [ms] RTT Std Dev [ms] RTT x% Val [ms] ULBandwidth [kbit/s] DLBandwidth [kbit/s] Lost Packets
25
10
Basic Function
The throughput calculated & recorded. SMTP Server Client
Results Analysis
The post processing application NQDI shows the following data on the result analysis tab sheet:
Table 10-1 Email Send Test - Result Analysis
Value
Msg Time SMTP Server Email Address Subject Size Connect Duration Send Duration Throughput Error Code
Description
Current timestamp of the message Name or IP address of the SMTP server First part of the email address. (user) Subject entered by the user. Total size of the email (7bit encoded). Time to connect to SMTP server Time to send email Throughput = Size / Send Duration OK, if no error has occurred; error code if an error has occurred.
26
27
11
Basic Function
The test client connects to the specified POP3 server and authenticates itself with username and password. In case an Email subject line has been defined, the client goes through all messages within the mailbox and searches for the first Email matching the subject. This search time is accounted separately. Once the message is found the transfer begins. The time used for transfer and the Email size is then being used to calculate the throughput value. Once the Email has been downloaded, the operation (delete Email; delete all Emails or leave Email is performed). Port 25 An email with a specific subject title is downloaded.
Figure 11-1 Email Receive Test Basic Function
SMTP Server
Results Analysis
The post processing application NQDI shows the following data on the result analysis tab sheet:
Table 11-1 Email Receive Test - Result Analysis
Value
Msg Time POP3 Server Email Address Subject Size Connect Duration Search Duration Receive Duration Message Number Throughput Error Code
Client
Description
Current timestamp of the message Name or IP address of the SMTP server First part of the email address. (user) Subject entered by the user. Total size of the email (7bit encoded). Time to connect to SMTP server Time to find message with same subject line Time to retrieve email. The message number matching the subject line (if found) Throughput = Size / Receive Duration OK, if no error has occurred; error code if an error has occurred.
28
29
12
Basic Function
In a first step the device will check whether or not it is already attached to the network. Network ATTACH ACCEPT is sent, if the attach is successful. Client Application attempts to attach to GPRS network.
Figure 12-1 GPRS Attach Test Basic Function
If yes, it will perform a GPRS detach command and afterwards start with the actual attempt to perform a GPRS attach.
Table 12-1 Timing of actions during a GPRS Attach Test
Phase
Preparation Preparation Preparation Test Clean Up
Action
Start test Opening COM Port Detaching device Attaching device Waiting for trace/clean up
Duration
< 100 ms 0.5* s 60 s max 60 s max 1.5* s
MsgLogTrace
GPRSAttach Test Started Waiting for the DevMan MobFunc service Performing action: Detaching device Sending attach command [AT+CGATT=1]
Results Analysis
The post processing application NQDI shows the following data on the result analysis tab sheet. The Analysis Window displays the following:
30
31
13
Basic Function
In a first step the device will check whether or not it is already detached to the network. If yes or in case the state could not be determined, it will perform a GPRS detach command. If already detached, the test will returns successful without performing any operation. Network DETACH ACCEPT will be returned, if the detach is successful. Client Application attempts to detach from GPRS network.
Figure 13-1 GPRS Detach Test Basic Function Table 13-1 Timing of actions during a GPRS Detach Test
Phase
Preparation Preparation Test Clean Up
Action
Duration
< 100 ms
MsgLogTrace
GPRSDetach Test Started Waiting for the DevMan MobFunc service Performing action: Detaching device
Results Analysis
The post processing application NQDI shows the following data on the result analysis tab sheet:
32
33
14
Basic Function
The application will start a dialup session to the network. During this dialup session, the mobile phone will attempt PDP context activation. Application starts a dialogue with the network. Network Client IP address is returned.
Figure 14-1 PDP Activation Test Basic Function Table 14-1 Timing of actions during a PDP Activation Test
Phase
Preparation Test Clean Up
Action
Start test Establishing dial-up connection Waiting for trace/clean up
Duration
< 100 ms 120 s max 1.5* s
MsgLogTrace
PDPActivation Test Started
Results Analysis
The Analysis Window displays the following:
34
35
15
Basic Function
The application will attempt to disconnect from the network by performing a PDP deactivation operation. Network DEACTIV. ACCEPT will be returned, if PDPDeactivation is successful. Client Application attempts a PDPDeactivation.
Figure 15-1 PDP Deactivation Test Basic Function Table 15-1 Timing of actions during a PDP Deactivation Test
Phase
Preparation Test Clean Up
Action
Duration
< 100 ms
MsgLogTrace
PDPActivation Test Started
30 s max 1.5* s
Results Analysis
The Analysis Window display the following:
16
WAP Test
Basic Function
The WAP test starts by connecting to the WAP gateway (redirection is handled as needed). A GET request for the desired URL is, then, sent to the gateway. Once the response has been received, the page is being parsed for any Picture tag (<IMG>). For each such tag found, another request is sent to retrieve the image. The test is finished when the last request has returned. The throughput is calculated with the size of the downloaded page and all images divided by the time between the request of the initial page and the response (last packet) of the last image.
Log WAP Gateway WAP/HTTP Server
START WAP
TRANS.
TRANS. DOCUMENT
WSP Reply
Results Analysis
The post processing application NQDI shows the following data on the result analysis tab sheet:
Table 16-1 WAP Test - Result Analysis
Value
Connect Duration Download Duration Throughput Error Code
Chapter 16 | WAP Test
Description
Connecting time Total time of web page download duration Calculated application throughput value 0 if successful; otherwise the error code
37 CONFIDENTIAL MATERIALS
Value
URL: Host Time out Msg Time Event Name Size
Description
Uniform Resource Locator Name or IP address of the host Configured timeout duration Current timestamp of event message Message event Complete file name File size in [Bytes]
38
17
Basic Function
The message text consists of auto created identifiers and random or user defined text data which can be used by an optional receiver for proper SMS identification. Once the SMS is constructed, it is being sent to the configured SMSC. Receiver Status Report will indicate success or failure. Client Application attempts to send SMS
Figure 17-1 SMSSend Test Diagram
Results Analysis
The post processing application NQDI shows the following data on the result analysis tab sheet:
39
Value
Send Duration End-To-End Delivery Time SMS Size Max Send Time Pause Time Sender Sender Number Receiver Number Error Code
Description
Time to send the text message Duration between sending and receiving a message (Not used in SMS send only tests) Number of characters within the text message Timeout value for sending the message Time between sending of two messages Phone number of sender (Not used in SMS send only tests) Phone number of receiver State message: OK if successful, otherwise the appropriate error text corresponding to the error code.
40
18
Basic Function
This test just waits for a SMS indication and then gets the SMS from the mobile. This SMS will be saved and the identifier used to identify the appropriate SMS send test. Network Client Application attempts to retrieve SMS
Figure 18-1 SMS Receive Test - Basic Function
Results Analysis
The post processing application NQDI shows the following data on the result analysis tab sheet:
Table 18-1 SMS Receive Test - Result Analysis
Value
Send Duration End-To-End Delivery Time SMS Size Max Send Time Pause Time Sender Sender Number Receiver Number Error Code
Description
Time to send the text message Duration between sending and receiving a message (Not used in SMS send/receive only tests) Number of characters within the text message Timeout value for sending the message Time between sending of two messages Phone number of sender Phone number of receiver (Not used in SMS receive only tests) State message: OK if successful, otherwise the appropriate error text corresponding to the error code.
41
42
19
Before you can run an MMS Send test, you need to manually configure the MMS options on your phone.
Phone Configuration
For an MMS test on an Android phone, you need to manually configure an Internet and an Internet + MMS APN on the phone. You also need to disable the Auto-retrieve function for MMS. For a description on how to perform these actions, see the user manual for your phone. Note: Some phone types, such as the HTC sensation, use different APN type name, for example, default,supl instead of Internet and default,supl,mms instead of Internet + MMS. For Nokia phones, you need to connect to the network that you want a test, send an MMS, and then configure the phone to never download an MMS. Note: If this option is not available, you need to configure and select an invalid Access Point on the phone.
Basic Function
The message text consists of auto created identifiers and random text data which can be used by an optional receiver for proper MMS identification. Then, the multimedia content is being added to the message. When the message object is created, a GPRS connection is established and the message is sent to the MMS Center (MMSC). When transmission completed, the dialup connection is being disconnected. Once the SMS is constructed, it is being sent to the configured SMSC. Receiver Client Application attempts to send MMS
Figure 19-1 MMS Send Test Basic Function
Results Analysis
The post processing application NQDI shows the following data on the result analysis tab sheet:
Table 19-1 MMS Send Test - Result Analysis
Value
Service Access duration
Description
Duration for accessing the service
43 CONFIDENTIAL MATERIALS
Value
Transmit Duration Overall Send Duration Error Code
Description
Duration for sending the message Overall duration State message: OK if successful, otherwise the appropriate error text corresponding to the error code.
MMS Size Max Send time Pause Time Sender Sender Number Receiver Number Attachment Gateway Protocol
Size of message Timeout to send message State message: OK if successful, otherwise the appropriate error text corresponding to the error code. Phone number of sender (Not used in MMS send only tests) Phone number of receiver Not used in Diversity Gateway address and port number Protocol used to transfer data (WSP or HTTP)
Current timestamp of message States the sender or receiver side Message payload identifier Message event
44
45
46
47
Network
GPRS_CONNECT
Wsp connect REQ WAP CONNECTED TRANSMISSION STARTED Wsp connect REPLY Send MMS
WTP Invoke (begin of MMS Send Req.) WTP Segmented Invoke WTP Segmented Invoke MMS Send Req (begin of MMS Send
TRANSMISSION FINISHED
MMS m-send-conf
MMm-send-conf
Close RAS
GPRS PDP Deact. Req GPRS PDP Deact. Acc GPRS Det. Req. GPRS Disconnected GPRS Det. Acc
48
20
Before you can run an MMS Send test, you need to manually configure the MMS options on your phone. For more information, see the "Phone Configuration" section on page 43.
Basic Function
This test, first, waits for an MMS notification indication sent via (usually two) SMS. Then, a GPRS and WAP connection is established and the MMS retrieved from the MMS Center. Network Client Application attempts to retrieve MMS
Figure 20-1 MMS Receive Test Basic Function
Results Analysis
The post processing application NQDI shows the following data on the result analysis tab sheet:
Table 20-1 MMS Receive Test - Result Analysis
Value
Service Access duration Receive Duration Overall Receive Duration Notification time
Description
Duration for accessing the service Duration for sending the message Overall duration Time between sent message (sender: transmission finished) and notification indication on receiver side. (Not used in MMS receive only tests) Complete time duration when starting to send message until it is fully received by opposite site. (Not used in MMS receive only tests) State message: OK if successful, otherwise the appropriate error text corresponding to the error code. Size of message Timeout to send message State message: OK if successful, otherwise the appropriate error text corresponding to the error code. Phone number of sender
49 CONFIDENTIAL MATERIALS
Error Code MMS Size Max Send time Pause Time Sender Sender Number
Chapter 20 | MMS Receive Test
Value
Receiver Number Attachment Gateway Protocol
Description
Phone number of receiver (Not used in MMS receive only tests) Not used in Diversity Gateway address and port number Protocol used to transfer data (WSP or HTTP)
Current timestamp of message States the sender or receiver side Message payload identifier Message event
50
51
52
Stack
Mobile Phone
Network
NOTIFICATION RECEIVED
Open RAS
GPRS Attach Req GPRS Attach Acc GPRS PDP Con. Req.
GPRS CONNECT
Wsp connect REQ WAP CONNECTED TRANSMISSION STARTED Wsp connect REPLY Send MMS
MMS Download
TRANSMISSION FINISHED
MMS m-retrieve-conf
MMS m-retrieve-conf
Close RAS
GPRS PDP Deact. Req GPRS PDP Deact. Acc GPRS Det. Req.
GPRS DISCONNECTED
GPRS Disconnected
53
In order to perform data tests effectively, Firewall settings on client and server side must be configured properly. Due to increasing IT security issues it is highly recommended to unblock only the necessary ports and applications used for testing. The following tables give an overview of required protocols and ports used by the data tests.
Table A-1 Port Number Listing
Server TCP
Incoming Outgoing (P)
Client TCP
Incoming (A) Outgoing 21 22 25 53 53 80 110 113 554 1220 (S, T) 8019 random random
FTP SSH SMTP DNS HTTP POP3 Authentication Service RTSP, Control Darwin Admin UDPPlus Configuration
Table A-2 UDP Ports
21 22 25
Server UDP
Incoming Outgoing 53 113 6970..6989 8010 8021 (T) 8023 8010 8021 (T) 8023
Client UDP
Incoming Outgoing
DNS Authentication Service RTSP, Data UDP Test UDPDl Test UDPPlus Test Legend: P A S T
53 113
FTP-aware firewall required for passive mode FTP-aware firewall / NAT required for active mode Configurable during Server startup Configurable in Test configuration
Note: Not all data tests require a previous dialup. Such as GPRS and PDP Tests! The measurement PC dials up to the Internet provider. At this point, the marker GPRS Dial is written to the log file. This marker can be shown in the GSM Analysis window:
The GPRS markers are also visible on the test-specific tab sheet (FTP, UDP, HTTP, Email, WAP).
55
And, last but not least, they are also visible on the Map, if you choose Call Markers.
After the GPRS Dial, the mobile will start with an Attach procedure and a PDP Context Activation process. This can be seen in the GSM Analysis window in the inter-layer table:
56
As soon as the mobile is connected to the Internet, you will get a GPRS Connect marker. If the dial-up fails,
Appendix A | Port Usage and Firewall Settings CONFIDENTIAL MATERIALS 57
you will get a GPRS Connect Failed marker. This means, that the mobile is not connected to the Internet and that the dial-up has to be repeated. You can search for such events, if you select the GPRS Connect Failed markers in the data selection part:
58
The minimum length of the pause between data tests in NQView is 0 seconds. Diversity automatically adds the waiting for trace messages time to this pause to create the effective pause time. If you want to modify the waiting for trace messages time for the following data test types, you need to create a Windows Registry key: HTTP Transfer HTTP Browser FTP Capacity IPERF Ping Trace UDP Email MMS.
2. Right-click Engine, point to New, click String Value, and type WaitForEtherealMessages as the name. 3. Right-click the new WaitForEtherealMessages value and click Modify. 4. Type the length of the pause in milliseconds and click OK.
59