Beyond MOS Ensuring VoIP Service Quality

June 2010

TABLE OF CONTENTS

A perceived reduction in the quality of data services causes customers to call for support, but a perceived reduction in the quality of voice services causes customers to churn - Tier 1 Communication Service Provider

BEYOND MOS ENSURING VOIP SERVICE QUALITY

INTRODUCTION 1 In the current telecommunication services environment, competitive forces and complex integrated service offerings are requiring communication service providers (CSPs) to alter their business strategies to focus on the customer, not just the network. As multiple services are delivered over a single IP infrastructure, it is no longer valid to assume that voice call quality experienced by an individual user is good as long as there are no network faults. In order to focus efforts on determining the quality of VoIP services delivered to each customer, CSPs are exploring ways to understand VoIP service quality for each user, each time they make a call. A perceived reduction in the quality of data services causes customers to call for support, but a perceived reduction in the quality of voice services causes customers to churn - Tier 1 Communication Service Provider The goal for many CSPs is to transition to an all-IP infrastructure from the access network to the core. The value of using the IP network for voice is the streamlining of infrastructure and decreased complexity that comes from implementing a single broadband network. But unlike data traffic, voice traffic is live, extremely time sensitive, and unforgiving of network congestion. The systems and tools used to manage existing IP data networks need to be augmented with in-call monitoring to ensure that the quality of voice calls remains high. The increasing number of VoIP calls and the complexity of managing distributed, multi-vendor VoIP platforms make it difficult for CSPs to deliver highly reliable, readily available voice service to potentially millions of users. Without a complete toolset to ensure performance and quality, CSPs may learn of problems but are left in the dark as to how to solve them. Current measures of VoIP service quality, including Mean Opinion Scores (MOS), are not frequent or detailed enough to allow operators to understand the nature of the
1 Please note that the insights and opinions expressed in this assessment are those of Stratecast and have been developed through the Stratecast research and analysis process. These expressed insights and opinions do not necessarily reflect the views of the company executives interviewed.

2010 Stratecast. All Rights Reserved.

VOIPFUTURE, June, 2010

impairment in a VoIP call or determine a solution. VoIP call quality is impacted by the IP endpoints and the IP network. As a result, it is not enough to monitor the health and status of the IP network and measure the quality at the conclusion of a VoIP call as many CSPs have found. The in-call quality of each VoIP call stream must be continuously monitored for the duration of the call and the source(s) of impairments discovered and remedied. Once impaired, VoIP traffic cannot be fixed, so the problems creating the impairments must be solved to ensure the quality of future VoIP calls. This paper will describe the problem of measuring voice call quality across an IP network, what VoIP call quality measurement solutions should include, and highlight examples of how one vendorVOIPFUTUREis providing a solution that addresses those challenges.

THE CASE FOR MONITORING VOIP CONNECTION QUALITY CSPs are in the position of having to differentiate themselves with the level and quality of both the services and customer experiences that they provide. Regardless of the source of the content or the manufacturer of the device, the customer will look to the CSP to ensure that services, content, and devices are integrated and that everything works properly. Many enterprises find themselves in a similar position as they take over CSP responsibilities for delivering voice services across their internal data networks. Any degradation in the performance of the underlying network, whether it belongs to the CSP or the enterprise, will affect call quality. Voice users are accustomed to nearly perfect levels of service quality, and anything less will affect customer satisfaction. Delivering voice services over IP networks obviously presents unique challenges that must be overcome in order to transition customers to VoIP services and reduce operational complexity and cost. Real-time Transport Protocol (RTP) is the standard protocol for the delivery of highly time-sensitive audio and video over IP networks. In order to understand what is necessary to ensure high service quality for VoIP, one must first understand how using RTP to transport voice is different from conventional TDM voice transport. Not a Nailed-Up Connection In traditional TDM voice networks, when a voice call is placed, a circuit is nailed up between the two parties and remains that way for the duration of the call. But an IP network is dynamic. Even once a voice call is established; the flow of speech is continuously disassembled into multiple RTP packets that are reassembled at the receiver. Poor voice quality with broken speech or dropped words typically results from disruptions to the RTP packet flow. Thus, high quality VoIP connections demand a constant and steady RTP packet stream. To this end, a variety of quality of service (QoS) enabling mechanisms for IP networks have been developed; yet even if these are implemented, VoIP call quality is far more likely to vary than TDM call quality. In addition to sharing the IP link with all other communication and data services, the flow of RTP packets is frequently exposed to variations and distortions that have a negative
3

VOIPFUTURE, June, 2010

2010 Stratecast. All Rights Reserved.

impact on voice quality. The network equipment, user devices, configurations and changing traffic load comprise highly dynamic sources of such impairments. In order to ensure high quality VoIP calls, it is important for VoIP monitoring systems to be able to continuously monitor quality metrics for the duration of every call. The following metrics are typically considered key to voice quality monitoring: Delay Delay or latency comprises several elements. The transit time through the IP network and associated access links; delay within the endpoints due to jitter buffering; encoding, and decoding delays; and potential delays within non-IP portions of the network can all lead to conversational delays. Difficulties arise when the added delay of the VoIP service interferes with the natural interaction between the participants on the call. Jitter Identifying the sources of delay variations will ultimately improve end-toend-delay. The variation in packet delay is referred to as jitter and is compensated for by buffers at the receiver of an RTP stream, which adds to the delay. The ability to control jitter greatly helps to reduce end-to-end delay and thus has substantial impact on voice call quality. Real-Time Control Protocol (RTCP) is used with RTP for collection of jitter and other QoS data; however RTCP measurements identify that impairments are occurring, but do not identify the sources of the problem. Packet Loss Packet loss is a serious problem in real-time applications like VoIP. Since RTP is used to carry voice traffic, lost packets are not retransmitted so the end device has to compensate to give the impression of a continuous signal. However, packet loss tends to occur in bursts and, as such, compensation at the receiver becomes ineffective. Packet loss concealment methods work provided there are enough good packets that are received in-between. Persistence of hearing allows for concealment of a missing packet here and there, but losing a burst of voice packets will create a gap or lost words. As such, these metrics only describe the transport characteristics of the underlying IP network and its componentsthe relation between network performance metrics and the subjective opinion of a voice service user is everything but obvious. One way to provide an estimate for the user experience is to feed these measurements and information about the codec into a model defined by ITU-standard G.107. The model essentially yields a transmission-rating factor R which can be converted to a MOS-score. Such measures of connection quality are typically calculated per call. However, in contrast to circuit switched networks which nail up a dedicated voice connection, voice quality in IP networks varies over time. A calls voice quality may be perfect for minutes and then suddenly degrade, e.g. because of network overload. Thus, analyzing and improving VoIP call quality requires continuous in-call monitoring of all voice streams.

VOIPFUTURE, June, 2010

2010 Stratecast. All Rights Reserved.

Determining and Improving Vo IP Call Quality The problem is that QoS measurements and MOS, while valuable, are not being calculated frequently enough and do not provide sufficient detail to identify VoIP quality problems and their sources. An additional problem may be introduced by the way the QoS data is collected. Active probing simulates VoIP traffic and compares the transmitted with the received speech signal. This approach provides good samples of end -to-end quality; however it does not include the end user device itself, and because an active probe generates traffic, it is not measuring actual calls. The test call may be impairment free, while the next customer call may incur degradation. Furthermore, as measurements are carried out end-to-end, little can be said about the root cause of potential problems. Active probing is thus effective for pre-deployment scenarios and determining that an IP network is able to carry VoIP traffic. In contrast, with permanent passive probing, actual VoIP calls are monitored and a model may be used to estimate VoIP call quality using information from both the VoIP signaling (SIP) and media (RTP) streams captured at the monitor point. Passive probing can be done by integrating the measurement algorithm into the end-points of RTP streams (e.g. IP phones and media gateways), or at a mid-point between the VoIP sender and the receiver as shown in Figure 1. Figure 1: Monitoring at Peering and Aggregation Points

Source: VOIPFUTURE

VOIPFUTURE, June, 2010

2010 Stratecast. All Rights Reserved.

Beyond measurements that describe perceived speech quality, it is important to also understand what caused the impairments and where along the IP transmission path the impairments occurred. By identifying root causes, any problems can be quickly resolved and VoIP quality improved. Simple averaging of measurements and MOS calculations per call neither provide sufficiently detailed views into the transport layer, nor readily available data for root cause analysis and traffic optimization. These additional benefits require a new approach to voice quality monitoring. Yet, any viable approach must ensure that the requirements of large CSPs are met. Making VoIP Call Quality Monitoring Carrier-Class For CSPs, managing an extended VoIP network requires the collection and processing of large amounts of data from numerous, multi-vendor sources, as well as the ability to rapidly deliver actionable results to network managers. Managing VoIP call quality for CSPs must therefore meet the stringent requirements that CSPs have for all of their OSS/BSS solutions. Specifically; Scalability Managing an extended VoIP network requires that both the VoIP platforms and the management platforms scale to support potentially millions of users and hundreds of thousands of simultaneous calls. Existing VoIP connection monitoring platforms do not scale well to this volume and CSPs are challenged to accommodate both the number of users and the number of calls. Reach To ensure end-to-end reliability, VoIP call monitoring must include the user devices as well as any intermediate networks. Impairments from underperforming or misconfigured end devices are difficult to identify but can cause significant delays. Likewise, the ability to monitor and measure supplier and interconnection partner performance are essential to guarantee agreed service levels for each customer. Availability As VoIP connection quality data is collected and correlated, it must be made available to operations, management, engineering, and IT administration personnel. Whether delivering real time status and quality data for root cause analysis or the trend data required for service level monitoring, each role requires a different view of the data in a useable format. Whether displayed directly to users or published to external OSS/BSS applications, VoIP call connection quality data is important across the entire CSP business. Without the ability to scale to monitor millions of connections, remain active during all calls, and continuously deliver results VoIP call quality monitoring solutions will not measure up.

VOIPFUTURE, June, 2010

2010 Stratecast. All Rights Reserved.

BEYOND AVERAGE MOS Voice service quality issues need real-time attention. The ability to evaluate IP packets at line rate and provide in-call quality information for every RTP stream is needed to compile a detailed view of each call. That data enables operators to determine the quality of the delivered VoIP service as well as the overall status of the IP network and VoIP infrastructure. The actual disposition of each customer call is an important piece in determining the customer experience and one that has been missing until now. As described, delay, jitter, and packet loss are the typical VoIP call quality measures across an IP transport network. If the derived MOS scores per call are good, thats a start, but average MOS scores do not entirely reveal VoIP service performance. As shown in Figure 2, without measuring VoIP call quality metrics at regular intervals throughout the duration of the call, there is no insight into the dynamics that are affecting the customer experience. Figure 2: Simplifyi ng Average vs. Reality

Source: VOIPFUTURE

The determination of single delay, jitter, packet loss and MOS value per call limits the ability of CSPs to rapidly identify VoIP impairments being reported by end-customers and interconnection partners. Those measurements can only act as a problem indicator when the user is already experiencing poor speech quality. By measuring quality criteria (including MOS) in call at regular 5-10 second intervals as suggested by the experts of the TM Forum, problems are detected and the distribution of the data can reveal the type of problem and its cause. Alarms can then be generated that enable CSPs to fix the
VOIPFUTURE, June, 2010 2010 Stratecast. All Rights Reserved. 7

problem before it becomes apparent to users. The following describes some important metrics and indicators that can be obtained from permanent passive in-call measurements. Jitter Sender jitter, as an example, is not introduced by the network but rather is caused by the senders device. If the sender is unable to maintain a constant signal at the desired sample rate, the packets are transmitted at two distinct and nearly identical delay variation times. In-call measurements discover this anomaly and allow the CSP to diagnose a sender jitter fault. Although sender jitter is not created in the network, it can trigger a response from dynamic jitter buffers and ultimately impact call quality. Recognizing that a particular handset or version of handset regularly creates sender jitter gives CSPs the information they need to confront the manufacturer and request a change. Figure 3 RTP sender is unable to maintain constant packet rate of 20 ms

Source: VOIPFUTURE

Packet Loss A smart packet loss detection algorithm not only allows detection of the number of lost packets but also provides an indication as to whether a lost packet is part of a burst. The combination of a representation of packet loss bursts and the correlating information as to the number of good packets received between loss events help the CSP determine the impact on speech quality. By collecting this information in real-time and presenting a histogram of both burst packet loss and burst packet receipt (good packets), a high level of detail about the length and frequency of packet loss events can be obtained. The monitoring system can then determine if the loss is tolerable or indicate that packet loss is unacceptable. Network Overload Further, it is important to know if the packet loss is related to a network overload condition, which is the most common cause in access networks. If packet loss due to network congestion can be ruled out, then the amount of time spent to localize the
VOIPFUTURE, June, 2010 2010 Stratecast. All Rights Reserved. 8

problem is significantly reduced and technicians can look for faulty hardware or configuration problems. In contrast, overload conditions indicate that the network resources are overbooked, which must be treated differently, e.g. by adding additional capacity. A special symptom of VoIP call degradation on overloaded network segments is that the inter-arrival time of the packets will vary greatly due to buffering and then bursting of the packets. There are strong variations of the inter-arrival times and when that is the case, the intended packet sequence and inter-arrival timing cannot be maintained by the network. This is shown by both very small and very large inter-arrival times between packets. Packets lost during such sequences indicate that packet buffers are being overrun and packets are being discarded. Such conditions can be detected with pattern matching technology when measurements are being made at high frequency. Additionally, detailed measurements can be used to identify network devices where the packet buffer has been configured too large for VoIP. Creating large packet buffers is a common way to reduce packet loss that results from high jitter; however large buffers are detrimental to real-time applications like VoIP due to the additional delay that is created. With this information it is possible to detect network configuration problems, which often occur in environments where auto-QoS is implemented. In an auto-QoS environment, the network equipment may change the priority of a packet to meet network performance policies, however switching an RTP VoIP packet to a lower class of service is unacceptable and risks introducing jitter or packet loss. Policy and Standard Conformance A number of problems are not related to the transport performance of a network itself, but to the VoIP traffics lack of conformance to network policies and standards. For example, changes to the codecs used in the network are often imperceptible to VoIP users and seldom lead to call quality problems. One exception is when the new codec is not supported by the end-user device. This should not happen as e.g. the SIP standard prescribes that endpoints exchange their capabilities; in practice, however, arbitrary incall codec changes are not uncommon. Similarly, if RTP streams entering a network do not use the agreed upon DSCP priority marking, then voice packets will likely be treated as best effort traffic. In moderately loaded networks, such misconfigurations may remain. However, if the network load rises then the non-conformant packets will not be treated with appropriate priority and may experience increasingly high jitter and packet loss. A final example of policy conformance problems is given by RTP streams which use voice codecs that do not comply with the policies of a network. For reasons of bandwidth efficiency, network policies may prescribe the use of high-compression codecs, such as G.729. Conformance to such network policies can only be assured if permanent passive monitoring systems are used to inspect every single packet of all RTP streams crossing the network boundaries.

VOIPFUTURE, June, 2010

2010 Stratecast. All Rights Reserved.

NEW PROBLEMS NEW SOLUTIONS Founded in Hamburg, Germany in 2007, VOIPFUTURE has developed a Voice Service Monitoring solution designed to monitor the connection quality of IP voice services. VOIPFUTURE has developed a unique technology for evaluating the quality and performance of voice services in IP networks and performing more frequent and robust MOS calculations. Based on its permanent passive measurements, the VOIPFUTURE solution can detect sender jitter, network overload situations, whether packet loss is tolerable for a given codec and many other sources of voice quality degradation. It enables CSPs to better monitor and control the user experience while constantly improving network performance to better accommodate voice services. The carrier-grade VOIPFUTURE solution includes passive monitoring probes combined with an intelligent management platform that implements open interfaces to distribute VoIP monitoring data and performance metrics to other OSS/BSS. Passive probes are deployed to monitor and analyze network transport and device media processing for quality impairments. Each probe is able to monitor up to four links providing full line rate performance for 1 Gbit/s links. The VOIPFUTURE algorithm determines root causes, provides diagnostics and statistics based on its comprehensive approach to RTP packet flow analysis. Quality Data Records (QDRs) containing over 400 metric parameters are available for each five second segment of every individual call stream to track changes during the course of a call. The detailed measurements of VOIPFUTUREs solution not only allow to pinpoint sources of voice quality degradation, they also provide data for highly accurate in-call MOS calculations. The solution can thus compute MOS values for every 5 seconds of a call and aggregate these, e.g. to a minute-level. Being able to go beyond per-call MOS in this manner has two significant advantages: 1. MOS values for short fixed-length intervals provide for a more meaningful voice quality assessment. Averaging MOS scores for a complete call frequently overestimates the quality of long calls. Severe quality problems will often lead one party to hang up. However, if for example only the last 10 seconds of a 5 minute call were severely affected, then the overall call may still be rated with an excellent MOS. As a result, the CSP will be lulled into a false sense of security. MOS values for short fixed-length time intervals allow for definition of SLAs for VoIP services in a very natural manner. For example, it can be agreed not to charge for call minutes when the quality is less than promised. The verification of such SLA agreements can be done in real-time.

2.

By integrating the VOIPFUTURE solution into network and service management systems, immediate responses to degradation can be made, e.g. permanently monitoring peering partners and automatically changing to better routes if need arises. The metric details of VOIPFUTUREs QDRs are grouped to indicate specific values for: Service Assurance and customer experience prediction
VOIPFUTURE, June, 2010 2010 Stratecast. All Rights Reserved. 10

Service Level Agreement KPIs for customers and suppliers Transport performance and analysis Network policy conformance, e.g. regarding traffic prioritization VoIP end-point standard conformance VoIP service quality monitoring is critical to understanding the customers experience as he/she makes VoIP calls. However, most calls do not originate and terminate on the same network so it becomes necessary to monitor inter-provider connections to determine end-to-end quality. As service level agreements (SLAs) become more complex and specific, additional data needs to be collected and correlated to ensure that agreedto service levels are being met. Because the network connections are not nailed up, RTP data can be used by network engineers to optimize the design and configuration of the network and maintain overall service quality. As failures are discovered, the ability to determine the root cause of a problem quickly and correctly ensures that customer complaints are resolved and the network continues to perform at optimum levels. Figure 3 illustrates how VOIPFUTUREs timely and accurate RTP monitoring can affect multiple business functions. Figure 3: Applicatio ns and Use Cases of RTP Monitoring

Source: VOIPFUTURE

The VOIPFUTURE solution enables CSPs to detect and correlate impairments originating from network equipment or end user devices, providing a complete end-to-end view of each call. Detailed connection monitoring of VoIP calls enables CSPs to offer premium voice services and meet stringent voice SLAs. By reducing the time and effort required

VOIPFUTURE, June, 2010

2010 Stratecast. All Rights Reserved.

11

to solve VoIP quality problems, overall service is improved and that improves customer satisfaction and reduces churn while improving network performance and utilization. VoIP Service Quality is a Differ entiator fo r KielNET As a competitive City Carrier KielNet serves voice customers in the Kieler Forde region of northern Germany. As KielNET migrates to an all-IP network, customers are being transitioned from traditional voice services to VoIP. Introducing new VoIP services was accompanied by challenges in preserving service quality and ensuring that support groups would have the tools they needed to verify and localize call quality degradation, while providing the best possible support to customers. In the KielNET heterogeneous network, active probing could show degradation in voice quality, but did not help to identify the source of the problem or resolve it. Trial-and-error troubleshooting methods ultimately helped to localize the problem, but also required tedious and timeconsuming engineering efforts. KielNET turned to VOIPFUTURE to implement continuous, real-time monitoring of VoIP traffic to qualify customer complaints of poor service quality and to rapidly identify and fix the problems. The VOIPFUTURE probes implement a unique quality evaluation algorithm that rapidly and reliably detects VoIP call quality degradation and localizes its root cause on IP/IMS networks. The VOIPFUTURE solution operates in heterogeneous IP environments and is scalable on standard IT hardware. In addition to user-friendly access to the critical information used to identify, analyze, and resolve VoIP call quality problems, KielNET also reduces the time required to resolve quality issues caused by faulty firmware upgrades or the addition of new network components. CONCLUSIONS As VoIP traffic continues to climb, CSPs are faced with the challenge of ensuring VoIP call quality. Monitoring of real-time VoIP services is complex, and existing IP network management solutions need to be augmented with more comprehensive and sophisticated data collection, correlation, and analysis capabilities. These capabilities need to consider the specific characteristics of voice services by operating with dedicated metrics that provide insight into the user experience. Policy and standard conformance issues also need to be considered, to ensure smooth network operations and pro-active handling of problems. An RTP monitoring solution that provides the diagnostics to assess quality and localize the root causes of impairments will subsequently improve voice quality and network performance. By continuously analyzing the RTP media flow in real time, CSPs can diagnose issues such as limited bandwidth or improperly functioning network components. When you feel sick, you dont need the doctor to tell you so. You want to know what is wrong and what to do about it. The VOIPFUTURE solution described in this paper addresses the complexity of real-time monitoring of VoIP call quality and delivers actionable results.

VOIPFUTURE, June, 2010

2010 Stratecast. All Rights Reserved.

12

Silicon Valley 331 E. Evelyn Ave. Suite 100 Mountain View, CA 94041 Tel 650.475.4500 Fax 650.475.1570 San Antonio 7550 West Interstate 10, Suite 400, San Antonio, Texas 78229-5616 Tel 210.348.1000 Fax 210.348.1003

CONTACT US

London 4, Grosvenor Gardens, London SWIW ODH,UK Tel 44(0)20 7730 3438 Fax 44(0)20 7730 3343

Beijing Bengaluru Bogot Buenos Aires Cape Town Chennai Delhi Dubai Frankfurt Kolkata Kuala Lumpur London Manhattan Melbourne Mexico City Milan Mumbai Oxford Palo Alto Paris Rockville Centre San Antonio So Paulo Seoul Shanghai Singapore Sydney Tel Aviv Tokyo Toronto Warsaw

877.GoFrost myfrost@frost.com http://www.frost.com

ABOUT STRATECAST Stratecast assists clients in achieving their strategic and growth objectives by providing critical, objective and accurate strategic insight on the global communications industry. As a division of Frost & Sullivan, Stratecasts strategic consulting and analysis services complement Frost & Sullivan's Market Engineering and Growth Partnership services. Stratecast's product line includes subscription-based recurring analysis programs focused on Business Communication Services (BCS), Consumer Communication Services (CCS), Communications Infrastructure and Convergence (CIC), OSS and BSS Global Competitive Strategies (OSSCS), and our weekly opinion editorial, Stratecast Perspectives and Insight for Executives (SPIE). Stratecast also produces research modules focused on a single research theme or technology area such as IMS and Service Delivery Platforms (IMS&SDP), Managed and Professional Services (M&PS), Mobility and Wireless (M&W), Multi-Channel Video Programming Distribution (MVPD), and Secure Networking (SN). Custom consulting engagements are available. Contact your Stratecast Account Executive for advice on the best collection of services for your growth needs. ABOUT FROST & SULLIVAN Frost & Sullivan, the Growth Partnership Company, enables clients to accelerate growth and achieve bestin-class positions in growth, innovation and leadership. The company's Growth Partnership Service provides the CEO and the CEO's Growth Team with disciplined research and best-practice models to drive the generation, evaluation, and implementation of powerful growth strategies. Frost & Sullivan leverages 49 years of experience in partnering with Global 1000 companies, emerging businesses and the investment community from more than 40 offices on six continents. To join our Growth Partnership, please visit http://www.frost.com.