WHITE PAPER Introduction of New Voice Quality KPI The

WHITE PAPER Introduction of New Voice Quality KPI The ITU-T G.107 E-Model Measures Impairments That Are External To The Network EXECUTIVE SUMMARY • • • • • • Service providers today often wonder how they can get a reasonable Mean Opinion Score (MOS) from their network performance benchmarking, yet see little or no corresponding decrease in churn rates or an increase in customer satisfaction. Approximately 20-30 percent of the problems that degrade voice quality for mobile and VoIP users occur from handsets, earpieces, noisy streets, new mobile phone technologies and a myriad of other impairments that are external to the network. The importance of measuring the affect of external impairments on voice quality has led to the adoption of the ITU-T G.107 E-Model as the new voice quality KPI. The E-Model is enabling many of the world’s mobile and VoIP service providers to establish new Key Performance Indicators (KPI) that compliment today’s testing methodologies, and allow network planners to accurately measure voice quality as it’s actually delivered to callers. The deployment of Ditech Networks Experience Intelligence (EXi) solution, which utilizes the E-Model KPI to non-intrusively measure live subscriber calls as they are occurring, has been successful in providing service providers new insight into voice quality as it’s actually experienced by subscribers. The combination of the E-Model and new solutions that utilize its methodology are allowing network planners and operators to tune mobile and VoIP voice service to the demands of the market, instead of overbuilding the network. INTRODUCTION Measuring voice quality has caused both debate and the creation of multiple technologies that have attempted to capture and score call clarity. Despite methodologies established over the years by the communications industry’s standards organizations, measuring voice quality has remained an inexact science. For example, service providers often wonder how they can get a reasonable Mean Opinion Score (MOS) from their network performance benchmarking, yet see little or no corresponding decrease in churn rates or an increase in customer satisfaction. In other words, the current way of measuring voice quality indicates that customers should be satisfied, yet they continue leaving in record numbers. Network planners and engineers have been challenged by this incongruence for years. As shown in Figure 2, networks generally are designed to provide voice quality that results in a score that’s above a certain threshold. But voice quality as it is actually experienced by a caller typically scores significantly lower than what was planned. Approximately 20-30 percent of the problems that degrade voice quality for mobile and VoIP users occur from handsets, earpieces, noisy streets, new handset technologies and a myriad of other impairments that are external to the network. The importance of measuring the affect of external impairments on voice quality has led to the adoption of a new methodology, created by the International Telecommunications Union (ITU) and being implemented now by many of the world’s service providers. Named the ITU-T G.107 E-Model, this model takes into account not just the affect of network infrastructure on voice quality, like transmission and speech compression, but includes external impairments like echo, delay times, speech level and even background noise from the caller’s environment. This model has enabled many of the world’s mobile and VoIP service providers to establish new Key Performance Indicators (KPI) that compliment existing methodologies, and allow network planners to accurately measure voice quality as it’s actually delivered to callers. . Page 2 of 13 Figure 1: Chart shows the source of the most common voice quality impairments that impact more than half of all mobile calls made worldwide. Figure 2: Network operators design and deploy systems that are engineered to deliver specific levels of voice quality to subscribers, commonly measured using MOS. However, network performance benchmarking doesn’t measure voice quality impairments that are external to the network. The result is a significant difference in voice quality that was planned, and voice quality that is actually 1 experienced by subscribers. THE LIMITATIONS OF CURRENT VOICE QUALITY KPIS This section describes why current voice quality KPIs are unable to measure voice quality as it’s actually experienced by subscribers. Also discussed is the impact of evolving network architecture on voice quality KPIs. The most common voice quality KPIs are based the ITU’s Telecommunication Standardization Sector (ITU-T) Recommendation P.800 and P.862. These standards were designed many years ago to address the specific voice 1 Data is derived from voice quality audits conducted by Ditech Networks at nine service providers around the world, from more than 600,000,000 calls. . Page 3 of 13 quality issues outlined in Figure 1. These models are used to derive voice quality measurements from subjective and objective tests. Standard ITU-T P.800 ITU-T P.862 Summary Subjective Voice Quality Test Objective Voice Quality Test-PESQ Environment Device Level RF All All Network Codec Codec Other Network Noise Level Figure 3: These are the most common ITU-T standards for voice quality testing. Subjective Method: ITU-T P.800 The communications industry has often used people to evaluate voice quality. A listener will hear speech recordings that are transmitted through a network, and will identify and evaluate impairments. The specifications for these speech recordings are described in ITU-T P.800, which gives the industry a set of standardized audio speech files that can be used to evaluate voice quality problems. The industry generally recognizes two problems with this approach: 1. 2. One person’s way of evaluating a speech file may differ from another person, and when testing is done across multiple companies and networks, the results can very widely. Hiring and maintaining organizations for large-scale, subjective testing and continuous monitoring has proven to be enormously costly. Objective Method: ITU-T P.862(PESQ) To avoid the two issues described above, the ITU created a model for mechanically measuring voice quality. This method uses technology to measure, for example, the radio quality of GSM networks. Audio recordings of a person talking distinctly, called “clean” speech files, are transmitted through the network to quantify how codec type and frame loss caused by poor coverage, call handovers, and interference impact voice quality. This testing is done by equipment that is installed in a van, and driven around specific geographic areas to measure how the network is performing. These drive tests primarily use the PESQ algorithm to measure voice quality. Other drive test tools are based on SwissQual’s SQUAD-LQ proprietary algorithm. However, PESQ/SQAD-LQ is only accurate for a limited range of impairments, none of which impact the speech signal itself, like background noise, echo, or speech level issues. Additionally, PESQ/SQAD-LQ only collect data during times when there is an active speech signal, and ignores the pauses and silences that naturally occur during normal mobile and VoIP calls. The two Figures below are from the ITU-T P.862 Recommendation and identify the capabilities and limitations of PESQ. PESQ has demonstrated acceptable accuracy for the applications listed in Figure 4. PESQ is generally known to provide inaccurate predictions and measurements when used in conjunction with the variables in Figure 5. Codec Evaluation Codec Selection Live network testing using digital or analog connection to the network Testing of emulated and prototype networks Figure 4: Chart shows the variables that PESQ measures with acceptable accuracy (ITU-T P.862). Test Factors Listening levels (See note below) Loudness Loss Effect of delay in conversational tests Talker echo Sidetone Coding Technologies Replacement of continuous sections of speech making up more than 25 percent of active speech by silence (extreme temporal clipping) Applications In-service non-intrusive measurement device Two-way communication performance . Page 4 of 13 Figure 5: PESQ is known to provide inaccurate predictions when used to measure the variables in the chart above. PESQ assumes a standard listening level of 79dB SPL and compensates for non-optimum signal levels in the input files. The subjective effect of deviation from optimum listening level is therefore not taken into account. The objective method and its reliance on clean speech files has proven successful at measuring the impact of RFrelated impairments in the network, but has been unable to capture and quantify two of the most significant and pervasive issues that affect voice quality: 1. 2. The impact of impairments that are external to the network like ambient noise and mismatched speech levels. The inability to measure conversational voice quality on live subscriber calls, which is significantly influenced by acoustic echo and transmission delay. THE EFFECT OF EVOLVING NETWORK ARCHITECTURE Dynamic growth in the mobile and VoIP industries has resulted in continuously evolving network architectures, all of which are heading towards convergence and a commonly shared IP infrastructure. This convergence is creating the exact opposite of how most of the world’s networks are operated today. Mobile carriers, for example, have relied on legacy TDM networks, which are largely “private” networks where the carrier can control and manage the transmission of speech signals among users. With IP networks, speech packets are fed by the carrier into a packet network, a veritable “wild, wild west” of network equipment and topologies that are used to move speech among users. Yet despite all of the variables in the evolving core network, callers expect uniform and consistent voice quality from their service provider. The current voice quality KPIs described in the previous section cannot be used effectively when the core voice network is based on IP, and when subscribers can access the network through a wide variety of technologies. Transmitting speech through the IP core results in multiple voice quality problems, including longer delay times, packet loss and jitter. All these factors adversely affect overall voice quality and customer satisfaction. Figure 6: The worldwide communications industry needs a common voice quality KPI that functions effectively across any network type. To provide consistent quality of service to mobile and VoIP callers, as well as to adequately engineer continuously evolving networks, it is imperative to have a common set of KPIs. Voice quality KPIs should be relevant not just in today’s networks, but also in any kind of future network. . Page 5 of 13 THE NEW VOICE QUALITY KPI The limitations and future compatibility issues of existing voice quality measurements, along with the rapid evolution of network topologies, provide insight into the kind of characteristics required for a new approach to voice quality KPIs. The communications industry now is coalescing around a set of basic requirements for adopting a new approach: 1. The ability to monitor and measure voice quality on live mobile and VoIP calls originating from any location, not just pre-recorded audio files transmitted during drive tests. 2. The ability to measure impairments that are external to the network, like background noise, acoustic echo, delay and mismatched speech levels. 3. A KPI that can be used across any global network architecture. 4. Worldwide adoption and support by the ITU-T. The following section defines the industry’s new KPI based on above requirements. The ITU-T G.107 E-Model The need to establish a method for adopting new KPIs that compliment existing methodologies actually goes back to 1998. The ITU-T during that year defined what it called the “E-Model” in Recommendation G.107. Since then, this standard has been continuously revised and modified to support the rapidly growing interest in implementing this methodology. G.107 Recommendation describes a computational model, widely known as the E-model, for assessing the combined effects of variations in several transmission parameters that affect conversational voice quality. Conversational voice quality, in this context, refers to characteristics that affect speech transmission, like codec type, frame loss, long transmission times, echoes, background noise and speech level. This makes the E-Model a valuable complement to drive test and switch tools that mainly measure listening quality as a function of clean speech files. The E-model is a non-intrusive method for measuring overall voice quality, and can be used to monitor and measure the subscriber experience on all calls in the network. The E-model complements the intrusive tools commonly used in drive tests, such as PESQ and TEMS, which measure the characteristics of a limited set of test calls in restricted conditions. The R-Factor The primary output from the E-model is the “Transmission Rating Factor,” commonly called the R-Factor. The RFactor is a numerical measure of voice quality on a scale of 0 to 100, with 100 being a crystal clear mobile or VoIP call with no impairments. It can also be converted to a MOS value, which provides an effective estimate of how subscribers perceive voice quality. The R-Factor is composed of: R = Ro - Is - Id – Ie.eff + A Where: • “Ro” is a base factor determined from circuit and room noise levels, signal level, loudness and related factors, and is representative of the signal-to-noise ratio. • • “Is” represents signal impairments occurring simultaneously with speech, including loudness, quantizing (CODEC) distortions and non-optimum side-tones. “Id” represents impairments that are delayed with respect to speech; including echo and conversational difficulties due to delay. . Page 6 of 13 • • “Ie.eff” is the “equipment impairment factor” and represents impairments caused by low bit-rate codecs. It also includes impairment due to packet-loss. “A” is the “advantage factor” that allows for compensation of impairment factors when there are other advantages of access to the user. For example, the advantage of mobility in wireless networks. Call Quality Analysis Using R-Factor The range of R values and their correlation to subscriber perceptions is carefully defined in the ITU-T recommendation G.109 “Definition of categories of speech transmission quality.” R-value range 90≤ R <100 80≤ R < 90 70≤ R < 80 60≤ R < 70 50≤ R < 60 User Satisfaction Very Satisfied Satisfied Some users dissatisfied Many users dissatisfied Nearly all users dissatisfied Figure 7a: Measurement of user satisfaction using R-Factor from ITU-T G.109. R-Factor values less than 50 are not recommended by the industry, and commonly result in churn. Figure 7a provides invaluable insight into user satisfaction based on the R value, and can be used to engineer voice quality in the network. It is important to note that the ITU has defined an R value of 50 as being the minimum needed for a mobile or VoIP network. Networks delivering the equivalent of R=50 or lower have proven to cause subscriber dissatisfaction that is so strong that callers either terminate their service, or reduce their usage of the service. The communications industry calls this effect “network-induced churn.” Based on extensive studies by major carriers and standards organizations, the ITU has published papers on how voice quality affects consumer behavior. This topic is discussed in the section “Consumer Behavior as a Function of R-Factor”. Although the R-Factor is increasingly being used by mobile and VoIP carriers to better understand delivered voice quality, both industries also rely on MOS values, which is a widely used method of scoring subscriber perceptions of voice quality. To help the industry understand the relationship between R-Factor and MOS, the ITU-T G.107 Recommendation defines this association, as shown in Figure 7b. . Page 7 of 13 Figure 7b: Chart shows MOS as a function of the R-Factor. USING THE E-MODEL AS KPI The E-Model has several advantages over traditional voice quality KPIs: 1. 2. 3. 4. For the first time, the ITU-T has been able to show the effect of voice quality on consumer behavior using the E-Model. The effect of different types of codecs on speech can be understood from implementing the E-Model. The E-Model can be used to capture data on live call quality as it’s actually delivered and experienced by mobile and VoIP subscribers. The model can be utilized to track the network’s performance against network-induced churn. The following subsections give in-depth coverage on these topics. Consumer Behavior and The E-Model It is commonly believed in the mobile and VoIP industries that superior voice quality leads to higher customer satisfaction. Several carriers around the world now have begun to differentiate themselves in the marketplace based on voice quality. But there remains little awareness in both industries on how voice quality materially affects consumer behavior, and exactly what constitutes “optimum” voice quality as perceived by the caller. In the G.107 Recommendation, the ITU-T puts measurable parameters around how consumers react to voice quality. Consumer behavior is plotted against the R-Factor to show the consumer response, for example, to “Good or Better” and “Poor or Worse” voice quality. The blue curve in Figure 8 2 shows the percentage of users who would be expected to terminate a call early based on poor voice quality. As can be seen in the chart, 80 percent of users would be expected to terminate a call early if the R Factor is 50 or below. 2 J. J. Gruber and G. Williams, “Transmission Performance of Evolving Telecommunications Networks,” Artech House, ISBN 0-89006-591 -8, 1992, p. 34. . Page 8 of 13 This study becomes more relevant for network planners when studied along with codec performance, discussed in the next section. Figure 8: The ITU has mapped consumer behavior to the R-Factor. For reference, different codec types have been added to show 3 their expected effect on the R-Factor. Codec Performance study using E-Model The E-Model is an excellent tool to analyze the performance of different codecs that are operating under similar conditions. Figure 8 shows the relative performance of GSM and AMR codecs. As the chart indicates, AMR codecs deliver a wide range of voice quality, but never higher than the quality provided by the GSM EFR codec. Additionally, another important observation from Figure 9 is that the performance of lowest bit-rate AMR codec is worse than GSM HR. Figure 9: Chart shows the results of a codec performance study and its impact on the R-Factor. 3 Data from ITU study conducted using ITU-T G.107 recommendations. . Page 9 of 13 Network planners should always consider the combined impact of external impairments and codecs on voice quality. As can be seen in Figure 9, the quality of a call made with AMR HR @4.75 kbps rate with a 40dBm noise level, the R-Factor score for the call decreases to 38. This quality is significantly lower than ITU-T suggested minimum of R=50, and would cause significant network-induced churn. Adoption of E-Model in VoIP The E-Model has been successfully adopted in most of the VoIP industry as the de-facto standard for measuring voice quality, and compliments other test and measurement tools. Major test and measurement firms like Brix and Telchemy have adopted the E-Model as the standard for measuring voice quality on IP networks. Moreover, entire markets like Japan have government regulations mandating a specific R-Factor for carrier VoIP services. With the global communications industry moving toward network convergence and an IP core, it is time for mobile industry to adopt the E-Model as the voice quality KPI of choice. E-Model is suitable for today’s TDM network as well as for tomorrow’s IP-based network. EXPERIENCE INTELLIGENCE (EXI): THE E-MODEL SOLUTION This section describes Ditech Network’s Experience Intelligence (EXi) solution and its use in mobile and VoIP networks to establish E-Model KPI. Experience Intelligence (EXi) Ditech Networks’ Experience Intelligence (EXi) solution is an innovative software application that is deployed in TDM networks on Ditech’s Broadband Voice Processor platform, and in IP networks on the company’s Packet Voice Processor. Both platforms are high-density, high-performance, carrier-grade products that currently are in production in networks around the world. EXi continuously and non-intrusively monitors every single call made, and collects data on a wide variety of voice quality impairments. The E-Model is utilized to evaluate these impairments and to deliver an R-Factor for both the downlink and uplink sides of the call. EXi is the industry’s only solution that assesses actual conversational quality on live subscriber calls as they are occurring. EXi measures and reports on the following voice quality variables: • • • • • • • • • Transmission Rating (R) Factors o Conversational Quality (R-CQ) Mean Opinion Scores (MOS) o Conversational Quality (MOS-CQ) Noise levels Speech levels Hybrid echo delay Hybrid echo return loss Acoustic echo delay Weighted acoustic echo path loss Echo Objection Rate per ITU-T G.131 The following two charts show the details of EXi reporting and the values associated with both ends of a call. . Page 10 of 13 Side 1 - IN Side 1 Side 1 - OUT Measured Metric Speech Level Side 1 IN Side 1 OUT Side 2 - OUT PSTN Side 2- IN Side 2 IN Side 2 OUT Side 2 √ √ √ √ √ √ √ √ √ √ √ √ √ √ √ √ √ √ √ Noise Level SNR Hybrid ERL Hybrid ERLE Hybrid Echo Delay Acoustic ERL Acoustic ERLE Acoustic Echo Delay Call Duration -- Min, max, variance and final value, -- Histogram of distribution of all final values Same as above Same as above One value One value One value √ √ √ One value One value One value One value Figure 10: EXi measurement parameters. Side 1- IN Side 2 - OUT Side 1 Side 1 - OUT Side 2 - IN PSTN Side 2 Measured Metric MOS-CQ R-CQ Figure 11: EXi scoring metrics. Side 1 IN Side 1 OUT Side 2 IN Side 2 OUT √ √ √ √ √ √ √ √ Same as above Same as above EXi And The E-Model EXi reports R-Factor results based on the G.107 scoring algorithm developed by industry’s leading expert, Telchemy. Telchemy has sold more than 2 million licenses for its G.107-based scoring algorithms. Figure 12 illustrates the impact of RF-related impairments and external impairments on the R-Factor. The chart also shows the impairments that are measured by EXi. . Page 11 of 13 Figure 12: Chart shows the impact of several variables on the R-Factor, and indicates voice quality impairments that are evaluated by EXi using the E-Model. R-Factor values less than 50 are not recommended by the ITU-T, and commonly result in churn. EXi Reports EXi analyzes and summarizes network-wide statistics, including: • Summary view of transmission quality • Detailed views of voice quality impairments • Historical data that is used for trend analysis EXi reports include charts, histograms and/or historical trends. They can be customized based on several variables: • • • • Equipment shelf System Facility Date range Figures 13: Example of EXi report. . Page 12 of 13 Figure 14: Example of EXi report. Please also see Figure 13, which shows an actual EXi report from a network audit in a TDM network. For further information on EXi, please refer to www.ditechnetworks.com CONCLUSION The ITU-T G.107 E-Model is a KPI that measures voice quality as it is actually experienced by mobile and VoIP subscribers on live calls. Current voice quality KPIs are highly effective in measuring RF performance in mobile networks, for example, but are unable to measure impairments that are external to the network. The E-Model compliments today’s test and measurement tools and is the basis for a new voice quality KPI. An effective solution that utilizes the E-Model is Experience Intelligence (EXi). When combined with existing test and measurement methodologies, EXi helps service provider’s obtain a greater understanding of the full effect of voice quality on churn, revenue and service differentiation. Figure 15: Chart shows a visual and numerical summary of the E-Model and its use to analyze voice quality impairments that are both internal and external to the network. . Page 13 of 13