Delivered Audio Quality Verification by mq49C2d

VIEWS: 69 PAGES: 41

									           Projects in Electrical &
           Computer Engineering
                2004-2005



     Delivered Audio
    Quality Verification

             1/c Roland T. Orr
Project Advisors: Dr. Gross, CDR Routhier
      Sponsor: Rescue 21 (G-AND)
                   Outline of Talk
   Introduction
   Objectives
   Approach
   Results
   Conclusion



24 November 2011    USCGA Electrical & Computer Engineering   2
                   Introduction
    Rescue 21
            General Dynamics and CG
            Headquarters
            19-year, $611 million contract
            Replace outdated VHF equipment
            Help better SAR capabilities




24 November 2011   USCGA Electrical & Computer Engineering   3
                   Introduction

   My focus in Rescue 21
         Holding General Dynamics to the
         performance standard in the requirement
         specification
         General Dynamics is required to meet a
         certain standard of speech clarity with its
         VHF equipment
         One way industry measures speech
         clarity is Delivered Audio Quality (DAQ)

24 November 2011   USCGA Electrical & Computer Engineering   4
                   Introduction
   Delivered Audio Quality (DAQ)
         Measurement of speech clarity published
         by the Telecommunications Industry
         Association (TIA)
         Scale of 1-5; 1 is noisiest, 5 is clearest
         Subjective – different people hear
         different DAQs
         We want an objective, logical way to
         measure speech clarity

24 November 2011   USCGA Electrical & Computer Engineering   5
                                                                            SINAD
DAQ                   Subjective Performance Description                   equivalent
                                                                         intelligibility
 1                  Unusable, Speech present but unreadable                 <8 dB
         Understandable with considerable effort. Frequent repetition
 2                                                                       12 +/- 4 dB
                          due to noise/distortion
       Speech understandable with slight effort. Occasional repetition
 3                                                                       17 +/- 5 dB
                      required due to noise/distortion
       Speech understandable with repetition only rarely required.
3.4                                                                20 +/- 5 dB
                        Some noise/distortion
 4          Speech easily understood. Occasional noise/distortion        25 +/- 5 dB
4.5         Speech easily understood. Infrequent noise/distortion        30 +/- 5 dB
 5                         Speech easily understood.                        >33 dB




 24 November 2011       USCGA Electrical & Computer Engineering                  6
                   Introduction
   SINAD (Signal, Noise, Distortion)
         Measurement of signal strength
         Scale from zero to infinity
           signal  noise  distortion
               noise  distortion
         Objective measurement
         If you have the original, noiseless signal
               The SINAD is not refutable – it is what it is
               SINAD is exact – only one SINAD for a file

24 November 2011    USCGA Electrical & Computer Engineering    7
                   Introduction
   DAQ vs. SINAD
         Subjective vs. objective
         Fuzzy vs. exact
         How do you objectively measure DAQ?
              TIA Standard includes a table that matches DAQ
              values to SINAD values. Excerpt on next slide
         Would be easier if reqt spec was SINAD – but is
         this best?
         Is equating subjective to objective reasonable?


24 November 2011    USCGA Electrical & Computer Engineering    8
                                                                            SINAD
DAQ                   Subjective Performance Description                   equivalent
                                                                         intelligibility
 1                  Unusable, Speech present but unreadable                 <8 dB
         Understandable with considerable effort. Frequent repetition
 2                                                                       12 +/- 4 dB
                          due to noise/distortion
       Speech understandable with slight effort. Occasional repetition
 3                                                                       17 +/- 5 dB
                      required due to noise/distortion
       Speech understandable with repetition only rarely required.
3.4                                                                20 +/- 5 dB
                        Some noise/distortion
 4          Speech easily understood. Occasional noise/distortion        25 +/- 5 dB
4.5         Speech easily understood. Infrequent noise/distortion        30 +/- 5 dB
 5                         Speech easily understood.                        >33 dB


             (TIA says: “SINAD values are NOT to be used
                  for system performance assessment)

 24 November 2011      USCGA Electrical & Computer Engineering                   9
                   Background
   Rescue 21 standard (from req’t specs)
         From a:
              1-watt radio
              Located 6 feet above the ocean’s surface
              20 miles away
         The new equipment must:
              Receive radio calls with a DAQ of at least 3.4
              Be able to locate the bearing of a caller within 2º
              And more… but here we are only worried about the two
              above
   General Dynamics has started replacing CG’s VHF
   equipment
   They have their own plan to measure the subjective
   DAQ values

24 November 2011    USCGA Electrical & Computer Engineering          10
                   Background
   What was accomplished in previous years
       1/c Brian Warner 2002-2003
       1/c Jen Jojola 2003-2004
       First 1 ¾ years were spent as if we do not know
       the transmitted signal
       From end of last spring until now, we do know the
       transmitted signal – this changes things
       considerably
   New approach uses known benchmark audio
   files for transmission


24 November 2011   USCGA Electrical & Computer Engineering   11
                   Objectives
   Goals
       Measure the DAQ of audio files received by
       General Dynamics equipment
       Assess the situation – does General Dynamics
       meet the standard of DAQ > 3.4?
   Deliverables
       Code for an algorithm that calculates SINAD given
       a flawless version of the signal
       Documentation for SINAD algorithm
       Analysis of General Dynamics’ equipment


24 November 2011   USCGA Electrical & Computer Engineering   12
         Technical Approach
           Transmitted
                                                           Received
           Benchmark           A             +             Noisy File
               File



                                           Noise



Test the system - RCVD = A*XMIT + Noise
    1.    Record a library of minimal noise benchmark audio files
    2.    Use noisify() to add filtered noise to the benchmark audio
          files at varying levels, yielding noisy counterparts
    3.    Acquire DAQ values from a test audience for the noisy
          audio files
    4.    Calibrate a SINAD-DAQ relationship


24 November 2011     USCGA Electrical & Computer Engineering            13
       Technical Approach
Once system calibration is complete, move to
field testing:
  1. Take to the field with benchmark files – broadcast
     them and record the received version of the
     benchmark
  2. Calculate SINADs of the received benchmarks
  3. Find DAQ via SINAD
  4. Assess the DAQ of General Dynamics’ equipment




24 November 2011   USCGA Electrical & Computer Engineering   14
        Technical Approach
 People with a                                                 DAQ values
                                           Subjective
range of voices                                                for the noisy
                                          test audience
                                                               benchmarks
 Benchmark            Noisy
  Library           Benchmarks
                                                               SINAD values
                                           My SINAD
                                                                for the noisy
                                           algorithm
                                                                benchmarks
   noisify()
                    System Test

 Analysis of                              Method for
supplied VHF                             testing VHF
  equipment                               equipment
 24 November 2011    USCGA Electrical & Computer Engineering                    15
                   Initial Results
  Used 16 benchmark audio files and noisy
  versions
  Played noisy versions for a test audience of
  24 people
  Averaged all DAQ’s together for each sample
  Used my SINAD.m to find SINAD
  Initial data does not support TIA’s SINAD-
  DAQ relationship.

24 November 2011    USCGA Electrical & Computer Engineering   16
                    Initial Results
                      SINAD vs. Subjective DAQ values

        35


        30


        25


        20
SINAD




        15

                                                                     My Data
        10
                                                                     My Fit
                                                                     TIA's Relationship
         5


         0
        0.0000   1.0000    2.0000     3.0000     4.0000     5.0000     6.0000
                             Subjective DAQ Values


 24 November 2011         USCGA Electrical & Computer Engineering                     17
           What was wrong?
   My results clearly did not match TIA’s
   Too small an audience? – increase size
   Too small a library? – increase library
   Simulated noise was not representative
   of real noise? – study VHF noise



24 November 2011   USCGA Electrical & Computer Engineering   18
                   Next Results
We increased the library to 30 benchmarks
Increased the audience to 44 people
Also used data from the early results for the
first 16 benchmarks (24 people)
Relationship became more linear, but still
left something to be desired…


24 November 2011   USCGA Electrical & Computer Engineering   19
                    Next Results
                          SINAD vs. Subjective DAQ values

        45
        40
        35
        30
SINAD




        25
        20
        15
        10
        5
        0
             0      1            2             3             4    5        6
                                     Subjective DAQ Values




 24 November 2011       USCGA Electrical & Computer Engineering       20
            What to do next?
The larger audience and benchmark library did
not clear up the results to our satisfaction
Recall: RCVD = A*XMIT + Noise
                                              Transmitted
Look at the “Noise” portion             A    +Benchmark
                                                  File
                                                                     Received
                                                                     Noisy File


   Modeling it incorrectly?
                                                             Noise
   Normally distributed?
We decided to launch into a study of VHF noise



24 November 2011   USCGA Electrical & Computer Engineering           21
  Noise Analysis Process
   Using VHF radios
       transmit a benchmark on one radio
       receive and record on the other
   Subtract the benchmark from the
   received recording to get noise
   Current equipment was inadequate, so
   we dished out some cash to buy new
   VHF radios

24 November 2011   USCGA Electrical & Computer Engineering   22
               Testing Radios
   Decided on Standard Horizon HX-370
   Had one jack for mic in/spkr out
       Bought “clone cable”
       Bought “speaker mics”
   Took apart speaker mics
       Hardwired one to be microphone only
       (transmitting)
       Other speaker only (receiving)

24 November 2011   USCGA Electrical & Computer Engineering   23
                   Transmitter




24 November 2011   USCGA Electrical & Computer Engineering   24
                      Receiver




24 November 2011   USCGA Electrical & Computer Engineering   25
Mobile Transmitting Unit




24 November 2011   USCGA Electrical & Computer Engineering   26
                   Receiving Unit




24 November 2011    USCGA Electrical & Computer Engineering   27
               Overall System




24 November 2011   USCGA Electrical & Computer Engineering   28
                    VHF Analysis
                                                      Transmitted
                                                                                     Received
                                                      Benchmark     A    +           Noisy File
                                                          File
   Recall, RCVD = A*XMIT + Noise
   SINAD.m algorithm used the equation                                  Noise


                       Benchmark T Received
                   A
                      Benchmark T Benchmark
   This equation estimates the “A” value (gain) poorly
   for large amounts of noise
   When using benchmarks, the A estimate was
   horrible, yielding negative SINADs!
   Voice benchmarks were simply not working for this
   part

24 November 2011      USCGA Electrical & Computer Engineering                   29
                   Sinusoid Test
   Recorded single tone sinusoids
   “A” estimate was much more accurate
   SINAD.m algorithm worked perfectly
   Results indicate normal-distributed
   noise
   Noise bandwidth perhaps too wide?


24 November 2011    USCGA Electrical & Computer Engineering   30
             Sinusoid Results




24 November 2011   USCGA Electrical & Computer Engineering   31
             Sinusoid Results




24 November 2011   USCGA Electrical & Computer Engineering   32
             Sinusoid Results




24 November 2011   USCGA Electrical & Computer Engineering   33
             Sinusoid Results




24 November 2011   USCGA Electrical & Computer Engineering   34
             Sinusoid Results




24 November 2011   USCGA Electrical & Computer Engineering   35
             Sinusoid Results




24 November 2011   USCGA Electrical & Computer Engineering   36
        Periodic Chirp Test
   Recorded periodic chirp, 0-25.6 kHz
   Transmitted the periodic chirp
   Eye-balled the “A” value
   Used data to get magnitude response of
   the system as a whole
   Magnitude response was NOT
   frequency independent!

24 November 2011   USCGA Electrical & Computer Engineering   37
   Periodic Chirp Results




24 November 2011   USCGA Electrical & Computer Engineering   38
         Analysis of Results
   Why is “A” not frequency independent?
       Clipping on transmitting side
       Clipping on received side
       Bandwidth limitations on radio transponders
   Maybe we should model “A” to be frequency
   dependent
       Relationship would then be VHF-only, specific to
       the two radios used in testing – not just any radio
       Is there a universal DAQ-SINAD relationship
       anyway?

24 November 2011   USCGA Electrical & Computer Engineering   39
                   Conclusions
   Summary
       RCVD = A*XMIT + Noise
           A assumed to be frequency independent
       Benchmark Library
           Perfect audio files
           Noisy audio files with measured DAQs and known
           SINADs to make a DAQ-SINAD relationship
       A DAQ-SINAD relationship takes the
       subjectiveness out of Rescue 21’s standard



24 November 2011   USCGA Electrical & Computer Engineering   40
                   Conclusions
   “A” was not frequency independent
       RCVD = A(w)*XMIT + Noise
                                                        (Noise was always a
       Internal amplifiers in radio                    function of frequency)
       PC sound cards
       Perhaps more realistic
   DAQ-SINAD relationship is harder to attain
   than we thought
   Many variables to dispute
   Any DAQ-SINAD relationship would have to
   be agreed on by both parties

24 November 2011   USCGA Electrical & Computer Engineering                      41

								
To top