ROC

							Receiver Operator Characteristics


  • What is it and where does it
    come from

  • Statistical aspects

  • Use of ROC
Early radar signals




                     Is this an enemy plane?




    Signal Noise Ratio
       The problems of decision
• Sound the alarm when the signal is very small
   – Advantages
      • Plenty of time to get the fighters off the ground
      • Reduce the number of bombers reaching the target
   – Disadvantages
      • Lots of false alarms
      • Waste of gasoline, wear and tear on fighter planes
      • Exhaust fighter pilots

• Sound the alarm when the signal is unmistakable
   – Advantages
      • No waste, no wear and tear, no exhaustion
   – Disadvantages
      • More bombers get through, more bombs, more destructions
            Solution to decisions
• Code Yellow
  – Signal suggests possible incoming bomber
  – Pilots get dressed, fighter planes get loaded with gasoline
    and ammunition

• Code Orange
  – Signal suggests incoming bomber likely
  – Fighter planes towed to runway, pilots goes to the planes

• Code Red
  – Signal is unmistakable
  – Fighter planes take off
          Refinement to solutions
• Responses variable
  – Radar receivers varies in signal strength and noise
    level
  – Technicians operating the receiver interpret the
    signals differently
  – Each receiver and its operator must be characterised,
    so that their reports can produce a consistent
    response
• The Receiver Operator Characteristic (ROC)
  – The relationship between not missing an incoming
    bomber (Sensitivity) and false alarms (False
    Positives)
        Receiver Operator Characteristics
Perfect operator
100% Sensitive
0% false alarms


                                 Most operators
                   Sensitivity




                                           Useless operator
                                           Sensitivity and
                                           false alarm rate
                                           changes together


                                      False alarms
     Receiver Operator Characteristics
Increasing
  signal
 strength

                                      Code Yellow
                              Code Orange
             Sensitivity




                           Code Red




                                      False alarms
  Receiver Operator Characteristics

Increasing
  signal
 strength

                                      Code Yellow
                              Code Orange
             Sensitivity




                           Code Red




                                      False alarms
              ROC since the war
• The ROC was effective translating measurements
  into decisions
• A system of different level of alerts are common
  decision processes
   – Economy and company performance
   – Risk of fire, drought, natural disasters, emergencies
   – International diplomacy, risk of war
• Extensive developments in statistics and
  mathematics to enhance the method
   – Introduced into medical decision making in the 1960s
   – popularised by medical educators in the 1980s as a
     method of teaching decision making in medicine
   – Becoming a common method to evaluate the quality of
     predictions and tests since the 1990s
Receiver Operator Characteristics


  • What is it and where does it
    come from

  • Statistical aspects

  • Use of ROC
                    Statistical ROC
• A measurement is normally distributed in two groups,
  those outcome negative and those outcome positive
• Using a cut off level to make a decision will create a
  number of TP, FN, FP, and TN. From these Sensitivity
  and Specificity is calculated
• If the cut off value changes
   – TP,FN,FP,TN changes
   – Sensitivity and Specificity changes

• The relationship between Sensitivity and Specificity
  over the range of the measurement defines the ROC
Statistical ROC
Receiver Operator Characteristics


  • What is it and where does it
    come from

  • Statistical aspects

  • Use of ROC
       Advantages of using ROC

• It defines the quality of a test or prediction
  using a measurement without specifying a
  cut off value for decision making
• Assuming Normal distribution
   – The mean and Standard Error can be estimated
   – The 95% CI can be estimated
   – Statistical significance can be determined

   – Whether one test is better than another can be
     determined
                 Use of the ROC

                     Sensitivity > Specificity
                       Cut off value for
                        screening test
Sensitivity




              Specificity > Sensitivity
                 Cut off value for
               intervention decision




                         1 - Specificity