Savings Estimation Methods for _savings

							Savings Estimation Methods
for Energy Efficiency Programs:
A Half-Hour Guide

          Kansas Corporation Commission
           Workshop on Energy Efficiency
                 March 25, 2008
                                                       Marian Brown
                               Manager of Measurement & Evaluation
                                           Energy Efficiency Division
                                          Southern California Edison
                                            marian.brown@sce.com
                                                     (626) 812-7662
Why Measure Energy Efficiency Program
Savings?
   Metering the un-metered resource
       Measured savings for resource planning
       Measured benefits from reduced pollution/GHG
       Accountability for ratepayer dollars and
        shareholder reward
   Improve programs
       More accurate savings estimates for measures
       Increased energy savings from better design
       Basis for future EE program funding
                        Southern California Edison     2
Types of Programs and Impact
Evaluations
   Resource Acquisition Programs –
       Create direct energy savings
   Education/Information Programs
       Create indirect energy savings
   Market Transformation Programs
       Create long-term changes in product availability,
        cost and features, plus customer awareness,
        understanding and purchase behavior


                         Southern California Edison         3
Direct Energy Savings

   Program participation is defined as purchasing or
    adopting the energy efficiency measures.
       That’s how rebate and direct installation programs work.
   So the first job is to estimate the energy savings
    achieved from those adoptions (called gross energy
    savings).
   Then the follow-up question is in which cases the
    program caused the purchase/installation/adoption
    (net energy savings; attribution; net-to-gross ratio).


                            Southern California Edison             4
Indirect Energy Savings

   The program activity is intended to influence a
    decision to adopt efficiency measures,
    rather than require it as a condition
    for participation.
       Examples: Energy audits, energy efficiency training,
        education, information, market transformation.
   So the evaluation must first determine whether the
    program did indeed lead to adoption of energy-
    efficient measures (attribution/net-to-gross ratio).
   Then, it must estimate what the resulting energy
    savings were.

                            Southern California Edison         5
Estimating Gross
     Energy
  and Demand
     Savings




For Both Direct and Indirect
     Savings Programs


                        Southern California Edison   6
The Two Basic Approaches
Statistical Billing (Energy Use)            Engineering Measurement and
   Analysis                                   Analysis

   Simple billing analysis                     Pre-post measurements and
    normalized for weather                       engineering algorithms
   Regression analysis
                                                Pre-post building energy
   Multi-stage regressions                      simulation models
    (can include net savings
    analysis)

             Sometimes, the two approaches are combined.
    We try to include economic/behavioral effects wherever possible.



                              Southern California Edison                     7
Which Measurement & Analysis Method
to Use for the Energy Savings?
   Methods range from simple and direct to complex
    and indirect and sometimes combined.
   More complex methods generally require more
    detailed data and higher cost.
   Guidelines for good measurement/analysis:
       The Model Energy Efficiency Program Impact Evaluation
        Guide covers both billing analysis/regression methods and
        engineering methods, plus net savings methods.
       IPMVP (International Performance Measurement &
        Verification Protocol) is a widely recognized standard for
        engineering measurement and analysis approaches.

                            Southern California Edison               8
Billing Analysis
Tends To Be Preferred When:
    Both pre- and post-retrofit billing data are available.
    Expected energy savings are large enough to be
     statistically observable in a billing analysis (at least
     10% of total usage; depends on several factors).
    Analysis can include large numbers of participants
     that are reasonably homogeneous.
    Inclusion of a comparison or control group allows for
     behavioral change corrections and/or simultaneous
     estimation of net savings


                        Southern California Edison              9
Engineering Measurement & Analysis
Tends To Be Preferred When:
   No pre-installation billing data are available
    (e.g., new construction).
   Expected energy savings are a small fraction of
    total energy usage.
   Program has a small number of participants or
    unique measures.
   Program itself includes substantial engineering
    analysis that can be built on.


                       Southern California Edison     10
Using a Sample to Estimate Program Savings
   Choosing when to use a sample:
       Can you get good enough data for the whole population of participants?
        (And what is the whole population?)
       Or should you gather more detailed and accurate data for a good sample?

   Questions to ask to assure reliable results from a sample:
       Is it unbiased (or properly weighted to be unbiased)?
        •   Non-response bias (do some groups respond at lower rates?)
        •   Survey selection bias: Did it exclude relevant groups from the sample?
            •   Can you think of some examples?

       Does it give the needed level of precision?
        •   Depends on sample size and variance in the population.
            • How far off can an estimate be and still be useful to you?



                                  Southern California Edison                         11
A Key Question: What Would Energy Use
Be in the Absence of the Program?
   We can measure energy use before the
    program.
   We can measure energy use after the program.
       But how much of the change was due to the program?
   Conventionally, we look at this question in two
    dimensions:
       One related to gross energy savings;
       One related to attribution and net energy savings.


                           Southern California Edison        12
Gross Energy Savings Baseline

   What technology/behavior choice was being
    made?
       Pure retrofit: Early replacement or add-on
           Then savings is prior energy use minus new energy
            use
       Replacement on burnout or new construction (lost
        opportunity market)
           Then savings is efficient equipment energy use
            minus usage of a standard efficiency new measure


                           Southern California Edison          13
Determining Attribution, the Net-to-
Gross Ratio, and Net Energy Savings


Attribution:
  What energy savings
  are the result of the
  program, rather than
  other influences?




                    Southern California Edison   14
Indirect Savings Programs

   Attribution is the first question for audit, training,
    education, marketing, and market transformation
    programs.
         Step 1: Who was actually exposed to the program?
         Step 2: How many of those exposed to the program
          activities changed their behavior in energy-saving ways?
         Step 3: Of those who changed, how much of the
          changes were due to the program rather than other
          influences?
         Step 4: Estimate the energy savings of those changes
          identified as due to the program.

                           Southern California Edison            15
Direct Savings Programs

   Attribution is usually the second question asked
    for rebate and direct install programs.
         Step 1: How much energy did participants
          save?
         Step 2: What fraction of participants’ changes
          were due to the program, rather than other
          causes?




                        Southern California Edison         16
The Traditional Direct Savings Approach:
The Net-to-Gross Ratio
   = The fraction of the apparent program energy savings
    that are truly the result of the program
   Example:
    Program pays rebates to 100 customers who install high
    efficiency windows and claims the energy savings of these 100
   Sources of difference for apparent vs. actual:
     Free riders
        20 of these customers were already planning to do this without
        the rebate offer.
       Spillover
        10 neighbors also install these windows because of their
        installing neighbors’ recommendations, but don’t get the rebate.


                              Southern California Edison                   17
Measurement Methods for the Net-to-
Gross Ratio (and their Key Difficulties)
   Self-Report Survey
       Minimize bias towards self-credit for decisions
   Regression Analysis Comparison of Participants
    and Non-Participants
       Identify a comparable group of non-participants
   Market Baseline
       Find data for proportion of efficient sales before
        and after program, for the program region and a
        comparison region

                          Southern California Edison         18
When the Traditional Net-to-Gross
Approach Doesn’t Work
   When one or more related programs have long-term
    effects on a market, estimating free riders and even
    some short-term forms of spillover misses the major,
    long-term sources of savings due to the program(s).
   Then, we need to monitor how the market itself
    changes over time and try to determine how much of
    the change is due to the programs.
   The next presentation will discuss the measurement
    of programs with these types of market
    transformation effects.

                       Southern California Edison      19
Attribution Estimation is Difficult and
Imprecise. Can’t We Just Skip It? No!

   We can’t just pretend a substantial effect doesn’t
    exist, because it’s hard to measure.

   We must pay attention to how much difference
    the program makes, both for:
       Accountability to ratepayers; and
       Picking the right measures and the right programs
        to maximize cost-effectiveness.

                         Southern California Edison     20
References and Website Locations (1)

Basic National Sources for Program Evaluation Methods
The Model Energy Efficiency Program Impact Evaluation Guide, developed by US EPA
    and the National Action Plan for Energy Efficiency,
http://www.epa.gov/cleanenergy/documents/evaluation_guide.pdf

The International Performance Measurement & Verification Protocol, developed by the
   Efficiency Valuation Organization, http://www.evo-world.org/

California’s database of deemed energy savings estimates for energy efficiency
    measures, the Database for Energy Efficiency Resources (DEER):
    http://eega.cpuc.ca.gov/deer/




                                   Southern California Edison                         21
References and Website Locations (2)
Basic California Energy Efficiency Analysis Sources
The California Standard Practice Manual (definitions and rules for cost-effectiveness tests)
    http://www.calmac.org/events/SPM_9_20_02.pdf

California’s database of deemed energy savings estimates for energy efficiency measures, the
     Database for Energy Efficiency Resources (DEER): http://eega.cpuc.ca.gov/deer/

The California Energy Efficiency Evaluation Framework, a comprehensive and in-depth guide to all
    aspects of EE program evaluation:
    http://www.calmac.org/events/California_Evaluation_Framework_June_2004.pdf

California Energy Efficiency Evaluation Protocols , the California Public Utilities Commission CPUC)
     rules on how to conduct EE program evaluations:
     http://www.calmac.org/events/EvaluatorsProtocols_Final_AdoptedviaRuling_06-19-2006.pdf

CALMAC Website: http://www.calmac.org/ . Toolkit sections has several CPUC and California EE
   evaluation guidance documents. Searchable Database section has over 600 California energy
   efficiency studies, including impact evaluations, process evaluations, market analysis studies. All
   of the California utilities’ EE evaluation studies are posted here.



                                         Southern California Edison                                    22