Introduction The Regional Evaluation Measurement and Verification by JasonDetriou


									Regional EM&V Forum – PJM EE Stakeholders
Comments to PJM Re: Draft Energy Efficiency M&V Manual (Revision 0), January 23, 2009


The Regional Evaluation Measurement and Verification Forum (‘the Forum’) submits these
comments and recommendations to the PJM Interconnection (‘PJM’) in regard to its DRAFT
PJM Manual: Energy Efficiency Measurement & Verification Revision: 0 (“Draft Manual”) for
energy efficiency resources participating in PJM’s Reliability Pricing Model. These comments
were assembled with input from the following energy efficiency stakeholders (“Stakeholders”),
all of whom reserve the right to submit their own comments directly to PJM: Pepco Holdings,
Inc., Baltimore Gas & Electric, Delaware Energy Office, the Maryland Energy Administration,
and the American Council for an Energy Efficient Economy.

The purpose of these comments is to provide suggested language changes and/or
background/context to support the comments, with the intent of ensuring that the M&V standards
ultimately adopted by PJM provide assurance that the Nominated Energy Efficiency Value of
energy efficiency resources participating in PJM’s wholesale capacity market are legitimate and
sufficiently rigorous, without creating unduly burdensome and costly M&V requirements on
Capacity Market Sellers that prevent participation in the market. Our comments focus on the
following specific sections of PJM’s Draft Manual:

            •   Section 5 – Post Installation M&V Reporting Components
            •   Section 9 – Establishing Baseline Conditions
            •   Section 10 – Statistical Significance
            •   Section 13 – Measurement Equipment Specifications

We look forward to discussing these comments with PJM staff to address any questions and/or
provide clarification and further comment.

Section 5 Post-Installation M&V Report Components


There are no suggested revisions to this section. The following comments are offered to PJM in
order to suggest a reasonable approach to a compliance process that is consistent with standard
M&V practice.

The M&V Manual requires the submission of post-installation M&V reports to document the
actual measure installations which determine the final Nominated EE Value. The post-
installation M&V report documents the implementation of the M&V plan, including a
description of the post-installation verification activities and performance measurements, the
results of these activities and revisions to the Nominated EE Value calculations to reflect the

Regional EM&V Forum – PJM EE Stakeholders
Comments to PJM Re: Draft Energy Efficiency M&V Manual (Revision 0), January 23, 2009

results of the post-installation M&V activities conducted between the submission of the Initial
M&V Plan and the beginning of the Delivery Year that are employed to inform the
determination of the final Nominated EE Value.

According to this understanding of the requirements, the report would document the measure
tracking processes used to provide a complete accounting of installed measures, the aggregate
numerical results of the tracking process and the revised calculations of measure savings and EE
Value. The report may also describe all quality assurance inspections to verify measure
installations and the methodology and results of any surveys or direct monitoring studies
employed to improve the accuracy of the revised calculations.

EE Resource measurement and verification activities may be conducted as part of EE program
implementation processes, separate M&V studies, or draw upon the results of review and
research by associations that establish standard calculation methods, equipment specifications
and assumptions regarding key parameters (e.g., rated efficiency) that determine measure
performance and savings.

These activities typically incorporate many of the following elements:

   •   Tracking of critical data for all measures which contribute to the EE Value.
   •   The implementation of quality assurance procedures to confirm the installation of
       measures and the accuracy of tracking data.
   •   The use of technical specifications provided by equipment manufacturers.
   •   The use of stipulated factors that are accepted by the industry.
   •   The use of standard engineering algorithms to calculate non-coincident demand
   •   On-site verification of measure installation and specifications in a sample of program
   •   On-site monitoring of measure performance during the relevant performance hours in
       order to determine coincidence factors for specific measure categories.
The timing and scope of each measurement and verification activity depend upon the nature of
the EE resource, the type of data that is being verified and the cost of the measurement process
employed. The following observations are generally applicable to the measurement and
verification of EE Resources:

   •   Measure tracking systems provide a detailed account of the timing, quantity and
       characteristics of all measure installations that comprise the EE Resource. The tracking
       system database is continuously updated as part of the program implementation process.

Regional EM&V Forum – PJM EE Stakeholders
Comments to PJM Re: Draft Energy Efficiency M&V Manual (Revision 0), January 23, 2009

   •   On-site verification of measure installation and specifications may be conducted as part
       of an ongoing quality assurance review or as part of a program impact evaluation of a
       sample of program participants.
   •   On-site measurement and verification activities are typically conducted once every
       several years because of the cost and time required to complete each study and the
       number of different resource types, i.e. programs and measure types, to be measured.
   •   The cost of continuous on-site monitoring of variable loads to measure end-use
       coincidence factors often requires that such studies be undertaken jointly by several
       providers to obtain pooled estimates of factors that are applicable to their respective
   •   In order to effectively manage the total cost of measurement and verification activities,
       the collection of on-site data and the use of end-use monitoring must be prioritized and
       limited to the measures and factors that are most significant in the determination of the
       Nominated EE Value.

We intend for this information regarding standard EM&V activities to help inform PJM in
understanding the types of EM&V data that can support post installation reporting requirements.

Section 9 Establishing Baseline Conditions


The M&V manual specifies two alternative Baseline Conditions based on either the standard
efficiency of new equipment or the efficiency of the equipment being replaced, depending on
assumptions regarding what would occur in the absence of the program. The new equipment
baseline applies to the replacement of equipment that has failed or that is still operating but is
intended to be replaced independently of the program. The existing equipment baseline applies to
equipment that would continue to operate in the absence of the program, possibly after being
repaired or refurbished, or operating equipment that has been removed.

The application of the baseline criterion should in most cases be established by program design.
The new equipment baseline would be employed for point-of-purchase efficiency programs that
are delivered by equipment vendors and efficiency measures that are supported by incentives
based on incremental cost above standard efficiency.

The EE Stakeholders recommend the following revisions, which are intended to clarify the
appropriate application of the Baseline Conditions.

Suggested revisions to the draft M&V manual Section 9, Baseline Requirements for All EE

Regional EM&V Forum – PJM EE Stakeholders
Comments to PJM Re: Draft Energy Efficiency M&V Manual (Revision 0), January 23, 2009

2. For projects in which failed equipment is replaced by a more efficient equivalent or by an
alternative strategy for delivering comparable output, the Baseline Condition shall be the
nameplate rating of the equipment meeting the level of efficiency required by applicable State
code, federal product efficiency standard, or standard practice, whichever is most stringent. If
there is no applicable State code or federal standard, then standard practice shall be used as the
basis for establishing Baseline Conditions and shall be documented in the M&V Plan. This
Baseline Condition shall also apply to projects in which operating equipment would be replaced
independently of the program.

3. For projects in which operating equipment is replaced with a more efficient equivalent unit,
and would not be replaced independently of the program, the Baseline Condition is the kW load
of that operating equipment across the Performance Hours. In the absence of a measured
baseline, the baseline values shall be level of efficiency required by applicable State code or
Federal product efficiency standard or standard practice if there is no applicable state code or
federal standard. If applicable, the EE Resource Provider shall describe the method by which the
Baseline Condition may be adjusted over the commitment years. If standard practice is used as
the basis for the Baseline Condition, it shall be documented in the Measurement & Verification

4. For Projects in which existing and operating equipment is removed from service during the
defined Performance Hours or has its electrical consumption reduced during the defined
Performance Hours, the Baseline Conditions shall be the kW load of that operating equipment
across the Performance Hours prior to such equipment removal or reduced use.

Section 10 Statistical Significance


PJM’s draft Manual includes a statistical precision requirement of 80 percent confidence level
with 10 percent precision, the criterion adopted by ISO-New England for demand resources
participating in ISO-NE’s Forward Capacity Market. PJM verbally indicated at its December 17
EETF meeting, however, that it would likely instead adopt a 90/10 criterion, on the basis of
precedence and that it is more stringent than the 80/10 criterion. The Stakeholders propose that
the 80/10 criterion is a reasonable level for PJM to include in its M&V Manual, as it represents a
tradeoff between the necessity to manage performance risk and the necessity to control M&V
costs. This tradeoff is illustrated via the comparison of the increase in sample size required to
improve the confidence level from 80% to 90% to the corresponding gain in relative precision, as
show in the table below and described herein. Assuming, for example, a Coefficient of Variation
equal to 1.0, a sample size of 164 is required to achieve 10% relative precision with 80%
confidence. The sample size required to achieve the same precision with 90% confidence is 271,
yet the decline in relative precision of the smaller sample at 90% confidence would be less than

Regional EM&V Forum – PJM EE Stakeholders
Comments to PJM Re: Draft Energy Efficiency M&V Manual (Revision 0), January 23, 2009

3%, i.e., ±12.8% instead of ±10%. Given that M&V studies can cost several thousand dollars per
sample, this tradeoff should be given serious consideration.

                  Relative              Confidence               Sample RP at 90%
                  Precision     CV         Level        2S/ 1S    Size  Confidence
                    10%         0.5         80%           2        41        12.8%
                    10%         0.5         90%           2        68        10.0%
                    10%         0.5         80%           1        18        15.2%
                    10%         0.5         90%           1        41        10.0%
                    10%         1.0         80%           2        164       12.8%
                    10%         1.0         90%           2        271       10.0%
                    10%         1.0         80%           1        71        15.2%
                    10%         1.0         90%           1        164       10.0%

What is more, the sample size required to attain 10% relative precision for a one-sided 90%
confidence level is the same as that required to comply with the 80/10 two-sided criterion. In
other words, compliance with the 80/10 two-sided protocol means that there will be 90%
confidence that the population value is no less than 10% lower than the sample estimate. This is
significant because the performance risk exposure in terms of system reliability is heavily
weighted toward under-performance. The Stakeholders further note that an 80/10 2-sided
criterion is more stringent than what is required by PJM for its Direct Load Control program as
put forth in PJM’s Manual 19 (Load Forecasting and Analysis) in Appendix B Direct Load
Control Research Guidelines which requires a 90/20 standard for load research studies.

As described in Section 10 of the Draft Manual1, compliance with the 80/10 criterion can be
achieved either through attainment of the required precision separately by each sample estimate
employed in the calculation of the EE Value, or attainment of the required precision by the total
EE Value calculated as the total of the separate sample estimates. Generally, the statistical
precision of a sum of independent sample estimates, each of which attains the same precision,
will be greater than the precision of each component. Therefore, the combined precision of an
EE Value equal to the sum of several component values that comply with the 80/10 criterion will
achieve a precision level that is better than 80/10.

 “If an EE project consists of multiple sites and/or measures, and the EE Resource Provider uses
multiple samples to estimate the aggregated demand reduction value during the Performance Hours in
each Zone as the sum of all individual measured demand reduction values, the sampling requirements
may be met (1) for each sample or combination of samples used, (2) for the combination of all samples,
or (3) by using strata as described below.”

Regional EM&V Forum – PJM EE Stakeholders
Comments to PJM Re: Draft Energy Efficiency M&V Manual (Revision 0), January 23, 2009

Finally, we suggest that it may be important to FERC that the ISO-NE and PJM adopt the same
statistical precision requirements in the interest of ensuring consistency in treatment of EE
resources from one region to the next.

Suggested revisions to the draft M&V manual Section 10, Statistical Significance:

For the reasons stated above, the EE Stakeholders recommend that the 80/10 criterion be retained
in the M&V Manual as currently written.

Section 13 Measurement Equipment Specifications


The M&V protocols require that measurement equipment, procedures, equipment calibration and
maintenance comply with various national standards. They also establish prescriptive criteria
(±2%) for the accuracy of direct and indirect measurements of kW demand. These standards,
based on the ISO New England M&V Manual, were carefully analyzed in a study commissioned
in Spring 2008 by a group of New England stakeholders consisting of EE program administrators
and utility regulatory staff. The authors of the study, RLW Analytics, express reservations about
certain elements of the standards, in particular, that the cost of compliance with the accuracy
criteria for kW measurement may be prohibitive. While the RLW study is not yet a public
document, the Stakeholders are sharing herein some of its key findings, and are in the
process of developing recommended alternative language to propose to PJM, in
collaboration with New England stakeholders who also plan to discuss such modified
language with ISO-NE staff in the near future.
Key findings from the RLW report include:
   •   “Measuring kW with an accuracy of ±2% in adherence to [ISO-NE] Requirements #6 and
       #9 appears to pose the greatest challenge to M-MVDR compliance... in practice, many
       portable measurement instruments – particularly those that are appropriate and practical
       for efficiency program evaluations – have difficulty achieving this accuracy level.”
   •   “When uncertainty propagation is considered with rigor, about half of popular handheld
       kW meters exceed the M-MVDR kW±2% accuracy requirement. Replacement may
       necessitate significant investment on the part of M&V contractors and Project Sponsors.”
Compliance with Requirement #9, which prescribes the accuracy criterion for indirect
measurement of kW (i.e., proxy variables), was identified as especially problematic:
   •   “the implications of this Requirement #9 are proving most contentious amongst utilities
       and M&V contractors.” Compliance could be “impractical, prohibitively expensive, or
       unreasonably burdensome to customers.” “Requirement #9 may have tremendous cost
       and accuracy implications for Other Demand Resources.”

Regional EM&V Forum – PJM EE Stakeholders
Comments to PJM Re: Draft Energy Efficiency M&V Manual (Revision 0), January 23, 2009

   •   “Prohibiting the use of such EMS data would be a tremendous loss for evaluation rigor,
       requiring much added cost to replace the informational value.”
These findings pose serious questions regarding the feasibility and desirability of compliance
with the PJM Draft Manual. Overly rigorous requirements in Section 13 of the manual could
make compliance onerous and create a barrier to entry into PJM’s forward capacity markets.

These requirements should also be reconsidered in the larger context of the statistical precision
criteria earlier discussed above. The calculated standard error of a sample estimate incorporates
random measurement errors in addition to sampling errors. Therefore compliance with the
precision criteria effectively controls the overall uncertainty in the estimated quantity of interest
regardless of compliance with the equipment accuracy requirements.
Sample design for M&V studies is typically informed by the empirical results obtained in similar
studies. Utilization of the coefficient of variation from a previous study with similar
measurement procedures in the sample design will accordingly account for the contribution of
random measurement error to the population variance and indicate a sample size large enough to
attain the desired precision. While random measurement errors can be statistically controlled, a
constant bias in the measurements results in a biased sample estimate, although equipment
calibration and maintenance should mitigate this problem.

Suggested revisions to the draft M&V manual Section 13, Measurement Equipment
Specifications: Forthcoming.


To top