Get documents about "Commonwealth Edison Company (PDF)
Document Sample
Commonwealth Edison Company
Energy Efficiency/Demand Response Plan
Plan Year 1 (6/1/2008-5/31/2009)
Evaluation Report:
Appliance Recycling
December 2, 2009
Submitted To:
ComEd
Final Report
Submitted to:
ComEd
Three Lincoln Centre
Oakbrook Terrace, IL 60181
Submitted by:
Summit Blue Consulting, LLC
1722 14th Street, Ste. 230
Boulder, CO 80302
720.564.1130
Contact: Randy Gunn, 312-938-4242, rgunn@SummitBlue.com
Jeff Erickson, 608-807-0082, jerickson@SummitBlue.com
Prepared by:
Jennifer Fagan Tami Buhr
Itron, Inc. Opinion Dynamics Corporation
(608) 235-1314 (617) 492-1400
Jennifer.fagan@itron.com tbuhr@opiniondynamics.com
TABLE OF CONTENTS
Executive Summary – Appliance Recycling .................................................................. 1
Evaluation Objectives ................................................................................................. 1
Evaluation Methods .................................................................................................... 1
Key Findings and Recommendations............................................................................ 2
Key Impact Findings ................................................................................................... 4
1 Introduction to the Program .............................................................................. 8
1.1 Program Description ........................................................................................ 8
1.2 Evaluation Questions ....................................................................................... 8
2 Evaluation Methods ........................................................................................... 10
2.1 Analytical Methods ......................................................................................... 10
1.1 Data Sources ................................................................................................. 12
2.2 Sampling Plan ................................................................................................ 13
2.3 Sampling Error ............................................................................................... 13
3 Program Level Results....................................................................................... 15
3.1 Impact Evaluation Results............................................................................... 15
3.1.1 Verification and Due Diligence .............................................................. 15
3.1.2 Tracking System Review ...................................................................... 18
3.1.3 Gross Program Impact Parameter Estimates.......................................... 20
3.1.4 Gross Program Impact Results ............................................................. 23
3.1.5 Net Program Impact Parameter Estimates............................................. 24
3.1.6 Net Program Impact Results ................................................................ 24
3.2 Process Evaluation Results .............................................................................. 25
3.2.1 Program Theory .................................................................................. 25
3.2.2 Creation of the Logic Model.................................................................. 26
3.2.3 Activities ............................................................................................. 26
3.2.4 Outputs .............................................................................................. 26
3.2.5 Proximal Outcomes.............................................................................. 26
3.2.6 Distal Outcomes .................................................................................. 27
3.2.7 External Factors .................................................................................. 27
3.2.8 Expanding the Impact Logic Model ....................................................... 27
3.3 Cost Effectiveness Review............................................................................... 36
4 Conclusions and Recommendations .................................................................. 38
4.1 Conclusions.................................................................................................... 38
4.1.1 Key Impact Findings ............................................................................ 38
4.1.2 Key Process Findings ........................................................................... 38
4.2 Recommendations .......................................................................................... 39
5 Appendices ........................................................................................................ 41
5.1 Data Collection Instruments ............................................................................ 41
5.1.1 ComEd Residential Appliance Recycling Participant Survey ..................... 41
5.1.2 ComEd Interview Guide – Sharon Madigan – 03/31/09 .......................... 72
5.1.3 JACO Interview Guide – Michael Dunham– 05/06/09 ............................. 74
5.1.4 JACO Interview Guide – Sam Sirkin – 04/14/09 ..................................... 75
Summit Blue Consulting, LLC December 2, 2009 Final 1
EXECUTIVE SUMMARY – APPLIANCE RECYCLING
Evaluation Objectives
The goal of this report is to present a summary of the findings and results from the evaluation of the
Program Year 1 (PY1) Residential Appliance Recycling (AR) program. The objectives of the evaluation
are to: (1) quantify net energy and peak demand savings impacts from the program during Program Year
1 (PY1); and (2) to determine key process-related program strengths and weaknesses and provide
recommendations to improve the program.
Evaluation Methods
To estimate gross energy savings, we have relied heavily on data from secondary sources, including, for
refrigerators and freezers, regression equations for estimating refrigerator and freezer Unit Energy
Consumption (UEC) that are based on a large database of over 1,600 previously metered units in
California based on the DOE lab metering approach. The regression equations estimate usage as a
function of unit characteristics (age, size, configuration, and defrost mode). The characteristics of units
collected by JACO for ComEd were then input into these models to estimate full-year UECs (representing
kWh savings) that are specific to ComEd’s program.
Our sole primary data collection activity for gross impact calculations was a telephone survey of program
participants to determine a part-use factor. This factor was used to adjust these annualized UEC estimates
to reflect the number of months the recycled unit would have been operated absent the program. This
element of the calculation is particularly important for ComEd’s program, since refrigerators and freezers
located in garages may have been shut down during the winter months, when cold weather reduces or
eliminates the need to run the unit. Similarly, room AC units may only have been operated during a few
of the hottest days during the summer months. In addition to providing information on part-use, this
phone survey also gathered information to support the net-to-gross and process evaluation elements of the
evaluation.
Table 1 below summarizes the key data collection activities in support of this evaluation.
Summit Blue Consulting, LLC December 2, 2009 Final 1
Table 1. Data Collection Activities
Data Collection Targeted Sample Sample Design Sample Size Timing
Type Population Frame
Tracking Data All Program Tracking - All Ongoing
Analysis Participants Database
In-depth Phone ComEd Contact AR program 2 April 14, 2009
Interviews program from managers Sharon
manager ComEd Madigan
Program Contact JACO program 2 Sam – Apr.
Implementers from implementers – 14,2009, Michael
ComEd Sam Sirkin and – May 6, 2009
Michael Dunham
CATI Phone Program Tracking Random Sample 100 Total – August 2009
Surveys Participants Database of AR Program 70 Refrigerator,
Participants 30 Freezer
Recyclers
Key Findings and Recommendations
The Residential Appliance Recycling program began operation in June 2008. The program offers free
pickup and recycling services for older, working refrigerators and freezers, and room air conditioners that
households no longer want. Program savings are based on the accelerated removal, dismantling and
recycling of these older, inefficient units. In exchange for participating in the program, ComEd pays
participants $25 each for up to two recycled refrigerators or freezers. Operational room air conditioner
(RAC) units are also eligible for pick up and recycling, but they can only be picked up from sites where
the recycler, JACO, is already collecting a refrigerator and/or freezer (so the room AC unit can “ride for
free”). Participants contributing these working room AC units also receive the $25 program rebate.
A total of 11,979 units were picked up by the program during PY1. About 70% of these units were
refrigerators, another 26% were freezers, and the remaining 4% were room air conditioners. Table 2
below provides the breakdown of recycled units by measure type.
Summit Blue Consulting, LLC December 2, 2009 Final 2
Table 2. Summary of Recycled Units by Appliance Type
Measure Number of Percent of
Type Units Units
Refrigerators 8,437 70%
Freezers 3,076 26%
Room Air 465 4%
Conditioners
Unlabeled 1 0%
Total Units 11,979 100%
Recycled
Table 3 below provides the first-year evaluation-adjusted gross and net savings estimates for each
measure and for the program overall. Table 4 shows the comparable values for kW savings.
Table 3. PY1 Gross and Net Impact Parameter and Savings Estimates (kWh)
Gross and Net Impact Refrigerators Freezers Room Total
Parameter and Savings AC Program
Estimates
Total units recycled through the 8,438* 3,076 465 11,979
Program
Verified Annual kWh Savings
Impacts
Verified annual Gross kWh 1,893 2,027 ---
savings per unit (full-load
operating hours)
Part-Use Factor 75% 59% ---
Verified annual Gross kWh 1,420 1,196 80 --
savings per unit adjusted for part-
use
Verified Program Gross MWh 11,982 3,678 37 15,698
Net-to-Gross Ratio (1-Free Rider 0.70 0.83 1.00
%)
Total First-Year Evaluation- 8,388 3,053 37 11,478
Adjusted Net MWh Savings
*Includes one unit that was unlabeled.
Summit Blue Consulting, LLC December 2, 2009 Final 3
Table 4, PY1 Gross and Net Impact Parameter and Savings Estimates (kW)
Gross and Net Impact Parameter and Verified kW Savings
Savings Estimates
Refrigerators Freezers Room Total
AC Program
Total units recycled through the Program 8,438 3,076 465 11,979
Verified kW Savings Impacts
Annual kW savings per unit (full-load 0.30 0.26 0.04 ---
operating hours)
Program Gross kW 2,531.4 799.8 18.6 3,350
Net-to-Gross Ratio (1-Free Rider %) 0.7 0.83 1.00
Total PY1 Net kW Savings 1,772 664 19 2,454
Key Impact Findings
The PY1 net energy savings goal for this program was 8,159 MWh and the program-reported energy
savings was slightly higher than this, 8,528 MWh 1 . The verified energy savings was actually significantly
higher than this – 11,478 MWhs, for an overall realization rate of 135%. For year 1, the kW saved by the
program are based on ComEd’s ex-ante planning estimates for per-unit kW savings for Refrigerators,
Freezers and Room AC units.
A comparison of program verified versus program-tracking system savings (as a proxy for the ComEd
savings claim) is provided in Table 5 below. Please note that the program-tracking system savings value
of 8,545 MWh is slightly higher than the savings reported by ComEd of 8,528 MWhs. The program
tracking savings estimate was computed using the deemed savings formula provided by ComEd in its
program plan. Although these numbers are very close, we were unable to completely match ComEd’s
reported savings estimate.
Gross savings per unit are fairly similar to the program planning estimate of kWh savings for
refrigerators. The verified gross savings estimate is somewhat higher than the ex-ante estimate for
freezers, which reflects the fact that the program collected more older units than anticipated in PY1. Fully
40% of the freezers picked up by the program are over 30 years old and another 42% are between 20 and
30 years old. Nearly all (92%) of the freezers collected by the program were manufactured before the
1993 standards change. The standards change resulted in a dramatic improvement in efficiency. Pre-1993
units are generally considered ‘energy hogs’ that use 3 to 4 times the energy of units made since the
standards change.
1
As reported in Table 2: Program Savings and Costs versus Plan, Program Year 1 of ComEd’s Annual Energy
Efficiency report filed with the Illinois Commerce Commission.
Summit Blue Consulting, LLC December 2, 2009 Final 4
Table 5. PY1 Program Tracking System Savings Versus Evaluation-Verified Savings (MWh and kW)
Program Tracking System Savings Verified Program Savings
Gross and Net Impact Parameter Room Total Room Total
and Savings Estimates Refrigerators Freezers AC Program Refrigerators Freezers AC Program
Total units recycled through the
Program 8,438 3,076 465 11,979 8,438 3,076 465 11,979
Annual kWh Savings Impacts
Annual Gross kWh savings per unit
(full-load operating hours) 1,946 1,662 80 --- 1,893 2,027 ---
Part-Use Factor --- --- --- 75% 59% ---
Annual Gross kWh savings per unit
adjusted for part-use 1,946 1,662 80 -- 1,420 1,196 80 --
Program Gross MWh 16,420 5,112 37 21,570 11,982 3,678 37 15,698
Net-to-Gross Ratio (1-Free Rider
%) 0.35 0.54 1.00 0.70 0.83 1.00
Total PY1 Net MWh Savings 5,747 2,761 37 8,545 8,388 3,053 37 11,478
Program Gross kW 2,531.4 799.8 18.6 3,350 2,531.4 799.8 18.6 3,350
Net-to-Gross Ratio (1-Free Rider
%) 0.7 0.83 1.00 0.7 0.83 1.00
Total PY1 Net kW Savings 1,772 664 19 2,454 1,772 664 19 2,454
Summit Blue Consulting, LLC December 2, 2009 Final 5
The primary reason for this higher-than-expected performance was a verified value for the NTG ratio
(including the part-use adjustment) of 0.52 that was considerably higher than the value assumed by the
program for refrigerators of 0.35. This has a significant impact, since refrigerators are the largest source
of energy savings for the program. The improvement in this factor alone is the main reason for
evaluation-verified net savings that are significantly higher than those claimed by the program in PY1.
Evaluation verified NTG ratios (excluding the part-use adjustment) are 0.70 for refrigerators and 0.83 for
freezers. When compared with the ex-ante NTG ratios applied by the program, these values are
considerably higher. However, the program applied NTG ratios also included the effect of the part-use
factor, which has been decoupled from the evaluation-verified values, and has been reflected in the gross
saving estimates (where it more appropriately belongs).
However, it is useful to provide an apples-to-apples comparison of the program assumed NTG values
with the evaluation verified NTG values to provide insight into the magnitude of any ex-post adjustments.
For this limited purpose, Table 6 below presents a comparison of the program NTG ratio assumptions and
evaluation verified NTG values, assuming in both cases full inclusion of the part-use factor. While the
refrigerator NTG value was dramatically improved from the planning estimate, the Freezer value was
lower, and the Room AC value was the same as the program assumption.
Table 6. Comparison of the Ex-Ante and Evaluation Verified NTG Values
Net-to-Gross Ratio Adjusted for Part-Use Refrigerators Freezers Room AC
Evaluation Net-to-Gross Ratio (1-Free Rider %) 0.70 0.83 1.00
unadjusted for Part Use
Part-Use Factor 75% 59% 100%
Program Verified NTG ratio adjusted for Part-Use 0.53 0.49 1.00
Program Applied Ex-Ante Value 0.35 0.54 1.00
Key Process Findings
The amount of marketing that was done for this campaign was sufficient to achieve the target goal for the
number of appliances picked up. The primary marketing tool, bill inserts, was also effective. When asked
unprompted where they had heard of the program, nearly three of four participants (74%) recalled seeing
the program mentioned in a bill insert with over two-thirds (69%) saying that was where they first learned
of the program. When prompted, another 11% recalled seeing the program in a bill insert bringing both
unprompted and prompted recall of bill inserts to 85%.
Participants were asked, unprompted, why they chose the ComEd Appliance Recycling Program to
dispose of their appliance instead of some other disposal method. The convenience of the home pick-up
was the main selling point of the program for more participants than any other reason. An additional 13%
said the home pick-up was a secondary reason. The $25 cash incentive was also a factor, but it plays more
of a secondary role with 25% saying it was the main reason and an additional 28% a secondary reason.
Overall, 96% of participants were satisfied with their experience with the Appliance Recycling Program,
with 86% saying they were “very satisfied”.
The program was well-administered. Participants reported a high degree of satisfaction with the sign-up
process and appliances were picked up and payments processed in timely fashion. The implementation
plan stipulates that 90% of customers will not have to wait for more than 14 days to have their appliance
Summit Blue Consulting, LLC December 2, 2009 Final 6
to be picked up. A large majority of participants surveyed (84% of those who could recall) scheduled a
pick-up date within two weeks of when they called. Nearly all participants surveyed (98%) said they were
able to schedule a pick-up date that was convenient for them.
Overall, 92% of respondents were satisfied with the collection team who came to pick up the appliance,
with 80% reporting that they were “very satisfied”. Only two respondents said they were dissatisfied, with
the reason being that the collection team was not careful removing the appliance.
Three of four respondents (74%) said that they were very satisfied with the amount of the incentive
payment. No respondents reported being dissatisfied with the size of the payment. Of the participants who
recalled when they received their incentive check, 91% said that they received payment within four weeks
of pickup. Only one respondent was dissatisfied with the amount of time it took to receive payment (4
weeks).
Summit Blue Consulting, LLC December 2, 2009 Final 7
1 INTRODUCTION TO THE PROGRAM
1.1 Program Description
The Residential Appliance Recycling program was designed to achieve energy savings through the
retirement and recycling of older, inefficient refrigerators, freezers, and room air conditioners. The
primary objectives of the program are to:
Decrease the retention of high energy-use refrigerators and freezers; and
Deliver long-term energy savings.
A secondary objective is to dispose of these older refrigerators and freezers in an environmentally safe
manner by offering comprehensive toxic material recycling and disposal that conforms with applicable
environmental laws and regulations and permitting requirements.
The Residential Appliance Recycling program began operation in June 2008. The program offers free
pickup and recycling services for older, working refrigerators and freezers, and room air conditioners that
households no longer want. Program savings are based on the accelerated removal, dismantling and
recycling of these older, inefficient units. In exchange for participating in the program, ComEd pays
participants $25 each for up to two recycled refrigerators or freezers. Operational room air conditioner
(RAC) units are also eligible for pick up and recycling, but they can only be picked up from sites where
the recycler, JACO, is already collecting a refrigerator and/or freezer (so the room AC unit can “ride for
free”). Participants contributing these working room AC units also receive the $25 program rebate.
ComEd hired JACO to be the program implementer for the Appliance Recycling program. JACO is
responsible for the following functions:
Appliance pickups and related scheduling
Processing program enrollments
Deconstructing and recycling program units
Responding to customer questions and complaints
Program tracking and reporting functions
The program is marketed through a combination of methods – bill stuffers, radio and TV spots,
newspaper ads, and word-of-mouth.
1.2 Evaluation Questions
The evaluation sought to answer the following key researchable questions. Some of the researchable
questions will be addressed in Program Years 2 and 3.
Impact Questions
1. What are the gross impacts from this program?
2. What are the net impacts from this program? What is the level of free ridership with this
program? What is the level of participant spillover? How can free ridership be reduced?
Summit Blue Consulting, LLC December 2, 2009 Final 8
3. Did the program meet its energy and demand goals? If not, why not?
Process Questions
1. Has the program as implemented changed from the plan filed on November 15, 2007? If so, how,
why, and was this an advantageous change?
2. What are key barriers to participation in the program for eligible ComEd customers? How can
they be addressed by the program?
3. How do customers become aware of the program? What marketing strategies could be used to
boost program awareness?
4. Is the program outreach to customers and program partners effective in increasing awareness of
the program opportunities?
a. What is the format of the outreach?
b. How often does the outreach occur?
c. Are the messages within the outreach clear and actionable?
5. Are program incentive levels appropriate to encourage participation?
a. What is the influence of the incentive level versus the marketing effort on program
participation levels?
b. How should the budget allocation between incentive spending and marketing
spending be adjusted to maximize participation?
Summit Blue Consulting, LLC December 2, 2009 Final 9
2 EVALUATION METHODS
This section describes the analytic methods and data collection activities implemented as part of the 2008
process and impact evaluation of the Appliance Recycling program, including the data sources and
sample designs used as a base for the data collection activities.
2.1 Analytical Methods
Gross Program Savings
Refrigerators and Freezers. Gross energy savings are expressed in terms of Full-year Unit Energy
Consumption (UECs). UEC estimates were made using a regression-based approach that models full-year
energy savings as a function of unit age, size, configuration, and defrost mode. These regression
equations are based on a large body of impact evaluation work that has already been completed in
California, which rely on DOE lab metered results for over 1,600 units. The regression equations were
applied to the characteristics of the population of units actually collected by JACO. In addition, gross
savings estimates were adjusted for part-use, by applying findings from the phone survey of program
participants.
The regression equation that was used to estimate gross unit savings for recycled refrigerators and
freezers is shown below in Table 7. This equation is from the recently completed evaluation of
California’s 2004-05 Appliance Recycling programs, and is based on a large database of over 1,600
previously metered units in California based on the DOE lab metering approach. The regression equation
estimates usage as a function of unit characteristics (age, size, configuration, and defrost mode). All of the
required data inputs to this equation were obtained from the program tracking data, except for the label
amps which was imputed from other unit characteristics using a simplified computer model.
Table 7. Regression Relating DOE Test Annual UEC for Recycled Appliances to
Explanatory Variables
Variable Description Coefficient t-value
Intercept -422.4106 -0.77
Freezer dummy (=1 if freezer) 169.0536 1.84
Bottom freezer dummy (=1 if unit is bottom freezer) 595.3794 2.91
Side by side dummy (= 1 if unit is side-by-side) -129.3553 -0.34
Single door dummy (= 1 if unit is single door) -417.1026 -4.73
Frost free dummy (= 1 if unit is frost free) -445.0348 -1.00
Natural log of unit age 405.2134 2.15
Cubic Feet of unit (per tracking system data) 43.6478 4.59
Label Amps 104.1018 4.83
Freezer dummy x frost free dummy 319.1097 1.94
Bottom freezer dummy x frost free dummy -302.0484 -1.28
Summit Blue Consulting, LLC December 2, 2009 Final 10
Variable Description Coefficient t-value
Side by side dummy x frost free dummy 1451.3206 3.80
Side-side dummy x amps -126.4332 -2.88
Frost free dummy x ln(age) 299.8206 2.09
Dummy if unit age is 15 years or greater 1197.8349 2.61
Ln age x age 15 up dummy -524.9782 -3.08
These estimates reflect the full-year Unit Energy Consumption or UEC.
Part-Use Adjustment. This full-year UEC value was then adjusted for part-use, based on self-reported
findings from the completed telephone surveys. This adjustment pro-rates the full-year value for the
proportion of the year that the unit would have been operated in the program’s absence. The value of this
adjustment was calculated directly from phone survey responses regarding the number of months during
the year that the participant indicated the appliance would have been operated if the program had not
picked it up. Average part-use factors were calculated across all respondents, separately for refrigerators
and freezers.
Room Air Conditioners. The deemed savings memo and procedure called for the energy consumption of
residential room AC units to be estimated using the following equation.
kWh = unit capacity x load x FLEH / (efficiency x 1000)
where:
unit capacity [BTU/h] is a nameplate value
load [dimensionless] is assumed to be 1.0 with partial loading accounted for in FLEH
FLEH (full-load equivalent hours) [hours] is basically the compressor run-time if we assume window
AC units are generally a two-state device – on or off.
Efficiency [Btu out / Watts in] or EER for equipment of this type
1000 is the conversion factor from Watts to kW
Net Savings Analysis
The primary objective of the net savings analysis for the Appliance Recycling program is to determine the
program's net effect on customers’ electricity usage. This requires estimating what would have happened
in the absence of the program. Thus, after gross program impacts adjusted for part-use have been
assessed, net program impacts are derived by estimating a Net-to-Gross (NTG) ratio which quantifies the
percentage of the gross program impacts that can reliably be attributed to the program. A customer self-
report method, based on data gathered during participant phone surveys, was used to estimate the NTG
ratio for this evaluation.
For PY1, the net program impacts were based solely on the estimated level of free-ridership in the
program. In this program, free ridership is defined based on the percentage of program participants that
would have disposed of their units absent the program in a manner that would have permanently removed
the unit from the grid. This includes participants who indicated they would have otherwise:
Sent the unit to a recycling facility, or
Taken the unit to a landfill
Summit Blue Consulting, LLC December 2, 2009 Final 11
Participant spillover was not assessed and does not fall within the scope of this program or this
evaluation.
1.1 Data Sources
Table 8 below summarizes the key data collection activities in support of this evaluation.
Table 8. Data Collection Activities
Data Targeted Sample Sample Design Sample Size Timing
Collection Population Frame
Type
Tracking All Program Tracking - All Ongoing
Data Analysis Participants Database
In-depth ComEd Contact from AR program 2 April 14, 2009
Phone program ComEd managers Sharon
Interviews manager Madigan
Program Contact from JACO program 2 Sam – Apr. 14,2009,
Implementers ComEd implementers – Michael – May 6,
Sam Sirkin and 2009
Michael Dunham
CATI Phone Program Tracking Random Sample 100 Total – August 2009
Surveys Participants Database of AR Program 70 Refrigerator,
Participants 30 Freezer
Recyclers
Below is a summary of how each of these data sources was used in the specific components of the
evaluation study.
Impact Evaluation
Estimation of gross savings/UECs. All of the required data inputs to the regression equation
used to develop final estimates of gross unit energy consumption for refrigerators and
freezers were obtained from the program tracking database, except for the label amps which
were imputed from other unit characteristics using a simplified computer model. The phone
survey also obtained several of these same characteristics. However, because they were based
on self-reported information, rather than the results of a visual inspection of the units picked
up by the program, they were deemed less reliable than the tracking data which was
ultimately used for the calculation.
Estimation of the Part-use factor and Net-to-gross ratio. Self-reported findings from the
telephone survey were the sole data source for both the part-use factor and the net-to-gross
ratio.
Process Evaluation
The process evaluation relied primarily on two data sources, program staff interviews, and
telephone surveys of program participants.
Summit Blue Consulting, LLC December 2, 2009 Final 12
o Program Staff interviews. The interview with the Appliance Recycling Program
Manager focused on program processes in order to better understand the goals of the
program, how the program was implemented, the perceived effectiveness of the program,
and also verified evaluation priorities. The interviews with the JACO managers focused
on the recycling process and the details of the appliance pickup.
o Telephone surveys. The process evaluation component of the surveys obtained
information on sources of program awareness, program satisfaction, rebate satisfaction,
and awareness of program features (e.g., rebates, technical assistance, marketing
materials).
2.2 Sampling Plan
The sample of Appliance Recycling participants was randomly selected from the Program Tracking
Database provided by ComEd. Basic data cleaning steps were undertaken before the sample was pulled
from the database so that for example, records with missing or invalid phone numbers were removed.
These records could not be included in the surveying efforts but were included in the final impact results.
The sample was stratified by appliance type and quotas were set based on the proportion of each
appliance in the general population. Therefore, no weights are necessary for the data analysis. In total,
9,936 pieces of sample were sent to ODC in order to complete the survey. ODC was then instructed to
randomly select and dial participants until they had reached the following quotas – 70 refrigerator
recyclers and 30 freezer recyclers, for a total of 100 completed surveys. A small portion of those in each
of these quotas was also expected to have recycled a room air conditioner.
2.3 Sampling Error
Table 9. PY1 Sample Size and Population Level Sampling Error
Population Population Size Sample Size Sampling Error
(N) (n) (90% CI)
Recycled Refrigerators 8,437 70 8.5%
Recycled Freezers 3,076 30 12.9%
Totals 11,513 100 7.1%
Survey Disposition
Table 10 shows the final dispositions for the 498 program participants we attempted to contact for this
evaluation. As the table shows, we completed interviews with 100 participants, or 20%. We were unable
to reach 44% for a variety of reasons such as a no one answering, an answering machine, or a busy signal.
Another 19% requested to be called back later to complete the survey but did not end up doing so. 2 There
2
Often, participants who are not inclined to participate do not outright refuse. Instead they agree to be called back,
but when called back, the time is once again inconvenient. These participants are typically called a number of times,
but many never complete a survey so that their final disposition is “call back”.
Summit Blue Consulting, LLC December 2, 2009 Final 13
were problems with the phone number, such as a disconnected number, for 8%. Only 4% of participants
who answered refused to participate in the survey.
Table 10. Sample Disposition
Sample Disposition Customers %
Participants Attempted to Contact 498 100%
Completes 100 20%
Appliance not picked up 5 1%
Electric company not ComEd 2 <1%
Quota on refrigerator met 13 3%
Respondent unaware of appliance details 1 <1%
Refusal 19 4%
Unable to Reach 220 44%
Language Barrier 3 1%
Phone Number Issue 38 8%
Non-Specific Callback/Appointment Scheduled 97 19%
Participants Attempted to Contact 498 100%
As outlined in Table 11, interviews were attempted with 129 participants with 100 completed surveys.
The remaining 29 did not complete full surveys for several reasons including the quota was filled on
refrigerators (n=13), participants terminated mid-interview (n=8), the participant claimed they signed up
for the program but the appliance was never picked up because they chose not to participate (n=5), or
ComEd was not their electric utility (n=2). For these latter two categories, we cannot say if the participant
database included some people in error or, more likely, these respondents had recall problems.
Table 11. Survey Contacts Disposition
Survey Contacts Disposition Customers %
Customers Surveyed 129 100%
Completed Interview 100 78%
Appliance not picked up 5 4%
Electric company not ComEd 2 2%
Quota on refrigerator met 13 10%
Respondent unaware of appliance details 1 1%
Mid-Interview Terminate 8 6%
Summit Blue Consulting, LLC December 2, 2009 Final 14
3 PROGRAM LEVEL RESULTS
This section presents the results of the impact and process evaluations of the Appliance Recycling
program.
3.1 Impact Evaluation Results
3.1.1 Verification and Due Diligence
This section provides the results of the evaluation of ComEd’s Verification and Due Diligence of the
Appliance Recycling program. Under this task, we explored the quality assurance and verification
activities currently carried out by program and implementation staff. We compared these activities to
industry best practices 3 for similar residential and C&I programs to determine:
1. If any key quality assurance and verification activities that should take place are currently not
being implemented.
2. If any of the current quality assurance and verification activities are biased (i.e., incorrect
sampling that may inadvertently skew results, purposeful sampling that is not defendable, etc.).
3. If any of the current quality assurance and verification activities are overly time-consuming and
might be simplified or dropped.
Data Collection
This assessment primarily relied on in-depth interviews with program and implementation staff and
documentation of current program processes, where available.
Results
Customer Eligibility
Appliance Recycling Program applications are collected electronically through the ComEd website,
JACO's call center, and Abt Appliance's staff members. During the customer application process,
customers provide their name, address, phone number, ComEd account number, make, model, size, age,
and color of appliance. The customer's address, phone number, and account number is then used to verify
eligibility. While collecting participant information electronically will cut down on human error, data still
needs to be verified through quality assurance and control checks. Currently, Appliance Distribution
collection staff performs quality checks on participant information when they have participants read and
sign the Appliance Turn-In Order (ATO) form.
3
See the Best Practices Self Benchmarking Tool developed for the Energy Efficiency Best Practices Project:
http://www.eebestpractices.com/benchmarking.asp.
Summit Blue Consulting, LLC December 2, 2009 Final 15
Assessment: ComEd verifies that an applicant is a ComEd customer during the application process using
the customer's contact information and account number. In addition, customer information is verified
during the collection process. ComEd’s procedures for the verification of customer eligibility are
successful in ensuring only specified customers participate in the programs. No changes are needed in this
area.
Appliance Eligibility
Product eligibility for the Appliance Recycling Program is clearly defined in the Program Manual.
Eligible refrigerators and freezers have to be:
Between 10 to 30 cubic feet
Empty
In working condition
Accessible for removal by the collection team
Room AC units must also be operational and can only be picked up when and where a refrigerator or
freezer is already being harvested. There is a limit of two refrigerators and/or freezers and two room AC
units in order to avoid "junkyard" customers who are not actually using the appliances.
The appliance collection is performed by a minority subcontractor, Appliance Distribution. Collection
staff from Appliance Distribution ensures that these four requirements are met when they arrive for
collection. If any of the requirements are not met, the collection team does not remove the appliance. If
the issue is accessibility, the program participant may reschedule a collection time after they have
provided a clear path for removal.
ComEd and Appliance Distribution management separately go on "ride-alongs" and observe the
collection staff during pick-ups to ensure that the pick-up staff is adhering to these collection
requirements. However there is no formal documentation of these quality assurance checks. At this time,
it is unknown whether Appliance Distribution staff formally document their "ride-alongs."
Assessment: Moving forward, it is recommended that collection "ride-alongs" be scheduled periodically
and with regularity to ensure eligibility requirements are being met consistently throughout the life of the
program. In addition, formal documentation is suggested including a record of how the routes were
sampled and when and where the "ride-alongs" occurred, as well as a review of the collection procedures
performed. Ideally, the routes for a "ride-along" would be sampled on a random basis, but with thought to
the different locations within ComEd's service territory.
Appliance Collection Procedures
The Appliance Recycling Program collection process is handled by the minority subcontractor, Appliance
Distribution. Each step in the collection process is outlined in the program manual. After enrollment, a
participant's pick-up wait time should be no longer than two weeks. Wait times are monitored through
tracking data and evaluated on a monthly basis as one of the program's key performance indicators.
Daily collection routes are optimized through Microsoft Streets and Trips Software. Once the Appliance
Distribution collection staff arrives they check the customers information by having them review and sign
an Appliance Turn-In Order (ATO). The ATO form verifies the customer's name, address, phone number,
ComEd account number, appliance type, make, model, size, pick-up date, and the customer's ownership
of the appliance. The crew then verifies the eligibility of the appliance and removes it from the home.
Summit Blue Consulting, LLC December 2, 2009 Final 16
Once outside the crew disables the appliance by cutting the power cord and destroying the cold control as
well as cutting part of the gaskets out. If the appliance is not eligible, the customer is provided with an
Appliance Recycling Program "spec sheet" which indicates the reasons for the unit not being eligible.
Similar to the quality assurance checks for appliance eligibility, "ride-alongs" are performed by ComEd
and Appliance Distribution to ensure that the collection crew is properly following all Appliance
Recycling Program collection procedures.
In addition, JACO's Appliance Recycling program manager conducted 10-15 customer interviews at the
beginning of the program to confirm that customers were satisfied with the application and collection
process.
Assessment: ComEd has sufficient procedures in place for verifying appliance eligibility. Customer
satisfaction with the program and collection procedures could be strengthened, which we understand will
happen for Program Year 2. JACO will be distributing such a survey when doing pick-ups in August,
September and October of 2009.
With regards to "ride-alongs" please see the assessment provided in the Appliance Eligibility section.
Appliance Recycling Procedures
JACO properly disposes of Appliance Recycling Program appliances using SEG's approach and
equipment. JACO documents at the recycling center all disposal materials including the disposal of
hazardous materials through waste management centers and Univar.
In addition, JACO's staff performs quality assurance checks by visiting the recycling center unannounced
approximately once a month. ComEd performs similar quality assurance checks through unannounced
recycling center visits. While ComEd and JACO both indicate having performed these checks since the
start of the program neither of them have formally documented their visits.
Assessment: In the future, ComEd and JACO should formally document their visits to the recycling
center as part of their regular quality assurance and control procedures. This would allow them to
determine how frequently these visits are currently occurring and if additional visits would be useful to
assure proper recycling procedures were being followed. The Summit Blue team reviewed ComEd’s
quality control and verification procedures for the Appliance Recycling Program and found them to be
good overall.
Summary and Recommendations
Overall, ComEd’s quality control and verification procedures for the Appliance Recycling Program are
good. Table 12 below summarizes the quality assurance and verification activities currently carried out by
the Appliance Recycling Program. It also features recommended changes to current procedures, as well as
suggestions regarding additional activities that ComEd could implement to enhance current quality
assurance and verification.
Summit Blue Consulting, LLC December 2, 2009 Final 17
Table 12. Summary of Quality Assurance Activities in Place and Recommendations
Quality Assurance Activities in Place Recommended Change
Eligibility Checks Formal documentation
Collection “ride-alongs” Periodic collection "ride-alongs" with a
specified frequency
Unannounced recycling center visits Periodic unannounced recycling center
visits with a specified frequency
Customer data checks Periodic data quality reviews for data
quality and completeness
Customer satisfaction interviews Scheduled customer satisfaction interviews
every six months or on-going customer
satisfaction survey
Source: Program manager interviews conducted in April and May 2009. Program documentation.
3.1.2 Tracking System Review
The Appliance Recycling tracking data for 2008 contained 11,979 records, one for each appliance that
was picked up and recycled. This is consistent with the claimed savings estimate which was also based on
this same total of recycled appliances.
Distribution by Appliance Type
About 70 percent of these units were refrigerators, another 26% were freezers, and the remaining 4%
were room air conditioners. Table 13 below provides the breakdown of recycled units by measure type.
Table 13. Summary of Recycled Units by Appliance Type
Measure Number of Percent of
Type Units Units
Refrigerators 8,437 70%
Freezers 3,076 26%
Room Air 465 4%
Conditioners
Unlabeled 1 0%
Total Units 11,979 100%
Recycled
Table 14 below provides a further breakdown of the population stratified by appliance type, of the
number of appliances turned in as reported by the tracking data.
Summit Blue Consulting, LLC December 2, 2009 Final 18
Table 14. Appliance Recycling Program: Appliance Type Versus Number Turned In
Refrigerators Freezers Room AC Number of Number of
Units Applications Participants
1 1 2
1 1 2,470
1 1 2 86
2 2 36
2 1 3 7
1 1 7,078
1 1 2 291
1 1 2 332
1 1 1 3 29
2 2 205
2 1 3 29
2 1 3 1
From these data, we observe the following patterns in terms of the distribution and count by appliance
type:
There are 10,567 unique participants, and most recycled one unit (7,078 refrigerators, 2,470 freezers).
Approximately 600 participants recycled 2 major units (defined as a refrigerator and/or freezer), and
of these, about 10% also recycled a room AC unit.
For room ACs, the majority of participants had AC units that were picked up at the same time as a
refrigerator or freezer, in accordance with program procedures.
In terms of anomalies, we found two types, neither of which resulted in any adjustment to the tracking
data:
There were 2 participants who recycled only a room AC, which the program managers had indicated
would not be allowed.
There was also one participant who recycled 3 major appliances, which exceeded the program limit of
2. We assume they did not receive a rebate for the 3rd unit.
Problems Found
Our review of the tracking data provided to the evaluation team also uncovered some relatively minor
problems, including:
Incomplete records for a number of tracked fields. Most fields were well-populated, and
particularly the most important fields for evaluation (appliance brand, model number, size,
age/year manufactured, defrost type, location at the time of pick up). However, some of the
tracked fields were sparsely populated, or the entry was designated ‘unknown’ or ‘N/A’. These
included:
Unit configuration. This refers to whether a refrigerator is a side-by-side unit, has a freezer at the
top, or at the bottom, or has one door with a freezer inside.
Recycled appliance unit details. This field provides information on the configuration of the
recycled unit – whether it was a side-by-side, top freezer, or bottom freezer unit. It is one of
Summit Blue Consulting, LLC December 2, 2009 Final 19
the key parameters that is used in the regression equation to estimate gross savings, and
unfortunately, it was only populated for a small fraction of the recycled units.
Information on the characteristics of the replacement unit (whether it was new or used, larger or
smaller than the recycled unit).
Information on Room AC model number. In a significant number of cases, the model number was
listed as N/A or ‘unknown’ (typically nameplate was missing, per JACO). This made it
impossible to develop independent savings estimates based on an engineering algorithm for
all units collected by the program.
Lack of data on label amps. One of the types of data needed to calculate gross savings impacts is the
rated amperage of the recycled unit. It was discovered that this data was not formally tracked, but
that JACO has a separate dataset with this information.
Although we were able to complete the evaluation without these incomplete data, it would be better if
they could be more fully populated in the future. We recommend the program tracking data receive
periodic data quality reviews for data quality and completeness. In addition, we recommend that the rated
amperage data be merged with the historical tracking data, and that this value be collected going forward.
Data exported for the evaluation team should also be checked for anomalies.
3.1.3 Gross Program Impact Parameter Estimates
Refrigerators and Freezers
Annualized Unit Energy Consumption (UECs)
As detailed in Section 1, regression based Unit Energy Consumption (UEC) estimates were made for both
refrigerators and freezers. The regression equation estimates usage as a function of unit characteristics
(age, size, configuration, and defrost mode). All of the required data inputs to this equation were obtained
from the program tracking data, except for the label amps which was imputed from other unit
characteristics using a simplified computer model.
Applying the regression coefficients to the full population of units collected through the program during
PY1 and their associated characteristics yielded the following UECs for each type of appliance (Table
15).
Table 15. Estimated UECs
Annualized Refrigerators Freezers
UECs
kWh 1,893 2,027
Both age (in years) and size (in cubic feet) are key explanatory variables that drive these estimates. In
general, the older a unit is, the larger it is and the more electricity it uses. This is the case for 2 reasons:
1. Because of a change in standards in 1993, units built since that time are much more energy
efficient and generally smaller than units made prior to the standards change.
2. There is degradation of a unit’s efficiency over time, as the unit ages.
Summit Blue Consulting, LLC December 2, 2009 Final 20
Because this is a brand-new program, the appliances collected during PY1 have been primarily older and
larger units than those collected via a more established program (as in California). Table 16 and Table 17
below provide the age and size characteristics of the units collected in PY1 through ComEd’s program.
Table 16. Age Characteristics of Recycled Appliances
Age in Years N
Appliance Type
Over 40
11 to 15
16 to 20
21 to 25
26 to 30
31 to 35
36 to 40
6 to 10
0 to 5
Refrigerators 0% 5% 10% 21% 20% 19% 9% 4% 12% 70
Freezers 0% 3% 5% 13% 19% 22% 16% 10% 13% 30
Room Air 0% 5% 7% 18% 37% 18% 5% 6% 5% —
Conditioners
Table 17. Size Characteristics of Recycled Appliances
Appliance Type 10 cubic 11 to 15 16 to 20 21 cubic N
feet and cubic cubic feet and
smaller feet feet larger
Refrigerators 9% 26% 41% 25% 70
Freezers 13% 37% 39% 10% 30
From these data, the following observations can be made:
Age
Fully 64% of refrigerators and 79% of freezers are over 20 years old
Approximately 40% of refrigerators and freezers are between 21 and 30 years old
One-fourth of refrigerators (25%) and nearly 40% of freezers are over 30 years old
The following percentages of appliances collected by the program were made before the 1993
standards change: 85% of refrigerators and 92% of freezers
Size
The majority of units collected are 16 cubic feet and larger, one fourth of refrigerators are larger
than 20 cubic feet
Recycled refrigerators tend to be larger on average than recycled freezers
The size distribution of freezers collected by the program is more diverse than refrigerators. The
most common freezer sizes are between 11 and 20 cubic feet, while those for refrigerators
range from 11 cubic feet to over 20 cubic feet.
One would expect the stock of these unwanted older appliances to decline over time, as the program
matures and the base of these very old, inefficient units available for recycling is reduced. This has
implications for the expected average UECs of units collected by the program in subsequent years, which
would likely be somewhat less than in PY1.
Summit Blue Consulting, LLC December 2, 2009 Final 21
Part use factors. The part-use factors account for the fact that a unit that would have stayed in use would
have been in use only part of the time. For example, the savings due to removal of a unit that would have
been used only three months of the year is only one-quarter (3/12) the savings associated with full-year
use (assuming essentially constant use over the year for a full-use unit). The part-use factor is used to
adjust gross savings UECs to yield estimates of annualized gross savings that can be attributed to the
program. The part-use factors are taken from the results of the telephone survey of participants.
Refrigerators. The assumption is that any refrigerator that would otherwise have been kept in use would
have been used as a secondary, not as a primary refrigerator. Therefore, the part-use for all primary
refrigerators that would otherwise have been kept is set at the average part-use reported by participants
who disposed of a secondary refrigerator. This part-use was the number of months, divided by 12, that the
participant reported the unit would have been plugged in and running had the program not picked it up.
This average was determined to be 75% or 0.75.
Freezers. For freezers, the average part-use is based on a similar question for all participants who
disposed of a freezer. This average was determined to be 59% or 0.59. The supplemental data collected in
the survey provide no further insight into the part-year usage, nor do the tracking data.
Table 18 below reports the distribution of unit usage by appliance type and frequency of use for both
refrigerators and freezers. The majority of participants claim they would have used the unit ‘always’ if the
program had not picked it up.
Table 18. Frequency of Usage in the Absence of the Program
Appliance Type Never 1 to 3 4 to 6 7 to 9 10 to 12 Always N
months months months months
Refrigerators 10% 14% 7% 0% 1% 67% 70
Freezers 23% 20% 3% 0% 0% 53% 30
Gross Savings (UEC) Impacts Adjusted for Part-Use
The next step is to develop gross savings estimates for each type of appliance adjusted for part use. The
application of the part-use factor reduces refrigerator savings/unit to 1,420 kWh per year, and freezer
savings/unit to 1,196 kWh/year. These estimates are provided in Table 19 below.
Table 19. Gross Savings (UECs) Adjusted for Part Use
Appliance Type Gross Part- Adjusted
Savings Use Gross
(UECs) Factor Savings
(kWh/unit)
Refrigerators 1,893 75% 1,420
Freezers 2,027 59% 1,196
Room Air Conditioners
The deemed savings memo and procedure called for the energy consumption of residential room AC units
to be estimated using an engineering algorithm. However, upon reviewing the program tracking data, it
Summit Blue Consulting, LLC December 2, 2009 Final 22
was discovered that none of the required data are tracked. In addition, only one survey was completed
with a participant that turned in a room AC 4 ; meaning we had insufficient data to complete an analysis of
partial use in the absence of the program, or to calculate the net-to-gross ratio.
Therefore, we have elected to accept ComEd’s ex-ante savings estimation procedure and claimed savings
for PY1 and to prioritize a room AC data collection effort in PY2 that provides robust enough data to
perform the above calculation.
3.1.4 Gross Program Impact Results
Table 20 below provides the first-year evaluation-adjusted gross savings estimates for each measure. The
resulting verified total program gross savings quantity is 15,698 MWh 5 . This value includes the
application of the part-use factor. Verified gross kW savings are 3,350 kW. For year 1, these have been
based on ComEd’s ex-ante planning estimates for per-unit kW savings for each measure type.
Table 20. PY1 Gross and Net Impact Parameter and Savings Estimates
Gross and Net Impact Refrigerators Freezers Room Total
Parameter and Savings AC Program
Estimates
Total units recycled through the 8,4381 3,076 465 11,979
Program
Verified Annual kWh Savings
Impacts
Verified annual Gross kWh
savings per unit (full-load 1,893 2,027 ---
operating hours)
Part-Use Factor
75% 59% ---
Verified annual Gross kWh
savings per unit adjusted for part- 1,420 1,196 80 --
use
Verified Program Gross MWh
11,982 3,678 37 15,698
Verified Gross kW Savings
Impacts 2,531.4 799.8 18.6 3,350
1
Includes 1 unit that was unlabeled.
4
We had not set a quota for room AC telephone survey respondents, since the number of RAC turn-ins was small
relative to refrigerators and freezers. We also were concerned about the excessive length of a survey for someone
who needed to respond to questions pertaining to 2 or more appliances, as would be the case for a participant that
turned in a room AC unit in addition to a refrigerator or freezer.
5
The kW saved by the program were not examined in this report, but will be estimated in the future.
Summit Blue Consulting, LLC December 2, 2009 Final 23
3.1.5 Net Program Impact Parameter Estimates
Once gross program impacts have been estimated, net program impacts are calculated by multiplying the
gross impact estimate by the Program Net-to-Gross (NTG) ratio. The NTG ratio is equal to 1 minus the
percentage of free riders.
In this program, free ridership is defined based on the percentage of program participants that would have
disposed of their units absent the program in a manner that would have permanently removed the unit
from the grid. This includes participants who indicated they would have otherwise:
Sent the unit to a recycling facility, or
Taken the unit to a landfill
In total, 21 out of 70 refrigerator respondents (30%) and 5 of 30 freezer respondents revealed they would
have used a method to dispose of their unit that would have permanently destroyed it, indicating they are
free riders. Resulting NTG ratios are 0.70 for refrigerators and 0.83 for freezers.
Comparison with NTG ratio applied by the program
When compared with the ex-ante NTG ratios applied by the program, these values are considerably
higher. However, the program applied ex-ante values also included the effect of the part-use factor, which
has been decoupled from the evaluation-verified values, and has been reflected in the gross saving
estimates (where it more appropriately belongs).
However, it is useful to provide an apples-to-apples comparison of the program assumed NTG values
with the evaluation verified NTG values to provide insight into the magnitude of any ex-post adjustments.
For this limited purpose, Table 21 below presents a comparison of the ex-ante and evaluation verified
NTG values, assuming in both cases full inclusion of the part-use factor.
Table 21. Comparison of the Ex-Ante and Evaluation Verified NTG Values
Net-to-Gross Ratio Adjusted for Part-Use Refrigerators Freezers Room AC
Net-to-Gross Ratio (1-Free Rider %) 0.70 0.83 1.00
Part-Use Factor 75% 59% 100%
Program Verified NTG ratio adjusted for Part-Use 0.53 0.49 1.00
Program Applied Ex-Ante Value 0.35 0.54 1.00
This comparison reveals a program-verified value for the NTG ratio (including the part-use adjustment)
that is considerably higher than the value assumed by the program for refrigerators, which are the largest
source of energy savings for the program. The improvement in this factor alone is the main reason for
evaluation-verified net savings that are significantly higher than those claimed by the program in PY1.
3.1.6 Net Program Impact Results
Table 22 below provides the program-level evaluation-adjusted net impact results for the PY1 Residential
Appliance Recycling program. As this figure shows, the ex post program-level first-year net energy
saving estimate resulting from this evaluation is 11,478 MWh, exceeding program claimed estimates by
Summit Blue Consulting, LLC December 2, 2009 Final 24
nearly 3,000 MWh, and resulting in a net realization rate of 135%. Verified kW savings are based on
ComEd’s ex-ante planning assumptions and are therefore identical to tracking system savings.
Table 22. PY1 Net Impact Parameter and Savings Estimates
Gross and Net Impact
Room Total
Parameter and Savings Refrigerators Freezers
AC Program
Estimates
Verified Program Gross MWh 11,982 3,678 37 15,698
Net-to-Gross Ratio (1-Free Rider
%) 0.7 0.83 1.00
Total First-Year Evaluation-
Adjusted Net MWh Savings 8,388 3,053 37 11,478
Net MWh Savings Claimed by
the Program 8,528
Net Realization Rate (MWh) 135%
Calculation of Verified Net kW
Program Gross kW 2,531.4 799.8 18.6 3,350
Net-to-Gross Ratio (1-Free Rider
%) 0.7 0.83 1.00
Total PY1 Net kW Savings 1,772 664 19 2,454
3.2 Process Evaluation Results
The process evaluation component of the Residential Appliance Recycling evaluation focused on
appliance usage data and satisfaction with program processes, including sign up, pickup and receipt of the
refund check. Data sources for the process evaluation include the Participant CATI survey (n=100) and
the in-depth interviews with program staff and program implementers (n=3).
3.2.1 Program Theory
This section contains the program theory, logic model, and performance indicators of the Appliance
Recycling Program. We created this model based on discussions with program management and
implementers as well as program documentation. The program theory and logic models is to be used:
As a communication tool by
allowing the implementer to show reasoning to other stakeholders
bringing common understanding between implementer and evaluator
As an evaluation tool to
Focus evaluation resources
Summit Blue Consulting, LLC December 2, 2009 Final 25
Clearly show what evaluation will do and expected answers from evaluation
Provide a way to plan for future work effort
The logic model (LM) is a graphic presentation of the intervention – what occurs and clear steps as to
what change the activities undertaken by the intervention are expected to bring about in the targeted
population. Logic models can be impact or implementation oriented. An impact model is sparse in terms
of how the programs works, but clearly shows the outputs of the program and what they are aimed at
affecting. Outcomes are changes that could occur regardless of the program and should be written as such.
The implementation model is how the program works and typically resembles a process flow chart. The
attached model is an impact model.
We use numbered links with arrows between each box in the logic model. These numbers allow us to:
Clearly discuss different areas of the model
Describe why moving from one box to the other brings about the description in the later box
Set up hypotheses for testing of specific numbered links
Explicate what we will and will not be testing within the evaluation
The program theory (PT) is a description of why the intervention is expected to bring about change. It
may reference theories of behavioral change (e.g., theory of planned behavior, normative theory) or be
based on interviews with the program managers as they describe their program.
3.2.2 Creation of the Logic Model
There are several different “looks” to logic models. For this evaluation, we are using a multi-level model
that has a generic statement about resources in the header, activities in the first row, outputs of those
activities in the second row, and outcomes in the third (proximal) and fourth (distal) rows. External
factors are shown on the bottom of the diagram.
When we created the boxes in the logic model, we used the following “road-map.”
3.2.3 Activities
These are discrete activities that roll up to a single “box” that is shown in the model. It separates out
activities that may be performed by different groups. Each activity typically has an output. We used
program documentation (implementation plans) and/or discussion with program managers to determine
activities.
3.2.4 Outputs
These are items that can be counted or seen. It may be the marketing collateral of a marketing campaign,
the audits performed by a program, or the number of completed applications. All outputs do not need to
lead to an outcome. We used the same sources as for activities to determine outputs.
3.2.5 Proximal Outcomes
These are changes that occur in the targeted population that the program directly “touches.” Multiple
proximal outcomes may lead to one or more distal outcomes.
Summit Blue Consulting, LLC December 2, 2009 Final 26
3.2.6 Distal Outcomes
These are changes that are implicitly occurring when the proximal outcome occurs. For example, an
energy efficiency program may use marketing to bring about changes in Awareness, Knowledge, or
Attitudes as a proximal outcome, which leads to the distal outcomes of: intent to take actions, which leads
to actual installation of EE equipment, which leads to energy impacts.
3.2.7 External Factors
These are known areas that can affect the outcomes shown, but are outside of the programs influence.
Typically, these are big areas, such as the economy, environmental regulations, codes/standards for
energy efficiency, weather, etc. Sometimes these can arise from our discussions with the program
managers, but often they were thought about and included based on our knowledge.
3.2.8 Expanding the Impact Logic Model
Once the impact logic model was drafted, a table was created that describes the links, the potential
performance indicators that could be used to test the link, the potential success criteria that would indicate
the link was successful, and potential data sources of the link.
When thinking about how to write each of the performance indicators, we asked ourselves “What would
we look at to judge whether the link description actions are occurring” and wrote the answer as the
performance indicator.
Success criteria were created by us and are thought to be reasonable.
Summit Blue Consulting, LLC December 2, 2009 Final 27
Figure 1. Preliminary Logic Model
Summit Blue Consulting, LLC December 2, 2009 Final 28
Table 23. Performance Indicators Table
Link Description of Link Potential Potential Success Criteria Evaluator Data
Performance for Performance Collection
Indicator Indicator Activities
Associated with
Link
1 The appliance recycling program 1. Number of bill 1. At least two bill inserts Interview with
was developed to ensure that inserts sent out to all ComEd program manager
when an appliance is disposed of, 2. Number of press residential customers Review of program
the appliance is taken off of the releases or PR events during PY1 materials
grid completely, not resold into a 3. Marketing 2. Press release sent out
secondary market. In order to let collateral is effective announcing the start of the
customers know about this program and another sent
program, ComEd's marketing during the program year.
team develops and launches At least one PR event.
marketing and public relations 3. All marketing materials,
efforts such as bill inserts, including web-site, are
newspaper ads, and press releases. written clearly and in such
The program is also featured on a way that information is
the ComEd website. These easy to understand.
marketing materials target ComEd
homeowners ages 35-60, living in
single family homes and also
target those customers that are
currently using their spare fridges
or freezers a minimal amount.
2 Marketing collateral provides 1. ComEd customers 1. 90% of participants Participant survey
information in a manner which become familiar with report learning of program
leads to awareness of the program program. opportunities via marketing
and incentive to participate. collateral
3 At the retailer (Abt), employees 1. More customers 1. 100% of eligible Due to the limited
advertise the program to eligible made aware of the customers at the retailer are involvement of the
customers. program through this made aware of the program retailer portion of
retailer channel when purchasing a spare this program, this
fridge or a freezer. evaluation will not
be collecting data on
Abt store customers.
4 Once a customer has applied for 1. Number of units 1. 12,158 refrigerators and Program tracking
an appliance pick-up, a date for collected freezers collected; 1,596 data
collection is scheduled and 2. Participant room AC units collected. Participant survey
collection crews arrive on time for satisfaction with 2. 150 of the appliances
their appointment and pick up the collection process were signed up through
appliance within 2 weeks from the 3. Wait times are Abt.
application date. Employees monitored through 3. 95% of participants are
leaves behind a flyer about other tracking data and satisfied with the collection
ComEd Efficiency Programs to evaluated on a process.
give the customers exposure to monthly basis as one 4. 90% of participants have
other ways to save energy and of the program's key appliances picked up no
save money. performance more than 14 days after the
indicators customer requests a pick-
4.Customer recall for up of the appliance. 75% of
additional ComEd participants have the
energy efficiency appliances picked up
program information. within 10 days of customer
request.5. 90 % of
customers recall
information left behind by
employee about other
programs.
Summit Blue Consulting, LLC December 2, 2009 Final 29
Link Description of Link Potential Potential Success Criteria Evaluator Data
Performance for Performance Collection
Indicator Indicator Activities
Associated with
Link
5 Appliances are collected from 1. Number of units 1. 98% of scheduled Program tracking
customers who had signed up for collected appliance pickups are data
the program. They collect the 2. Paperwork is filled collected.
number of units that they are out correctly and 2. 100% of forms are filled
scheduled to pick up and fill out completely out correctly and
necessary paperwork correctly. completely
6 Units picked up by the retailer's 1. Number of units 1. 100% of units picked up Program tracking
collection crew are delivered to recycled are recycled data
JACO's recycling center and 2. Documentation of 2. 100% of units have Documentation and
promptly recycled. Because units the disposal of documentation of disposal records from JACO
are recycled, they are permanently hazardous materials of hazardous materials recycling facility
removed from the grid.
7 Instead of being used as a spare 1. kWh savings 1. Program meets its kWh Impact analysis
appliance, or resold into goals
secondary markets, used
refrigerators, freezers, and room
AC units are recycled and taken
off of the grid. Customers do not
purchase an additional
replacement appliance. Energy
savings is created because the
appliances are no longer in use.
Changes to Program
There have been some slight changes to the program since the implementation plan was filed on
November 15, 2007.
Between the initial ICC filing and program implementation, ComEd dropped the incentive level from $50
to $25 per appliance. Research done by JACO indicated that $25 was a sufficient incentive. In addition,
the target quantity for the first year of the program was relatively low and ComEd wanted to control the
number of appliances that would be enrolled in the program. This change to the program was an
advantageous change as the program achieved the target quantity of appliances and adequately managed
demand. This incentive level will be monitored at the end of each 12 month period in the program.
After the November 15, 2007 filing, ComEd also added a retailer partnership component to the program.
ComEd partnered with ABT Electronics, a local appliance retailer. This channel accounted for a small
number of units (150 of the 11,300) that were recycled through the program. JACO suggested the
partnership based on other work they were doing with ABT. This was an advantageous change because
partnering with ABT has allowed ComEd another avenue through which to reach customers with an
appliance that could be recycled. ABT also allows for promotional signs to be hung in the stores so the
program is advertised to customers who may have a spare appliance at home but are not purchasing a new
unit. ABT already was in the practice of removing older appliances from customers’ homes, so an added
benefit to this partnership is that those appliances that were previously sold from ABT to a reseller are
now taken off of the grid completely.
Summit Blue Consulting, LLC December 2, 2009 Final 30
Barriers to Participation
Program Year 1 evaluation activities do not include a non-participant survey so it is somewhat difficult to
assess barriers to program participation, particularly since the program met its goals in terms of number of
appliances recycled. Customer awareness of the program is likely a key barrier to participation. This will
be explored in PY2.
Another possible barrier is convincing people who have a spare refrigerator that they are using to give it
up and use just a single refrigerator. We can look at some of the survey responses of the participants who
were currently using their spare refrigerator or freezer to understand what barriers might need to be
overcome to convince this group to participate.
A sizable majority of participants are recycling a spare appliance they are using at least at during part of
the year if not all the time (77%). Looking just at these participants (refrigerator 53%, freezer 24%), we
see that 21% of those recycling a refrigerator and 29% of those recycling a freezer had not been
considering getting rid of their appliance when they learned of the ComEd program. When asked if they
would have kept their appliance or gotten rid of it if the ComEd program did not exist, 40% said they
would have kept the appliance (results are the same for refrigerators and freezers). This compares to 22%
of the rest of the program participants. ComEd customers who are using their spare appliance chose the
program first for the convenience of the home pick-up and second for the $25 incentive. These
motivations for participating are the same as other participants, a topic discussed in more detail below.
Marketing and Promotion Strategy
The marketing and promotion of the Appliance Recycling Program is managed by ComEd, with JACO
involved in an advisory capacity.
For the first year of the program, ComEd deliberately set a somewhat conservative goal for the number of
appliances recycled compared to the potential demand that exists in the residential market. Because the
recycling center was set up specifically for the program, participation levels needed to be carefully
managed as the center began its operation. As a result, the program did not invest in an extensive
marketing campaign. The roll-out of the program in Summer 2008 was ideal as it coincided nicely with
the peak months for home improvement projects The main marketing vehicle for Program Year 1was
through four bill inserts. Advertisements were also placed in the ComEd Energy at Home Newsletter
when the bill inserts were not included. The program used newspaper ads sparingly, but did run some in
the winter months when fewer people consider getting rid of old appliances.
A content review of the marketing material shows the messages to be clear and actionable and are also
consistent among bill inserts, newspaper and newsletter advertisements. Advertisements and bill inserts
are in full color, and with very clear language about the intent of the program (picking up old refrigerators
with no cost to the customer) and prominently display the amount of the incentive ($25). The
advertisements clearly show how to schedule the appointment and also give various explanations about
why someone should get rid of the secondary fridge (“it’s probably costing you more money than you
think.”)
The amount of marketing that was done for this campaign was sufficient given that ComEd reached the
target goal of number of appliances picked up. The bill insert method was also effective. When asked
unprompted where they had heard of the program, nearly three of four participants (74%) recalled seeing
the program mentioned in a bill insert with over two-thirds (69%) saying that was where they first learned
Summit Blue Consulting, LLC December 2, 2009 Final 31
of the program. When prompted, another 11% recalled seeing the program in a bill insert bringing both
unprompted and prompted recall of bill inserts to 85%.
Other sources where participants heard of the program include word of mouth, advertising in the
newspaper, TV, the internet and the ComEd Energy at Home Newsletter.
Table 24. Where Customers Have Heard of the Appliance Recycling Program
Source First heard of Additional Total
program sources (n=100)
(n=100) (n=100)
Bill Insert 69% 5% 74%
Word of Mouth 7% 8% 15%
Newspaper 6% 2% 8%
TV/NEWS 2% 3% 5%
ComEd Energy at Home Newsletter 3% 1% 4%
Internet 2% 1% 3%
Other 6% 2% 8%
Don’t know/No other sources 5% 77% --
For Program Year 2, ComEd has suggested running a direct mail pilot using a profiling database to target
customers with a similar demographic to those that have already participated in the program. They are
also intending to send out six bill inserts and may also run more newspaper advertisements so that they
are a more consistent component of the marketing strategy.
Incentive Level for Participation
The incentive level was dropped from a planned $50 to $25. At this lower level, the program was able to
still successfully achieve the target quantity of appliances and adequately manage demand. The $25
incentive was a motivating factor for over half of participants surveyed (53%) said the $25 was one
reason why they were using the program to dispose of their appliance.
Incentive amounts are reviewed at the end of each 12 month program cycle. Incentive amounts are in line
with other Appliance Recycling Programs, such as PG&E and are adequate for the participation levels
desired for this program.
Participation in the Program
Participants were asked, unprompted, why they chose the ComEd Appliance Recycling Program to
dispose of their appliance instead of some other disposal method. The convenience of the home pick-up
was the main selling point of the program for more participants than any other reason. An additional 13%
said the home pick-up was a secondary reason. The $25 cash incentive was also a factor, but it plays more
of a secondary role with 25% saying it was the main reason and an additional 28% a secondary reason.
Some were just happy they did not have to pay for the pick-up (9% main, 13% secondary). Finally, the
environmental benefits of the program were also a factor but not a large one.
Summit Blue Consulting, LLC December 2, 2009 Final 32
Table 25: Reasons Why Customers Chose the Appliance Recycling Program
Reason Main Add’l Reasons Total
Reason (Mult. Response)
Percent of Respondents (n=100)
Convenience of home pickup 43% 13% 56%
$25 Cash Incentive 25% 28 53%
Recycling/environmentally 16% 7% 23%
friendly
Pick up was free 9% 13% 24%
Other 5% 4% 9%
Don’t know /no other reasons 2% 48% --
Source: Participant Survey
Participant Sign-Up for Program
Participants had a few different options to sign up for the program. They could have called ComEd to set
up an appointment, they could have gone through the ComEd website, or they could have signed up at an
ABT Retailer. A majority of the participants surveyed signed up by telephone (69%) and most others
signed up using the ComEd website (20%).One person reported participating through ABT while the rest
did not recall how they signed up.
Customer satisfaction with the sign-up process is very high, with 92% of participants rating their
satisfaction as a nine or ten on a 0 to 10 point scale where 0 is very dissatisfied and 10 is very satisfied.
Furthermore, no customer surveyed rated the experience lower than a 7 on the 0 to 10 point satisfaction
scale. Participants who signed up via the phone said that the representative was polite and courteous
(100%), answered all of their questions about the program (95%), and that they only needed to call once
to successfully sign up for the program (90%). Participants who signed up online reported that the sign up
screen was easy to find (94% 17/18), and that they received confirmation that the sign up had been
successful (100%).
ComEd strives to ensure that customers are able to schedule an appointment within 7 to 10 days of the
initial contact, unless the customer requests otherwise. The implementation plan stipulates that 90% of
customers will not have to wait for more than 14 days to have their appliance to be picked up. ComEd did
an excellent job achieving this goal, as most customers were able to schedule a pickup date that was
within 2 weeks of the appointment (84% of those who could recall) and were generally very satisfied with
the amount of time between then time they made the appointment and the date it was picked up (82%
very satisfied, 15% somewhat satisfied). Most importantly, 98% of participants said they were able to
schedule a pick-up date that was convenient for them.
Summit Blue Consulting, LLC December 2, 2009 Final 33
Table 26. Time Between Appointment and Pick-Up of Appliance
Amount of Time Total
(n=70)
Less than 1 week 9%
1 week 37%
2 weeks 39%
3 or more weeks 16%
The program is also supposed to promote the energy and environmental benefits of recycling an older
appliance. When learning about the program, 63% of participants said they learned that older refrigerators
and freezers are less efficient and use more energy than newer ones. Just under three of four (74%) said
that they learned that the coolant in the unit would be safely removed and that material that makes up the
appliance would be reused. There was no significant difference between the method used for sign-up and
learning this information.
Appliance Collection Process
JACO Collection crews are instructed to call customers two days ahead to confirm appointments and
remind customers that the appliances are supposed to be plugged in, defrosted and cleaned out. A second
reminder call is to be delivered 30 minutes before the scheduled appointment to serve as a final
confirmation and to also give customers an update if the time has changed due to traffic or weather
conditions. JACO is doing an excellent job meeting this goal as 87% of respondents said that they
received a call in advance to confirm the appointment. The rest of the participants said that either a
confirmation call was not applicable to their situation (3%) or that they could not recall receiving a
confirmation call (10%).
Employees arrived on time for the pickup with 94% of participants reporting that the collection crew
arrived on time, and only two respondents said the crew was late arriving at the appointment.
Overall, 92% of participants were satisfied with the collection team who came to pick up the appliance,
with 80% reporting that they were “very satisfied”. Only two participants said they were dissatisfied, with
the reason being that the collection team was not careful removing the appliance.
Payment Process
The implementation plan stipulates that most incentive checks should be issued within 14 days of the
actual pickup, with the customer receiving the check within four weeks of the pickup. The survey results
indicate that JACO has done an excellent job meeting this goal. Of the participants who recalled when
they received their incentive check, 91% said that they received payment within four weeks of pickup.
Only one customer was dissatisfied with the amount of time it took to receive payment (4 weeks).
As reported earlier, a slight majority of participants said that the incentive payment was a reason why they
participated in the program. Furthermore, three of four participants (74%) said that they were very
satisfied with the payment. No participants reported being dissatisfied with the size of the incentive
payment they received as a result of their participation in the program.
Summit Blue Consulting, LLC December 2, 2009 Final 34
Table 27: Time Between Appointment and Receipt of Incentive Check
Amount of Time Total
(n=100)
1 week or less 10%
2 weeks 21%
3 weeks 14%
4 weeks 16%
5 weeks or more 6%
Don’t know/Refused 33%
Overall Satisfaction
Overall, 96% of customers were satisfied with their experience with the Appliance Recycling Program,
with 86% saying they were “very satisfied”. Only one person said he or she was somewhat dissatisfied
with the program due to how the appliance was removed.
Table 28: Aspects of Appliance Recycling Program Customers Liked
What people liked about the program (Mult. Response) Total
(n=98)
Did not have to remove appliance by myself 63%
$25 incentive payment 31%
Recycling of the appliance/environmental component 18%
Pick-up team did a nice job 15%
Short wait time between sign up and pick up of appliance 6%
Other 5%
Don’t know/Refused 3%
Overall satisfaction with ComEd is also high, with 89% saying they are at least somewhat satisfied. Four
customers said they were dissatisfied with ComEd, and two of those indicated that the reason was because
their bill was too high. Participation in the program made 65% of the participants have a more favorable
view of ComEd.
Only a third of customers said they have actually seen a reduction in their energy bill since their appliance
was removed. Sixteen percent of customers were not sure if they had seen a decrease. These responses
may indicate that people are not that attentive to changes in their electric bill. It is also possible that given
seasonal variations in electric use, customers have a difficult time attributing changes in their bills to their
own behaviors.
Additional Actions
Over half (57%) of the participants surveyed said that based on the participation in the program they have
taken additional actions to save energy at their home. The most common changes that people have made
are installing CFLs, installing energy efficient appliances and installing new energy efficient windows.
Summit Blue Consulting, LLC December 2, 2009 Final 35
Other actions include turning off lights when not using them, reducing the running time of appliances
unplugging appliances when not in use.
Six percent of customers said they have participated in other ComEd energy efficiency programs, namely
the AC program and the energy audit programs. Those customers heard about these additional programs
through bill inserts and the ComEd Energy at Home Newsletter.
3.3 Cost Effectiveness Review
This section addresses the cost effectiveness of the Appliance Recycling program. Cost effectiveness is
assessed through the use of the Total Resource Cost (TRC) test. The TRC test is defined in the Illinois
Power Agency Act SB1592 as follows:
“ ‘Total resource cost test’ or ‘TRC test’ means a standard that is met if, for an investment in
energy efficiency or demand-response measures, the benefit-cost ratio is greater than one. The
benefit-cost ratio is the ratio of the net present value of the total benefits of the program to the net
present value of the total costs as calculated over the lifetime of the measures. A total resource
cost test compares the sum of avoided electric utility costs, representing the benefits that accrue
to the system and the participant in the delivery of those efficiency measures, to the sum of all
incremental costs of end-use measures that are implemented due to the program (including both
utility and participant contributions), plus costs to administer, deliver, and evaluate each
demand-side program, to quantify the net savings obtained by substituting the demand-side
program for supply resources. In calculating avoided costs of power and energy that an electric
utility would otherwise have had to acquire, reasonable estimates shall be included of financial
costs likely to be imposed by future regulations and legislation on emissions of greenhouse
gases.” 6
ComEd uses DSMore™ software for the calculation of the TRC test. 7 The DSMore model accepts
information on program parameters, such as number of participants, gross savings, free ridership and
program costs, and calculates a TRC which fits the requirements of the Illinois legislation.
One important feature of the DSMore model is that it performs a probabilistic estimation of future
avoided energy costs. It looks at the historical relationship between weather, electric use and prices in the
MISO region and forecasts a range of potential future electric energy prices. The range of future prices is
correlated to the range of weather conditions that could occur, and the range of weather is based on
weather patterns seen over the historical record. This method captures the impact on electric prices that
comes from extreme weather conditions. Extreme weather creates extreme peaks which create extreme
prices. These extreme prices generally occur as price spikes and they create a skewed price distribution.
High prices are going to be much higher than the average price while low prices are going to be only
moderately lower than the average. DSMore is able to quantify the weighted benefits of avoiding energy
use across years which have this skewed price distribution.
Table 29 summarizes the unique inputs used in the DSMore model to assess the TRC ratio for the
Appliance Recycling program in PY1. Most of the unique inputs come directly from the evaluation
6
Illinois Power Agency Act SB1592, pages 7-8.
7
Demand Side Management Option Risk Evaluator (DSMore) software is developed by Integral Analytics.
Summit Blue Consulting, LLC December 2, 2009 Final 36
results presented previously in this report. Measure life estimates and program costs come directly from
ComEd. All other inputs to the model, such as avoided costs, come from ComEd and are the same for
this program and all programs in the ComEd portfolio.
Table 29. Inputs to DSMore Model for Appliance Recycling Program
Item Value Used
Measure Life 8 years
Participants 11,979
Annual Gross Energy Savings 15,698 MWh
Gross Coincident Peak Savings 3.35 MW
Net-to-Gross Ratio 73%
Utility Administration Costs $285,166
Utility Implementation Costs $1,301,348
Utility Other Costs $422,302
Utility Incentive Costs $297,475
Participant Contribution to Incremental Measure Costs $0
Based on these inputs, the TRC for this program is 2.58 and the program passes the TRC test.
At this time, additional benefits related to reduction of greenhouse gas emissions have not been quantified
in the calculation of the TRC. These additional benefits would increase the given TRC benefit/cost ratio.
Summit Blue Consulting, LLC December 2, 2009 Final 37
4 CONCLUSIONS AND RECOMMENDATIONS
This section highlights the findings and recommendations from the evaluation of the Appliance Recycling
Program implemented by JACO on behalf of ComEd. The objectives of the evaluation were to: (1)
quantify net energy and peak demand savings impacts from the program during Program Year 1 (PY1);
and (2) to determine key process-related program strengths and weaknesses and provide
recommendations to improve the program.
Below are the key conclusions and recommendations.
4.1 Conclusions
4.1.1 Key Impact Findings
The PY1 net energy savings goal for this program was 8,159 MWh and the program-reported energy
savings was slightly higher than this, 8,528 MWh. The verified energy savings was actually significantly
higher than this – 11,478 MWhs, for an overall realization rate of 135%.
Gross savings per unit are fairly similar for refrigerators. The verified gross savings estimate is somewhat
higher for freezers, which reflects the program’s collection of primarily older units in PY1. Fully 40% of
the freezers picked up by the program are over 30 years old and another 42% are between 20 and 30 years
old. Nearly all (92%) of the freezers collected by the program were manufactured before the 1993
standards change, which resulted in a dramatic improvement in energy use of units made since that time.
Pre-1993 units are generally considered ‘energy hogs’ that use 3 to 4 times the energy of units made since
the standards change.
The primary reason for this higher-than-expected performance was a verified value for the NTG ratio
(including the part-use adjustment) of 0.52 that was considerably higher than the value assumed by the
program for refrigerators of 0.35, which are the largest source of energy savings for the program. The
improvement in this factor alone is the main reason for evaluation-verified net savings that are
significantly higher than those claimed by the program in PY1.
Because of inadequate tracking data and completion of only one survey with a participant that recycled a
room AC unit in addition to a refrigerator or freezer, it was not possible to fully evaluate room AC
savings. Program-claimed savings for room ACs are accepted as verified. This appliance contributes a
very small proportion of the program savings (0.3%).
4.1.2 Key Process Findings
The amount of marketing that was done for this campaign was sufficient given that ComEd reached the
target goal for the number of appliances picked up. The primary marketing took, bill inserts, was also
effective. When asked unprompted where they had heard of the program, nearly three of four participants
(74%) recalled seeing the program mentioned in a bill insert with over two-thirds (69%) saying that was
where they first learned of the program. When prompted, another 11% recalled seeing the program in a
bill insert bringing both unprompted and prompted recall of bill inserts to 85%.
Summit Blue Consulting, LLC December 2, 2009 Final 38
Participants were asked, unprompted, why they chose the ComEd Appliance Recycling Program to
dispose of their appliance instead of some other disposal method. The convenience of the home pick-up
was the main selling point of the program for more participants than any other reason. An additional 13%
said the home pick-up was a secondary reason. The $25 cash incentive was also a factor, but it plays more
of a secondary role with 25% saying it was the main reason and an additional 28% a secondary reason.
Overall, 96% of customers were satisfied with their experience with the Appliance Recycling Program,
with 86% saying they were “very satisfied”.
The program was well-administered. Customers reported a high degree of satisfaction with the sign-up
process and appliances were picked up and payments processed in timely fashion. The implementation
plan stipulates that 90% of customers will not have to wait for more than 14 days to have their appliance
to be picked up. A large majority of participants surveyed (84% of those who could recall) scheduled a
pick-up date within two weeks of when they called. Nearly all participants surveyed (98%) said they were
able to schedule a pick-up date that was convenient for them.
Overall, 92% of participants were satisfied with the collection team who came to pick up the appliance,
with 80% reporting that they were “very satisfied”. Only two participants said they were dissatisfied, with
the reason being that the collection team was not careful removing the appliance.
Three of four participants (74%) said that they were very satisfied with the amount of the incentive
payment. No participants reported being dissatisfied with the size of the payment. Of the participants who
recalled when they received their incentive check, 91% said that they received payment within four weeks
of pickup. Only one customer was dissatisfied with the amount of time it took to receive payment (4
weeks).
4.2 Recommendations
Impact-Related Recommendations
1. We recommend the program tracking data receive periodic data quality reviews for data
quality and completeness. Incomplete data fields need to be populated, particularly those data
fields that are critical to the evaluation, such as appliance brand, model number, age/year
manufactured, size, configuration and location. In addition, we recommend that the rated
amperage data be merged with the historical tracking data, and that this value be collected
going forward. Data exported for the evaluation team should also be checked for anomalies.
Process Related Recommendations
2. Continue using the marketing approach as laid out. Using bill inserts as the main vehicle for
marketing has worked well in the past. By increasing the number of bill inserts used, ComEd
should be able to increase participation further. By increasing the number and maintaining the
consistency of the newspaper advertisements even more potential customers will be exposed
to the program.
3. Continue to reinforce the value of recycling older appliances. Just over one-third (36%) of
participants surveyed said that they would have continued to use the secondary appliance had
it not been for the program. This highlights that there are customers out there that need
convincing that they do not need that second refrigerator or stand alone freezer.
Summit Blue Consulting, LLC December 2, 2009 Final 39
4. Only 1/3 of participants noticed a reduction in their electric bills due to recycling their
appliance. Participants may have a difficulty associating changes in their electric bill to their
own energy saving behaviors. ComEd could help educate their customers on how much
money they could save by taking different energy saving actions, including recycling an old
appliance. When customers do see the savings and are able to attribute them to their
behaviors, they will be more likely to make additional changes in the future.
5. For PY2, ComEd has suggested running a direct mail pilot using a profiling database to target
customers with a similar demographic to those that have already participated in the program.
The appliances recycled in PY1 were much older, on average, than appliances recycled in
states with more mature programs. Not surprisingly, the age of the participants was older as
well. Although it is important to remove these much older appliances from the grid as they
use a much larger amount of energy, ComEd should be aware that these appliances are the
“low hanging fruit” and be careful to not limit their message. ComEd should also make sure
that people purchasing newer appliances, who might turn their old appliance into a spare, are
aware of the program. An expanded retailer partnership could help reach this market.
6. Of the participants surveyed, 6% went on to participate in another ComEd program. It would
be beneficial for ComEd to continue providing information about ComEd’s other residential
programs when the contractors come to pick up the appliance. It can also be something that is
added to the website so that customers who sign up online can find out more information if
they choose.
Summit Blue Consulting, LLC December 2, 2009 Final 40
5 APPENDICES
5.1 Data Collection Instruments
The data collection instruments used in this evaluation consisted of in-depth interview guides for the
ComEd program manager and JACO program management and implementers.
5.1.1 ComEd Residential Appliance Recycling Participant
Survey
QUOTA CHECK:
USE SAMPLE:
- IF REF_NUM>=1 and REFRIGERATOR QUOTA NOT MET OR
- IF FRZ_NUM>=1 and FREEZER QUOTA NOT MET
INTRODUCTION AND SCREENER
Hello, this is [SURVEYOR NAME] from Opinion Dynamics calling on behalf of Commonwealth Edison
company. This is not a sales call. We are contacting customers who had refrigerators, freezers or room air
conditioners removed through an appliance pick-up and recycling program offered by Commonwealth
Edison. May I please speak with [CUSTOMER_NAME]?
Are you the person who was most involved and familiar with the removal? (If not may I please speak
with the person who was most involved with the removal?)
IF NO, NO REFRIGERATOR OR FREEZER PICKED UP: THANK AND TERMINATE
CONTINUE WITH RIGHT PERSON: We are conducting a study to evaluate Commonwealth Edison’s
appliance pick up and recycling program and would like to include your opinions. This is required by the
Illinois Commerce Commission and will be used to verify the effectiveness of the program and to make
improvements.
(IF NEEDED: It will take about 15 minutes.)
This call may be monitored or recorded for quality purposes.
SCREENING QUESTIONS
S0. Is ComEd your electric company or do you receive electricity from someone else?
ComEd
Someone Else [TERMINATE]
(Don’t know)
(Refused)
Summit Blue Consulting, LLC December 2, 2009 Final 41
S1. Our records show that you had [ONE OR MORE REFRIGERATOR if REF_NUM>0, ONE OR
MORE FREEZERS if FRZ_NUM>0, AN AIR CONDITIONER if AC_NUM=1] picked up by ComEd or
its subcontractor JACO. Is this correct?
01 Yes, correct
00 No, it was [RECORD VERBATIM and TERMINATE]
98 (Don’t know) [TERMINATE]
99 (Refused) [TERMINATE]
[Read if REF_NUM>=1 and if REFRIGERATOR_QUOTA not met]
SECTION A: REFRIGERATOR CHARACTERISTICS
S2b Next, I'm going to ask you some specific questions about the refrigerator that was picked up by
ComEd.
A1 Were you using this refrigerator as your main refrigerator, or had it been a secondary or spare? If you
recently bought a new main refrigerator and were just waiting for the old one to be picked up, it should be
classified as “main.” (IF NEEDED: A MAIN REFRIGERATOR IS TYPICALLY IN THE KITCHEN, A
SECONDARY OR SPARE IS USUALLY KEPT SOMEPLACE ELSE AND MIGHT OR MIGHT NOT
BE RUNNING.)
1 Main
2 Secondary or Spare
3 (N/A - Respondent is not primary user of fridge (landlord, etc.)) [TERMINATE]
8 (Don’t know) [TERMINATE]
9 (Refused) [TERMINATE]
QUOTA CHECK … Use responses to 1 for Main quota, 2 for Secondary quota. Once quota met,
T&T
[ASK A2 IF A1=2 ELSE SKIP TO A5]
A2 How long had you been using this refrigerator as a secondary or spare?
[IF NEEDED: If respondent is confused, reinforce that “how long had it been a spare when you decided
to get rid of it.”]
[NUMERIC OPEN END RECORD IN YEARS]
00 (Less than one year)
98 (Don’t know)
99 (Refused)
Summit Blue Consulting, LLC December 2, 2009 Final 42
A3 Thinking just about the past year, was the spare refrigerator plugged in and running …
1 All the time
2 For special occasions only
3 During certain months of the year only, or
4 Was it never plugged in and running
8 (Don't know)
9 (Refused)
[ASK A4 and A4a IF A3=02 OR 03, ELSE A5]
A4 If you add up the total time your spare refrigerator was plugged in and running during the last 12
months that you had it, about how many total months would that be? Your best estimate is okay. (GET
NEAREST MONTH)
[RECORD IN MONTHS]
00 (Less than 1 month)
98 (Don’t know)
99 (Refused)
A4a Was the refrigerator running during the summer or was it mainly running during other times of the
year?
1. Running during the summer
2. Mainly running other times of the year
3. (A mix of both summer and other times of the year)
8. (Don’t know)
9. (Refused)
A5 Where would the refrigerator have been located if it had not been removed by ComEd?
01 (Kitchen)
02 (Garage)
03 (Porch/Patio)
04 (Basement)
00 (Other (SPECIFY:))
98 (Don't know)
99 (Refused)
[SKIP A5B IFA5=1 OR 98 or 99]
A5B Was the space heated or not?
1 Yes
2 No
3 (Heated part of the year)
8 (Don’t know)
9 (Refused)
Summit Blue Consulting, LLC December 2, 2009 Final 43
[SKIP A5C IFA5=98 or 99]
A5C Was the space air-conditioned or not?
1 Yes
2 No
3 (Air conditioned part of the year)
8 (Don’t know)
9 (Refused)
A6 How old was the refrigerator when ComEd removed it?
[NUMERIC OPEN END RECORD IN YEARS]
00 (Less than one year)
98 (Don’t know)
99 (Refused)
A7 Did you replace the refrigerator that ComEd picked up with another one?
1 Yes
2 No
8 (Don’t know)
9 (Refused)
[ASK IF A7=1 else skip to A9]
A8aa. Did you get the replacement refrigerator before or after the old refrigerator was picked up?
1 Before [read in before in A8a]
2 After [read in after in A8a]
3 (Got it the same day) skip to 8b
8 (Don’t know) Skip to A8b
9 (Refused) Skip to A8b
A8a How long <before/after> the old one was picked-up did you get the replacement refrigerator?
01 Within one to two weeks
02 Within one month
03 Within two to three months
04 Within four to six months
05 Within six to twelve months/ one year
06 More than one year later
00 (Other (record verbatim))
98 (Don't know)
99 (Refused)
A8b Was this replacement refrigerator brand new or used?
1. Brand new
2. Used
Summit Blue Consulting, LLC December 2, 2009 Final 44
8. (Don’t know)
9. (Refused)
A8c Does your replacement refrigerator have … (READ)
01 A single door, with a freezer compartment inside
02 Two doors, side by side
03 A Top freezer
04 Or a Bottom freezer?
00 (Other (SPECIFY:___))
98 (Don't know)
99 (Refused)
A8d Is the replacement refrigerator frost free or manual defrost?
01 Frost free
02 Manual defrost
00 (Other (SPECIFY:___))
98 (Don't know)
99 (Refused)
A8e1 Is your replacement refrigerator larger, smaller or the same size as the one it replaced?
1 Larger
2 Smaller
3 Same Size
8 (Don't know)
9 (Refused)
A8f Was getting the replacement a major reason you decided to discard the old one?
1 Yes
2 No
8 (Don’t know)
9 (Refused)
[SKIP IF A8b=1, ELSE TA9]
A8g How old is this replacement refrigerator?
[NUMERIC OPEN END RECORD IN YEARS]
00 (Less than one year)
98 (Don’t know)
99 (Refused)
[ONLY READ TA9 IF A7=1]
TA9. Now let’s get back to your old refrigerator that was removed by ComEd.
Summit Blue Consulting, LLC December 2, 2009 Final 45
A9 When you first heard about ComEd’s Appliance Recycling Program, were you already
considering getting rid of this refrigerator? This could have been by selling it, giving it away,
having someone pick it up, or taking it to the dump or a recycling center.
1 Yes
2 No
8 (Don’t know)
9 (Refused)
A10a. If you had been unable to get rid of your refrigerator through the ComEd appliance recycling
program, would you have still gotten rid of the refrigerator, or would you have kept it?
1 Gotten rid of it
2 Kept it
8 (Don’t know)
9 (Refused)
[ASK IF A10a = 1, ELSE SKIP TO B2]
A10b. If the ComEd program hadn’t been available, would you have gotten rid of the refrigerator within 6
months of when you did, within a year of when you did, or would it have taken longer than a year for you
to get rid of this refrigerator?
1. Within 6 months
2. Within a year
3. Over a year
8. (Don’t know)
9. (Refused)
SECTION B: CONSIDERATION OF ALTERNATIVES SECTION
B1 Now suppose that ComEd appliance recycling program hadn’t been available. I am going to read a
list of alternative ways that you could have disposed of this refrigerator. Please tell me which one you
would have been most likely to use to get rid of this refrigerator. Would you have…
1. Sold it
2. Given it away for free
3. Have it removed by the dealer you got your new or replacement refrigerator from
4. Taken it to a dump or recycling center
5. Hired someone to take it to a dump or recycling center
6. (Keep it)
8. (Don’t know)
9. (Refused)
B2. What was the condition of the refrigerator? Would you say …
1. It worked and was in good physical condition
2. It worked but needed minor repairs like a door seal or handle, or
3. It worked but had some bigger problems
4. (It didn’t work)
8. (Don't know)
Summit Blue Consulting, LLC December 2, 2009 Final 46
9. (Refused)
B3. Thinking about the refrigerator that ComEd picked up, how much money do you think it
would have cost each month to run it if it were running full-time?
1 Nothing
2 $1 to $5
3 $6 to $10
4 $11 to $15
5 $16 to $20
6 More than $20
8 (Don't know)
9 (Refused)
[ASK B4A THRU B4E IF A10a=2. OTHERWISE, SKIP TO B5]
B4A. You mentioned you would have kept this refrigerator if the ComEd appliance recycling program
wasn’t available. If you had kept the refrigerator, would it have been stored unplugged, or used as a
spare?
1 Stored it unplugged
2 Used it as a spare
3 (Both-store it and use it)
4 (Would not have kept it)
8 (Don't know)
9 (Refused)
[ASK IF B4A=2 or 3, Else B5]
B4B For how many years would you have used this refrigerator as a spare? IF NEEDED: Your best
estimate is fine.
[NUMERIC OPEN END]
00 (Less than 1 year)
96 (Until it broke, indefinitely)
98 (Don't know)
99 (Refused)
B4C. Where would this refrigerator have been located if you hadn’t gotten rid of it and had used it as a
spare? IF NEEDED, CLARIFY: What room? IF NEEDED: Your best estimate is fine.
01 (Kitchen)
02 (Garage)
03 (Porch)
04 (Basement)
00 (Other (SPECIFY:___))
98 (Don't know)
99 (Refused)
[SKIP IF B4C=1 OR B4C=98 or 99]
Summit Blue Consulting, LLC December 2, 2009 Final 47
B4D. Would this have been a heated space?
1. Yes
2. No
3. (Part of the year)
8. (Don’t know)
9. (Refused)
[SKIP IF B4C=98 or 99]
B4E Would this have been an air-conditioned space?
1 Yes
2 No
3 (Part of the year)
8. (Don’t know)
9. (Refused)
B5. There may have been a number of reasons why you chose to get rid of the refrigerator that we’ve
been discussing. Using a 0 to 10 scale where 0 is not at all important and 10 is extremely important,
please tell me how important each reason was in your decision to get rid of it?
a. The refrigerator was expensive to run
b. The refrigerator was a spare that I did not use very much [ASK IF A1=2]
[ASK B5C AND B5D IF A7=1 ELSE SKIP TO C1]
c. The refrigerator was old and I wanted something with more modern features
d. I wanted a bigger refrigerator [SKIP IF A8E1 =2,3]
FREEZER SECTION
[READ IF FRZ_NUM>=1]
QUOTA CHECK:
IF REF_NUM=0 THEN COUNT THIS AGAINST FREEZER QUOTA.
IF REF_NUM>=1 THEN DO *NOT* COUNT THIS AGAINST FREEZER QUOTA, WE NEED
AS MANY MULTIPLE APPLIANCE RECYCLERS AS WE CAN GET.
SECTION C: FREEZER CHARACTERISTICS
Next, I'm going to ask you some specific questions about the freezer that was picked up by ComEd.
C1 How long had you been using this freezer?
[IF NEEDED: If respondent is confused, reinforce that “how long had it been used when you decided to
get rid of it.”]
[NUMERIC OPEN END RECORD IN YEARS]
00 (Less than one year)
96 (N/A – Respondent not primary user (landlord, etc.)) [TERMINATE]
98 (Don’t know)
Summit Blue Consulting, LLC December 2, 2009 Final 48
99 (Refused)
C2 Thinking just about the past year, was the freezer plugged in and running …
1 All the time
2 For special occasions only
3 During certain months of the year only, or
4 Was it never plugged in and running
8 (Don't know)
9 (Refused)
[ASK C3 and C4 IF C2=02 OR 03, ELSE C5]
C3 If you add up the total time your freezer was plugged in and running during the last 12 months
that you had it, about how many total months would that be? Your best estimate is okay. (GET
NEAREST MONTH)
[RECORD IN MONTHS]
00 (Less than 1 month)
98 (Don’t know)
99 (Refused)
C4 Was the freezer running during the summer or was it mainly running during other times of the year?
1. Running during the summer
2. Mainly running other times of the year
3. (A mix of both summer and other times of the year)
8. (Don’t know)
9. (Refused)
C5 Where would the freezer have been located if it had not been removed by ComEd?
01 (Kitchen)
02 (Garage)
03 (Porch/Patio)
04 (Basement)
00 (Other (SPECIFY:))
98 (Don't know)
99 (Refused)
[SKIP IF C5=1 OR 98 or 99]
C5B Was the space heated or not?
1 Yes
2 No
3 (Heated part of the year)
8 (Don’t know)
9 (Refused)
Summit Blue Consulting, LLC December 2, 2009 Final 49
[SKIP IF C5=98 or 99]
C5C Was the space air-conditioned or not?
1 Yes
2 No
3 (Air conditioned part of the year)
8 (Don’t know)
9 (Refused)
C6 How old was the freezer when ComEd removed it?
[NUMERIC OPEN END RECORD IN YEARS]
00 (Less than one year)
98 (Don’t know)
99 (Refused)
C7. Did you replace the freezer that ComEd picked up with another one?
1 Yes
2 No
8 (Don’t know)
9 (Refused)
[ASK IF C7=1 else skip to C9]
C8aa. Did you get the replacement freezer before or after the old freezer was picked up?
1 Before [read in before in C8a]
2 After [read in after in C8a]
3 (Got it the same day) Skip to C8b
8 (Don’t know) Skip to C8b
9 (Refused) Skip to C8b
C8a. How long <before/after> the old one was picked-up did you get the replacement
freezer? RECORD TIME INTERVAL
01 Within one to two weeks
02 Within one month
03 Within two to three months
04 Within four to six months
05 Within six to twelve months/ one year
06 More than one year later
00 (Other (record verbatim))
98 (Don't know)
99 (Refused)
Summit Blue Consulting, LLC December 2, 2009 Final 50
C8b. Was this replacement freezer brand new or used?
1. Brand new
2. Used
8. (Don’t know)
9. (Refused)
[SKIP IF C8b=1, ELSE C8c]
C8g How old is this replacement freezer?
[NUMERIC OPEN END RECORD IN YEARS]
00 (Less than one year)
98 (Don’t know)
99 (Refused)
C8c. Is your replacement freezer … (READ)
01 A chest freezer or
02 An upright freezer
00 (Other (SPECIFY:___))
98 (Don't know)
99 (Refused)
C8d. Is the replacement freezer frost free or manual defrost?
01 Frost free
02 Manual defrost
00 (Other (SPECIFY:___))
98 (Don't know)
99 (Refused)
C8e1 Is your replacement freezer larger, smaller or the same size as the one it replaced?
1 Larger
2 Smaller
3 Same Size
8 (Don't know)
9 (Refused)
C8f Was getting the replacement a major reason you decided to discard the old one?
1 Yes
2 No
8 (Don’t know)
9 (Refused)
[ONLY READ TC9 IF C7=1]
TC9. Now let’s get back to your old freezer that was removed by ComEd.
Summit Blue Consulting, LLC December 2, 2009 Final 51
C9 When you first heard about ComEd’s Appliance Recycling Program, were you already
considering getting rid of this freezer? This could have been by selling it, giving it away, having
someone pick it up, or taking it to the dump or a recycling center.
1 Yes
2 No
8 (Don’t know)
9 (Refused)
C10. If you had been unable to get rid of your freezer through the ComEd appliance recycling program,
would you have still gotten rid of the freezer, or would you have kept it?
1 Gotten rid of it
2 Kept it
8 (Don’t know)
9 (Refused)
[ASK IF C10=1 ELSE SKIP TO D2]
C11b. If the ComEd program hadn’t been available, would you have gotten rid of the freezer within 6
months of when you did, within a year of when you did, or would it have taken longer than a year for you
to get rid of this freezer?
1. Within 6 months
2. Within a year
3. Over a year
8. (Don’t know)
9. (Refused)
SECTION D: CONSIDERATION OF ALTERNATIVES SECTION
D1 Now suppose that the ComEd appliance recycling program hadn’t been available. I am going to read
a list of alternative ways that you could have disposed of this freezer. Please tell me which one you
would have been most likely to use to get rid of this freezer. Would you have…
1. Sold it
2. Given it away for free
3. Have it removed by the dealer you got your new or replacement freezer from
4. Taken it to a dump or recycling center, or
5. Hired someone to take it to a dump or recycling center?
6. (Keep it)
8. (Don’t know)
9. (Refused)
D2 What was the condition of the freezer? Would you say …
1 It worked and was in good physical condition
2 It worked but needed minor repairs like a door seal or handle
3 It worked but had some bigger problems
4 (It wasn’t working)
8 (Don't know)
Summit Blue Consulting, LLC December 2, 2009 Final 52
9 (Refused)
D3 Thinking about the freezer that ComEd picked up, how much money do you think it would
have cost each month to run it if it were running full-time?
1 Nothing
2 $1 to $5
3 $6 to $10
4 $11 to $15
5 $16 to $20
6 More than $20
8 (Don't know)
9 (Refused)
[ASK D4A THRU D4E IF C10=2. OTHERWISE, SKIP TO D5]
D4A. You mentioned you would have kept this freezer if the ComEd appliance recycling program wasn’t
available. If you had kept the freezer, would it have been stored unplugged, or would you have continued
using it?
1 Stored it unplugged
2 Continued using it
3 (Both-store it and use it)
4 (Would not have kept it)
8 (Don't know)
9 (Refused)
[ASK IF D4A=2 or 3, ELSE D5]
D4B For how many years would you have used this additional freezer? IF NEEDED: Your best estimate
is fine.
[NUMERIC OPEN END]
00 (Less than 1 year)
96 (Until it broke, indefinitely)
98 (Don't know)
99 (Refused)
D4C. Where would this freezer have been located if you hadn’t gotten rid of it and had continued using
it? IF NEEDED, CLARIFY: What room? IF NEEDED: Your best estimate is fine.
01 (Kitchen)
02 (Garage)
03 (Porch)
04 (Basement)
00 (Other (SPECIFY:___))
98 (Don't know)
99 (Refused)
[SKIP TO D4E IF D4C=1, 98,99]
Summit Blue Consulting, LLC December 2, 2009 Final 53
D4D. Would this have been a heated space?
1. Yes
2. No
3. (Part of the year)
8. (Don’t know)
9. (Refused)
[SKIP to D5 IF D4C=98,99]
D4E Would this have been an air-conditioned space?
1 Yes
2 No
3 (Part of the year)
8. (Don’t know)
9. (Refused)
D5. There may have been a number of reasons why you chose to get rid of the freezer that we’ve been
discussing. Using a 0 to 10 scale where 0 is not at all important and 10 is extremely important, please tell
me how important each reason was in your decision to get rid of it?
a. The freezer was expensive to run
b. I did not use the freezer very much
[ASK D5C AND D5D IF C7=1 ELSE SKIP TO E00]
c. The freezer was old and I wanted something with more modern features
d. I wanted a bigger freezer [SKIP IF C8E1 =2,3]
AC Section
[READ IF AC_NUM=1]
SECTION E: ROOM AIR CONDITIONER CHARACTERISTICS
Next, I'm going to ask you some specific questions about the air conditioner you had picked up.
E00 Was this your own AC or were you discarding someone else’s unit?
01. (My own unit)
02. (Someone else’s unit)
03. (Landlord updating appliances for apartment building)
00. (Something else) (RECORD VERBATIM)
98. (Don’t know)
99. (Refused)
SKIP TO G1 IF E00 IS NOT = TO 1 OR 3
ASK IF E00=1, ELSE E5
Summit Blue Consulting, LLC December 2, 2009 Final 54
E1 At the time the room air conditioner was picked up, was it your only AC, or did you have additional
AC units?
1 Only AC
2 Had additional ACs
8 (Don’t know)
9 (Refused)
E2 Thinking just about the most recent summer that you still had this AC, was it plugged in and running?
1. Yes
2. No
8. (Don’t know)
9. (Refused)
[ASK E3 IF E2=1 ELSE SKIP TO E5]
E3. Still thinking about this last summer that you had the room AC unit, did you run it most days
regardless of the temperature or only on days when the temperature reached a certain level?
1. Most days
2. Only when temperature reached a certain level
3. It was never plugged in and running.
8. (Don’t know)
9. (Refused)
[ASK E3A IF E3=2, ELSE E4]
E3a. How hot did it have to get inside your home or condominium before you ran the room AC unit?
01 Less than 70 degrees
02 70 to 75 degrees
03 76 to 80 degrees
04 81 to 85 degrees
05 Above 85 degrees
00 Other (record verbatim)
98. (Don’t know)
99. (Refused)
E4. When you were cooling your home or condominium, did you tend to run the room AC unit all day
long, or only when you were home or using that room?
1. All the time
2. Only when home/using the room
8. (Don’t know)
9. (Refused)
Summit Blue Consulting, LLC December 2, 2009 Final 55
E5 In what room was the room AC unit located? (IF NEEDED: Please tell me the room where it was
most often located.)
1. (Bedroom)
2. (Living room)
3. (Dining room)
4. (Kitchen)
5. (Hallway)
6. (Other)
8. (Don’t know)
9. (Refused)
E6. At the time of the pick-up, how old was the room air conditioner?
[NUMERIC OPEN END RECORD IN YEARS]
00 (Less than one year)
98 (Don’t know)
99 (Refused)
E7 Did you replace the AC unit ComEd picked up with a different one? [IF NEEDED: This could have
been a different type of AC unit, such as a central AC unit.]
1. Yes
2. No
8. (Don’t know)
9. (Refused)
[ASK IF E7=1 ELSE SKIP TO 10]
E8aa. Did you get the replacement AC before or after the old AC was picked up?
1 Before [read in before in E8]
2 After [read in after in E8]
3 (Got it the same day) Skip to E8a
8 (Don’t know) Skip to E8a
9 (Refused) Skip to E8a
E8 How long <before/after> the old one was picked-up did you get the replacement
AC?
RECORD TIME INTERVAL
01 Within one to two weeks
02 Within one month
03 Within two to three months
04 Within four to six months
05 Within six to twelve months/ one year
06 More than one year later
00 (Other (record verbatim))
98 (Don't know)
Summit Blue Consulting, LLC December 2, 2009 Final 56
99 (Refused)
E8A. Was the replacement another room air conditioner or a central AC system?
1. Room air conditioner
2. Central AC
8. (Don’t know)
9. (Refused)
E8B. Was the replacement AC brand new or used?
1. Brand new
2. Used
8. (Don’t know)
9. (Refused)
[ASK IF E8B=2, ELSE E8D]
E8C. How old is the replacement air conditioner?
[NUMERIC OPEN END RECORD IN YEARS]
00 (Less than one year)
98 (Don’t know)
99 (Refused)
[ASK IF E8A=1, ELSE E8E]
E8D Is your replacement AC larger, smaller or the same size as the one it replaced?
1 Larger
2 Smaller
3 Same Size
8 (Don't know)
9 (Refused)
E8E Is the replacement AC energy-efficient?
1. Yes
2. No
8. (Don’t know)
9. (Refused)
E9 Can you provide me any more information about the replacement AC unit, such as the brand name and
model number, size in tons, or any other characteristics?
[OPEN END: RECORD INFORMATION ON BRAND NAME, MODEL #, ETC.]
96. No
98. (Don’t know)
99. (Refused)
Summit Blue Consulting, LLC December 2, 2009 Final 57
Now let’s get back to the room air conditioner that you had disposed of.
E10. When you first heard that ComEd would pick up an AC along with your other appliance, were
you already considering getting rid of this room air conditioner? This could have been by selling it,
giving it away, having someone pick it up, or taking it to the dump or a recycling center.
1 Yes
2 No
8 (Don’t know)
9 (Refused)
E11AIf you had been unable to get rid of your AC through the ComEd appliance recycling program, would
you have still gotten rid of the AC, or would you have kept it?
1 Gotten rid of it
2 Kept it
8 (Don’t know)
9 (Refused)
[ASK IF E11a=1 ELSE SKIP TO F3a]
E11b. If the ComEd program hadn’t been available, would you have gotten rid of the AC within 6 months
of when you did, within a year of when you did, or would it have taken longer than a year for you to get
rid of this AC?
1. Within 6 months
2. Within a year
3. Over a year
8. (Don’t know)
9. (Refused)
SECTION F: CONSIDERATION OF ALTERNATIVES SECTION
F1 Now suppose that the ComEd appliance recycling program hadn’t been available. I am going to read
a list of alternative ways that you could have disposed of this AC. Please tell me which one you would
have been most likely to use to get rid of this AC. Would you have…
1. Sold it
2. Given it away for free
3. Taken it to a dump or recycling center
4. Hired someone to take it to a dump or recycling center
5. (Keep it)
8. (Don’t know)
9. (Refused)
F2 What was the condition of the AC? Would you say …
1 It worked and was in good physical condition
2 It worked but needed minor repairs
3 It worked but had some bigger problems
4 (It wasn’t working)
Summit Blue Consulting, LLC December 2, 2009 Final 58
8 (Don't know )
9 (Refused)
[ASK F3A THRU F3E IF E11a=2. OTHERWISE, SKIP TO F4]
F3A. You mentioned you would have kept this air conditioner if the ComEd appliance recycling program
weren’t available. If you had kept the AC, would you have used this AC or would you have stored it and
not used it?
1 Used it
2 Stored it and not used it
3 (Both-store it and use it)
4 (Would not have kept it)
8 (Don't know)
9 (Refused)
[ASK IF F3A=1 or 3, ELSE F4]
F3B For how many years would you have used this AC? IF NEEDED: Your best estimate is fine.
[NUMERIC OPEN END]
00 (Less than 1 year)
96 (Until it broke, indefinitely)
98 (Don't know)
99 (Refused)
F4. There may have been a number of reasons why you chose to get rid of the air conditioner that we’ve
been discussing. Using a 0 to 10 scale where 0 is not at all important and 10 is extremely important,
please tell me how important each reason was in your decision to get rid of it?
a. The AC was expensive to run (0 to 10 Scale)
b. I did not use the AC very much(0 to 10 Scale)
c. The AC was old and wasn’t cooling the best anymore(0 to 10 Scale)
(Skip if E8D=2,3) d. I wanted a bigger AC unit or system(0 to 10 Scale)
PROCESS QUESTIONS
Next I have some questions about your experiences with the ComEd Appliance Recycling Program.
G1. How did you first learn about the Appliance Recycling Program?
01. (Retailer)
02. (Internet)
03. (Bill Insert)
04. (ComEd Energy at Home Newsletter)
05. (Friend/relative/neighbor)
06. (ComEd website)
07. (Municipal Website or Municipal newsletter)
08. (Radio)
09. (Newspaper)
00. (Other____)
98. (Don’t know)
Summit Blue Consulting, LLC December 2, 2009 Final 59
99. (Refused)
G2. Since you first learned about the program, have you heard about the program from any other sources?
If yes, where else? (Categories eliminated based on QG1)
01. (Retailer)
02. (Internet)
03. (Bill Insert)
04. (ComEd Energy at Home Newsletter)
05. (Friend/relative/neighbor)
06. (ComEd website)
07. (Municipal Website or Municipal newsletter)
08. (Radio)
09. (Newspaper)
00. (Other____)
96. (No/No other sources)
98. (Don’t know)
99. (Refused)
[SKIP IF G1=03 OR G2=03]
G2a. Have you seen the program mentioned in a ComEd bill insert?
1 Yes
2 No
8 (Don't know)
9 (Refused)
G3. The appliance recycling program includes not only the pick-up service, but also consumer education.
At the time you found out about the pick-up service, did you receive information or learn that older
refrigerators and freezers are less efficient and use more energy than newer ones?
1. Yes, received information
2. No
8. (Don’t know)
9. (Refused)
G3aa. And did you learn that the refrigerator or freezer that is picked up by the program would be
recycled, which means that the coolant in the unit would be safely removed and the materials that the unit
is made of would be reused?
1. Yes, received information
2. No
8. (Don’t know)
9. (Refused)
Summit Blue Consulting, LLC December 2, 2009 Final 60
G3a. There are a number of ways you could have gotten rid of your appliance(s). What is the MAIN
reason you chose the ComEd Appliance Recycling Program instead of some other way?
01. ($25/cash incentive)
02. (The convenience of the home pick-up/Don’t have to take it someplace myself)
03. (Pick up was free)
04. (Appliance was recycled/Was disposed of in a way that was good for environment)
05. (Was recommended by friend/family)
06. (Was recommended by retailer)
07. (Did not know of any other way/No other option)
00. (Other_specify)
98. (Don’t know)
99. (Refused)
G3b. Were there any other reasons? (Categories eliminated based on QG3a)
01. ($25/cash incentive)
02. (The convenience of the home pick-up/Don’t have to take it someplace myself)
03. (Pick up was free)
04. (Appliance was recycled/Was disposed of in a way that was good for environment)
05. (Was recommended by friend/family)
06. (Was recommended by retailer)
07. (Did not know of any other way/No other option)
00. (Other_specify)
96. (No other reason)
98. (Don’t know)
99. (Refused)
G4aa. Once you decided to participate, the first step was signing up for the program. Are you the one that
took care of this, or did someone else in your household sign up?
1. I signed up
2. Someone else signed up
8. (Don’t know)
9. (Refused)
[ASK if G4aa=1, ELSE G8b]
G4b. Did you sign up online, on the phone or in person at Abt (Pronounced: “Apt”) Electronics?
01. Telephone
02. Online
03. In person at Abt Electronics
00. (Other [OPEN END])
98. (Don’t know)
99. (Refused)
[ASK IF G4b=02, ELSE G4f ]
Summit Blue Consulting, LLC December 2, 2009 Final 61
G4c. Was it easy to find the sign up screen on the website?
1. Yes
2. No
8. (Don’t know)
9. (Refused)
G4d. Did the website answer all your questions about the appliance recycling program?
1. Yes
2. No
3. (Not applicable)
8. (Don’t know)
9. (Refused)
G4e. Did you receive confirmation that your sign up had been successful?
1. Yes
2. No
3. (Not applicable)
8. (Don’t know)
9. (Refused)
[ASK IF G4b=1, ELSE G4i]
G4f. Was the representative you spoke to on the telephone polite and courteous?
1. Yes
2. No
3. (Not applicable)
8. (Don’t know)
9. (Refused)
G4g. Did the representative answer all your questions about the program?
1. Yes
2. No
3. (Not applicable)
8. (Don’t know)
9. (Refused)
G4h. Did you have to call more than once?
1. Yes
2. No
3. (Not applicable)
8. (Don’t know)
9. (Refused)
Summit Blue Consulting, LLC December 2, 2009 Final 62
[ASK IF G4b=3, ELSE G5]
G4i. Was the employee you spoke to at Abt (Pronounced: “Apt”) polite and courteous?
1. Yes
2. No
3. (Not applicable)
8. (Don’t know)
9. (Refused)
G4j. Did the employee answer all your questions about the program?
1. Yes
2. No
3. (Not applicable)
8. (Don’t know)
9. (Refused)
G5. Were you able to schedule a pick-up date and time that was convenient for you?
1. Yes
2. No
8. (Don’t know)
9. (Refused)
G4. On a scale of 0 to 10 where 0 is very dissatisfied and 10 is very satisfied, how satisfied are you with
the sign up experience?
0. 0
1. 1
2. 2
3. 3
4. 4
5. 5
6. 6
7. 7
8. 8
9. 9
10. 10
98. (Don’t know)
99. (Refused)
(ASK IF G4=0,1,2,3, ELSE G6)
G4a. Why did you rate it that way?
00 (OPEN END)
98 (Don’t know)
99 (Refused)
Summit Blue Consulting, LLC December 2, 2009 Final 63
G6. How much time passed between when you scheduled the appointment and when your appliance(s)
was/were picked up? (NOTE TO INTERVIEWER: IF RESPONDENT SAYS “ABOUT A WEEK”,
RECORD AS 1 WEEK)
00[ENTER DAYS AND WEEKS]
98. (Don’t know)
99. (Refused)
G7. On a scale of 0 to 10 where 0 is very dissatisfied and 10 is very satisfied, how satisfied are you with
the time it took between when you scheduled the appliance pickup and when it actually got picked up?
0. 0
1. 1
2. 2
3. 3
4. 4
5. 5
6. 6
7. 7
8. 8
9. 9
10. 10
98. (Don’t know)
99. (Refused)
G8b. Just before the pick-up took place, did you receive a call in advance to confirm the appointment or
to let you know the collection team was coming?
1. Yes
2. No
3. (Not applicable)
8. (Don’t know)
9. (Refused)
G8c. Did the collection team arrive on time?
1. Yes
2. No
3. (Not applicable)
8. (Don’t know)
9. (Refused)
Summit Blue Consulting, LLC December 2, 2009 Final 64
G8. On a scale of 0 to 10 where 0 is very dissatisfied and 10 is very satisfied how satisfied were you with
the collection team who picked up your appliance(s)? [REPEAT SCALE IF NECESSARY]
0. 0
1. 1
2. 2
3. 3
4. 4
5. 5
6. 6
7. 7
8. 8
9. 9
10. 10
11. (Wasn’t at home)
98. (Don’t know)
99. (Refused)
(ASK IF G8=0, 1,2,3, ELSE G9)
G8a. Why did you rate it that way?
00 (OPEN END)
98 (Don’t know)
99 (Refused)
G9. On that same scale from 0 to 10, how satisfied are you with the size of the payment you received as a
result of your participation in the ComEd Appliance Recycling Program? [REPEAT SCALE IF
NECESSARY]
0. 0
1. 1
2. 2
3. 3
4. 4
5. 5
6. 6
7. 7
8. 8
9. 9
10. 10
98. (Don’t know)
99. (Refused)
(ASK IF G9=0, 1,2,3, ELSE G10)
Summit Blue Consulting, LLC December 2, 2009 Final 65
G9a. Why did you rate it that way?
(OPEN END)
(Don’t know)
(Refused)
G10b. How long did it take to get the check after your appliance was picked up?
01. 1 week or less
02. 2 weeks
03. 3 weeks
04. 4 weeks
05. 5 weeks
06. 6 weeks
07. 7 weeks
08. 8 weeks or more
00. (Other, specify)
98. (Don’t know)
99. (Refused)
G10. How satisfied are you with the amount of time it took to receive your payment from ComEd, using
the same scale from 0 to 10? [REPEAT SCALE IF NECESSARY]
0. 0
1. 1
2. 2
3. 3
4. 4
5. 5
6. 6
7. 7
8. 8
9. 9
10. 10
98. (Don’t know)
99. (Refused)
(ASK IF G10=0, 1,2,3, ELSE G11)
G10a. Why did you rate it that way?
00 (OPEN END)
98 (Don’t know)
99 (Refused)
Summit Blue Consulting, LLC December 2, 2009 Final 66
G11. Thinking about your entire experience with the ComEd Appliance Recycling Program, overall, how
satisfied are you with the service, using the same scale from 0 to 10?
[REPEAT SCALE IF NECESSARY]
0. 0
1. 1
2. 2
3. 3
4. 4
5. 5
6. 6
7. 7
8. 8
9. 9
10. 10
98. (Don’t know) – Skip to G13
99. (Refused) – Skip to G13
[ASK G11A IF G11 >= 5, ELSE G11B]
G11A. What aspects of the program did you particularly like? [Multiple Response accept 3]
01. (Positive comment about pick-up team)
02. (The service was easy/Didn’t have to dispose of appliance myself)
03. (Short wait between signing up and pick-up)
04. (It was free)
05. (The $25 payment)
06. (Like that appliance was recycled/helps the environment.)
00. (Other-specify)
96. (None of it/Didn't like any of it)
98. (Don't know/Not sure)
99. (Refused)
[ASK G11B IF G11 <= 5, ELSE G13]
G11B. What aspects of the program did you particularly dislike? [Multiple Response accept 3]
01. (Pick up team did not arrive on time)
02. (Other negative comment about pick-up team)
03. (Had to wait a long time to get appointment)
04. (Other negative comment about scheduling appointment)
05. (Someone had to be home for pick-up)
06. (Refund wasn't as much as I was told/false advertising)
07. (Took too long to receive payment; haven’t received payment yet)
00. (Other-specify)
96. (None of it/Was satisfied with all)
98. (Don't know)
99. (Refused)
Summit Blue Consulting, LLC December 2, 2009 Final 67
G13. Overall how satisfied are you with ComEd, using the same scale from 0 to 10? [REPEAT SCALE
IF NECESSARY]
0. 0
1. 1
2. 2
3. 3
4. 4
5. 5
6. 6
7. 7
8. 8
9. 9
10. 10
98. (Don’t know)
99. (Refused)
(ASK IF G13=0,1,2,3, ELSE G14)
G13a. Why did you rate it that way?
00 (OPEN END)
98 (Don’t know)
99 (Refused)
G14. Would you say participating in this program has made you feel more favorable, less favorable, or no
different about ComEd?
More favorable about ComEd
Less favorable about ComEd
No different about ComEd
8. (Don’t know)
9. (Refused)
G15. How long have you been a ComEd customer at any location?
(NUMERIC OPEN END 1-99)
00. Less than one year
98. (Don’t know)
99. (Refused)
G16. Based on your participation in the ComEd Appliance Recycling Program, have you taken any
additional actions to save energy in your home?
Yes
No
8. (Don’t know)
9. (Refused)
Summit Blue Consulting, LLC December 2, 2009 Final 68
[ASK G16a IF G16=1, ELSE G16B]
G16a. What energy saving actions have you taken?
00. (OPEN END)
98. (Don’t know)
99. (Refused)
G16b. Since participating in the program, have you participated in any other ComEd energy efficiency
programs?
1 Yes
2 No
8 (Don't know)
9 (Refused)
[ASK G16c and G16d IF G16b=1, ELSE G17]
G16c. Which other program did you participate in?
00 (OPEN END)
98 (Don't know)
99 (Refused)
G16d. How did you hear about this program?
01. (Retailer)
02. (Internet)
03. (Bill Insert)
04. (ComEd Energy at Home Newsletter)
05. (Friend/relative/neighbor)
06. (ComEd website)
07. (Municipal Website or Municipal newsletter)
08. (Radio)
09. (Newspaper)
00. (Other____)
98. (Don’t know)
99. (Refused)
G17. Have you noticed a reduction in the amount of your electric bill since your appliance(s) [was/were]
removed?
1. Yes
2. No
8. (Don’t know)
9. (Refused)
I have just a few questions left for background purposes only.
Summit Blue Consulting, LLC December 2, 2009 Final 69
H1. Do you own or rent your home?
Own
Rent
8. (Don’t Know)
9. (Refused)
[ASK IF H1 = 2, ELSE H3]
H2. Do you pay your own electric bill or is it included in your rent?
Pay bill
Included in Rent
8. (Don’t Know)
9. (Refused)
H3. How many people live in your household year-round?
[NUMERIC OPEN END]
98. (Don’t Know)
99. (Refused)
H4. What is the age of the Head-of-the Household? (IF THE ROLE IS SHARED, PLEASE ASK THEM
TO PROVIDE AN AVERAGE)
[NUMERIC OPEN END]
98. (Don’t Know)
99. (Refused)
H5. What is the approximate square footage of home that you live in?
[NUMERIC OPEN END]
99998. (Don’t Know)
99999. (Refused)
[ASK H5a IF H5 = DK,ELSE H6]
H5a. Is it…
01. Less than 500 square feet
02. 500 to less than 1000 square feet
03. 1000 to less than 1500 square feet
04. 1500 to less than 2000 square feet
05. 2000 to less than 2500 square feet
06. 2500 to less than 3000 square feet
07. 3000 to less than 4000 square feet
08. 4000 to less than 5000 square feet
09. 5000 square feet or more
98. (Don’t Know)
99. (Refused)
Summit Blue Consulting, LLC December 2, 2009 Final 70
H6. How long have you lived at your current residence?
[RECORD YEARS]
00. Less than 1 year
98. (Don’t Know)
99. (Refused)
H6a. Was your total family income in 2008 before taxes UNDER OR OVER $50,000?
1. Under $50,000
2. Over $50,000
3. (Exactly $50,000)
8. (Don’t know)
9. (Refused)
[ASK IF H6a=1, ELSE H6c]
H6b. Was it under $15,000, between $15,000 and $30,000 or between $30,000 and $50,000?
[INTERVIEWER NOTE: IF EXACTLY $30,000 ENTER AS ‘3. $30,000-$50,000’]
1. Under $15,000
2. $15,000-$30,000
3. $30,000-$50,000
8. (Don’t know)
9. (Refused)
[ASK IF H6a=2, ELSE H7]
H6c. Was it between $50,000 and $75,000 or between $75,000 and $100,000 or was it over $100,000?
[INTERVIEWER NOTE: IF EXACTLY $75,000 ENTER AS ‘2. $75,000-$100,000’. IF EXACTLY
$100,000 ENTER AS ‘3. OVER $100,000’]
1. $50,000-$75,000
2. $75,000-$100,000
3. Over $100,000
8. (Don’t know)
9. (Refused)
H7. What is the highest level of education you have completed?
01. Less than high school
02. High school graduate or equivalent (e.g., GED)
03. Attended some college (includes junior/community college)
04. Bachelors degree
05. Advanced degree
00. (Other, Specify)
98. (Don’t know)
99. (Refused)
Summit Blue Consulting, LLC December 2, 2009 Final 71
5.1.2 ComEd Interview Guide – Sharon Madigan –
03/31/09
1. In your opinion, how effective has the overall Residential Appliance Recycling Program been
thus far?
a. What elements of the program are working best?
b. What elements need improvement?
2. Next I want to learn more about the program implementation approach. We have read JACO’s
program manual but are wondering if the program implementation has changed from what is
described in the manual?
a. (If yes) Why were changes made?
b. Specifically, what changes were made?
c. Have the changes produced favorable results? (describe)
3. Effectiveness of program implementation
a. What challenges have you have experienced to-date with the implementation of the
program, and how have they been addressed?
b. Is the program on track for meeting this year’s goals?
c. How effective has JACO been implementing the program? How responsive has JACO
been at resolving problems? What improvements could be made?
d. What do you think about the size of the incentive provided to participating ComEd
customers? Have incentives been adequate to stimulate participation? What, if any,
feedback have you received from customers regarding the incentive level?
e. Has the participation in this program been greater than or less than expected?. How could
it be improved?
f. It appears that the program tracking data is supposed to be updated quarterly. Can you
confirm that this is the case? Updated more frequently? Less? (Answer: Quarterly)
4. Effectiveness of the collection process
a. How smoothly has the collection process been running thus far? Have there been any
problems with the appliance pick up and recycling process?
b. How much time do you estimate it takes to guide customers through the application
process? Have both the phone and email application processes been working well? Does
the process present any barriers to program participation?
c. What is the length of the appliance pick up wait times? Has it been about what you
expected? Longer? How do customers perceive the pick up wait times? Have you
received any complaints about it?
d. Have you experienced any problems distributing incentives to customers?
Summit Blue Consulting, LLC December 2, 2009 Final 72
(If yes) What problems have you experienced?
Have they been resolved?
e. Do pick up crews leave ComEd packets with customers?
(If Yes) What marketing or informational materials are included in these
packets?
f. Do crew members conduct a brief customer survey during pick ups?
g. What additional customer feedback has the program received to date? How has this been
used to refine the program (if at all)?
5. How is the program currently marketed to customers? (PROBE: Bill inserts, radio ads, posters,
newspaper ads?)
a. Who is leading these efforts? Is it your in-house marketing department?
b. Which of these channels is the most effective (if you know)?
c. Is this approach working well? If not, what aspects could use improvement?
d. What plans are in place for future marketing efforts? Other marketing channels?
Timeline? (PROBE: Cable TV, PR activities, store flyers?)
e. What changes could be made to increase the effectiveness of current marketing efforts?
6. The implementation plan indicates that you have a partnership with Abt Electronics to allow
purchasers of new refrigerators to recycle their old appliance. Is this partnership active currently?
a. (If Yes) Please describe how the partnership works.
b. How effective has the partnership been in increasing program participation?
c. What are the advantages of this retailer partnership approach versus traditional curbside
pick-up?
d. Are there plans to expand the program to include other retailers? Which ones?
Quality Assurance and Quality Control
7. The Program Manual specifically describes two Quality Assurance activities for this program
including recycling center visit and ride-alongs. How many center visits and ride-alongs have
been performed to date? Can you provide a brief description of each activity?
8. Are there any additional quality assurance and control procedures in place?
9. What processes are in place in terms of documenting and reporting these procedures?
Summit Blue Consulting, LLC December 2, 2009 Final 73
5.1.3 JACO Interview Guide – Michael Dunham–
05/06/09
Introduction
My name is ___ and I’m calling from Opinion Dynamics, we are part of the team hired to conduct an
evaluation of ComEd’s Residential Energy Efficiency programs. We’re currently in the process of
conducting interviews with program managers and key staff in order to improve our understanding of
ComEd’s programs. At this time we are interested in asking you some questions about the Residential
Appliance Recycling program.
Roles and Protocols
1. What is your role in the program? What are your main responsibilities? Have these changed over
time? How long have you carried these out?
2. My understanding is that when the program started there was not a facility in the Chicago area
already, is that correct? Can you tell me about the process of set up a recycling center in Chicago?
3. I spoke with Sam Sirkin about the collection process, but it would be great if you could describe
in detail the recycling process starting with the point where the units are picked up and what
happens to them at the time of pick-up versus in the recycling center.
a. How smoothly has the recycling process been running thus far? Have there been any
problems with the appliance pick up and recycling process?
b. What does the collection staff do if they find an appliance that is not in working condition
or does not meet other requirements such as the correct size?
c. In what percentage of the homes would you say that the equipment does not qualify for
the program?
Quality Assurance and Quality Control
4. The Program Manual specifically describes two Quality Assurance activities for this program
including recycling center visit and ride-alongs. Are there any additional quality assurance and
control procedures in place at the recycling center?
5. What processes are in place in terms of documenting and reporting these procedures?
6. Is there anything else relevant to the program or program’s progress that we have not discussed
that we should know about?
Summit Blue Consulting, LLC December 2, 2009 Final 74
5.1.4 JACO Interview Guide – Sam Sirkin – 04/14/09
1. What is your role in the program? What are your main responsibilities? Have these changed over
time? How long have you carried these out?
2. In your opinion, how effective has the overall Residential Appliance Recycling Program been
thus far?
a. What elements of the program are working best?
b. What elements need improvement?
3. Next, I want to learn more about the program implementation approach. We have read the
program manual but are wondering if the program implementation has changed from what is
described in the manual?
a. (If yes) Why were changes made?
b. Specifically, what changes were made?
c. Have the changes produced favorable results? (describe)
4. Effectiveness of program implementation
a. What have been the biggest challenges you have experienced to-date with the
implementation of the program, and how have they been addressed?
b. Is the program on track for meeting this year’s goals?
c. What do you think about the size of the incentive provided to participating ComEd
customers? Have incentives been adequate to stimulate participation? What, if any,
feedback have you received from customers regarding the incentive level?
d. Has the participation in this program been greater than or less than expected?. How could
it be improved?
e. How frequently is the program tracking data updated? Is it correct that you send weekly
updates as well as quarterly KPI updates? What information is provided in the weekly
updates? Could you possibly copy me on the updates? (Answer: Quarterly)
5. Now, I'd like to talk about the collection process. It would be great if you could first describe the
collection process in detail.
a. How smoothly has the collection process been running thus far? Have there been any
problems with the appliance pick up and recycling process?
b. How much time do you estimate it takes to guide customers through the application
process? Have both the phone and email application processes been working well? Does
the process present any barriers to program participation?
Summit Blue Consulting, LLC December 2, 2009 Final 75
c. What is the length of the appliance pick up wait times? Has it been about what you
expected? Longer? How do customers perceive the pick up wait times? Have you
received any complaints about it?
d. What is the process for distributing incentives to customers?
Have you experienced any problems distributing incentives?
(If yes) What problems have you experienced?
Have they been resolved?
e. Do pick up crews leave ComEd packets with customers?
(If Yes) What marketing or informational materials are included in these
packets?
f. Do crew members conduct a brief customer survey during pick ups?
g. What customer feedback has the program received to date? How has this been used to
refine the program (if at all)?
6. How is the program currently marketed to customers? (PROBE: Bill inserts, radio ads, posters,
newspaper ads?)
a. Which of these channels is the most effective (if you know)?
b. Is this approach working well? If not, what aspects could use improvement?
c. What plans are in place for future marketing efforts? Other marketing channels?
Timeline? (PROBE: Cable TV, PR activities, store flyers?)
d. What changes could be made to increase the effectiveness of current marketing efforts?
7. The implementation plan indicates that there was a partnership with Abt Electronics to allow
purchasers of new refrigerators to recycle their old appliance. Is this partnership active currently?
a. (If Yes) Please describe how the partnership works.
b. How effective has the partnership been in increasing program participation?
c. What are the advantages of this retailer partnership approach versus traditional curbside
pick-up?
d. Do you know of plans to expand the program to include other retailers? Which ones?
8. The implementation plan indicates that you will be calculating energy savings for each recycled
appliance. Have these calculations begun?
a. (If Yes) How are the energy savings calculated? (annual kWhs?) Do you have a database
to track the savings? I've seen on the Sharepoint site some mentions of metering. Is that
still a possibility?
b. Have you experienced any problems in calculating energy savings?
(If yes) What problems have you experienced?
Have they been resolved?
Summit Blue Consulting, LLC December 2, 2009 Final 76
Quality Assurance and Quality Control
9. The Program Manual specifically describes two Quality Assurance activities for this program
including recycling center visit and ride-alongs. Do you know how many center visits and ride-
alongs have been performed to date?
10. Do you have any data checks or consistency checks in place??
11. Are there any additional quality assurance and control procedures in place?
12. What processes are in place in terms of documenting and reporting these procedures?
Summit Blue Consulting, LLC December 2, 2009 Final 77
Related docs
