Get documents about "as oil prices and drilling cratered
Shared by: HC120809031524
-
Stats
- views:
- 2
- posted:
- 8/8/2012
- language:
- Unknown
- pages:
- 122
Document Sample
1
AMERICAN STATISTICAL ASSOCIATION
- - - - -
COMMITTEE ON ENERGY STATISTICS
- - - - -
MEETING
- - - - -
FRIDAY, APRIL 26, 1996
The Committee convened in the Clark Room
of the Holiday Inn Capitol, 550 C Street, S.W.,
Washington, D.C., at 9:00 a.m., DR. TIMOTHY D.
MOUNT, Chair, presiding.
PRESENT: TIMOTHY D. MOUNT, Chair
SAMPRIT CHATTERJEE
BRENDA G. COX
JOHN D. GRACE
CALVIN KENT
GRETA M. LJUNG
RICHARD A. LOCKHART
DANIEL A. RELLES
BRADLEY O. SKARPNESS
G. CAMPBELL WATKINS
ALSO PRESENT: RENEE MILLER
YVONNE M. BISHOP
DIANE LIQUE
L.A. PETTIS
JAY HAKES
JOHN WOOD
GORDON M. KAUFMAN
ROY KASS
NEAL R. GROSS
COURT REPORTERS AND TRANSCRIBERS
1323 RHODE ISLAND AVE., N.W.
(202) 234-4433 WASHINGTON, D.C. 20005-3701 (202) 234-4433
2
NANCY LEACH
I-N-D-E-X
Introductory Remarks:
Announcement of Winners of the Contest on
Graphs and Visuals Displays ........................ 3
Restructuring the Oil and Gas Crude
Reserves Program (Agenda Item 5)
Presenter: John Wood, Office of Oil and Gas ........ 8
Discussants: Gordon Kaufman, MIT .................. 19
Brenda Cox, ASA ...................... 29
An Update on Issues Pertaining to the
Restructuring of the Natural Gas Industry
(Agenda Item 6)
Presenter: Roy Kass, Office of Oil and Gas ........ 54
Statistical Issues Pertaining to
Re-engineering at EIA (Agenda Item 7) ............. 77
Measurement Model for Information Management
Process (Agenda Item 7a)
Presenter: Nancy Leach, Energy Markets
and End Use ............................ 77
Is the a Summary Measure of Data Quality
(Agenda Item 7b)
Presenter: Renee Miller, Office of
Statistical Standards ............................. 84
Discussant: Dan Relles, ASA Committee ............. 93
Public Comment ................................... 117
NEAL R. GROSS
COURT REPORTERS AND TRANSCRIBERS
1323 RHODE ISLAND AVE., N.W.
(202) 234-4433 WASHINGTON, D.C. 20005-3701 (202) 234-4433
3
1 P-R-O-C-E-E-D-I-N-G-S
2 MR. MOUNT: I'd like to get started,
3 please. If people would get seated. I'd like to
4 welcome everybody to the meeting of the ASA
5 Committee. A couple of introductory remarks: I'd
6 like to welcome Larry Pettis and Gordon Kaufman from
7 MIT, a former committee member; and I would also
8 like to congratulate Jay Hakes for his TV appearance
9 last night and a promotion of the Web site. I
10 imagine we'll see a great surge of interest as a
11 result of that.
12 We're now turning over to a very
13 important event, and that is to announce the winners
14 of the contest on graphs and visual displays. Jay
15 Hakes.
16 MR. HAKES: Good morning. I don't think
17 we need with this group to discuss the importance of
18 graphic portrayal of information. We live in an age
19 where at least some people need to absorb
20 information very quickly. They don't have time to
21 go over a lot of numbers, and graphics often tell a
22 good story. I think you all have had some good
NEAL R. GROSS
COURT REPORTERS AND TRANSCRIBERS
1323 RHODE ISLAND AVE., N.W.
(202) 234-4433 WASHINGTON, D.C. 20005-3701 (202) 234-4433
4
1 discussions in the Committee here that have helped
2 us see the role of graphs. I would report to you
3 that we've had considerable success over the last
4 year or two getting a number of our graphs into the
5 popular press.
6 USA Today, from time to time, will carry
7 EIA graphs. There have been -- I think there was
8 one a day or two ago in the New York Times. They've
9 had several recently. The Wall Street Journal has
10 had some and the Washington Post. They now do a
11 better job of attributing EIA as the source. So I
12 think the emphasis on graphics has helped educate
13 the public on energy issues and brought attention to
14 the even richer data that is available for people
15 who want to drill down and find it.
16 A couple of years ago, we collectively
17 decided that it might be a good idea to highlight
18 those who did good graphical work and give them some
19 special recognition. One of the roles of a manager
20 is to keep track of different ideas and make sure
21 they're well coordinated, and I failed a little bit
22 in that regard because I offered at that time to
NEAL R. GROSS
COURT REPORTERS AND TRANSCRIBERS
1323 RHODE ISLAND AVE., N.W.
(202) 234-4433 WASHINGTON, D.C. 20005-3701 (202) 234-4433
5
1 personally take out to eat the members who won the
2 graphic contest and neglected to remember that when
3 I, several weeks later, emphasized the importance of
4 teams. So now there are more and more people to
5 take out to dinner every time we have this contest.
6 So there will be less lobster and more pizza.
7 I'd like to thank the nine judges who
8 worked on this. We had three from the Committee:
9 Samprit Chatterjee, John Grace and Bradley
10 Skarpness. We appreciate your work, going over and
11 making these hard decisions; and then within EIA we
12 had Theresa Hellquist, Bob Rutchik, Susan Shaw,
13 Sandra Smith, Alan Swenson and Ann Whitfield.
14 We're about to announce the winners.
15 Winner No. 1 is the team of Mary Carlson and Phil
16 Shambaugh; and accepting for this group is Jim
17 Todaro from the Office of Oil and Gas.
18 Come on up.
19 Mary is a repeat winner, incidentally.
20 I remember she won either last year or the year
21 before.
22 Jim?
NEAL R. GROSS
COURT REPORTERS AND TRANSCRIBERS
1323 RHODE ISLAND AVE., N.W.
(202) 234-4433 WASHINGTON, D.C. 20005-3701 (202) 234-4433
6
1 This is the graph that won this
2 particular award, and I actually had seen this
3 before now, but if you can see how the graphic does
4 give you a better sense of the impacts of
5 temperature.
6 Then our second winner is John Herbert,
7 also of the Office of Oil and Gas; and he developed
8 this graphic on the premium value of gas, reflecting
9 supply uncertainty and weather conditions. That is
10 another one. I know John fairly well. I play
11 tennis with him occasionally, and I suspect that's
12 what he's doing today. So deliver that back to
13 John. We would appreciate it.
14 Then our next award winners -- actually,
15 one of the winners I see is here: Michael Lawrence
16 and Hattie Ramseur from Energy Markets and End Use.
17 This is their graphic. This office has done an
18 excellent job of developing color brochures that
19 summarize a big, large document.
20 Come on up, Mike.
21 These brochures have been widely used,
22 and they've also helped bring press attention to the
NEAL R. GROSS
COURT REPORTERS AND TRANSCRIBERS
1323 RHODE ISLAND AVE., N.W.
(202) 234-4433 WASHINGTON, D.C. 20005-3701 (202) 234-4433
7
1 bigger document. We find a lot of policy makers
2 don't have time to read a big document, but they
3 will read a brochure. In fact, the only problem
4 I've had, I handed them out at some meetings --
5 secretary's senior staff meetings, these brochures
6 -- and the problem was as we moved on to the other
7 agenda items, people kept reading the EIA brochures.
8 So I guess that's a sign you did a good job.
9 Then the final set of winners would be
10 Joelle Davis and Imelda Rivers from Energy Markets
11 and End Use, and accepting on behalf of the team
12 would be Nancy Leach.
13 MS. LEACH: They're also off.
14 MR. HAKES: Yes. Okay, this is another
15 pamphlet. If you haven't seen this pamphlet, I
16 think you would want to get a hold of it. Okay,
17 it's being distributed, and I think you do see it is
18 a nice way. In a sense, this is part of our vision,
19 I think, that in the future we may be having smaller
20 printed publications that may be viewed somewhat
21 like these pamphlets that are then backed up by a
22 rich array of resources electronically. That mix
NEAL R. GROSS
COURT REPORTERS AND TRANSCRIBERS
1323 RHODE ISLAND AVE., N.W.
(202) 234-4433 WASHINGTON, D.C. 20005-3701 (202) 234-4433
8
1 seems to make some sense.
2 I don't know if any of the winners want
3 to thank their mothers or anybody else; but other
4 than that, I will pass it back to Tim.
5 I always hate to miss a presentation by
6 John Wood, but I do have to go give a briefing on
7 energy to some of the staffers in the Senate this
8 morning. We will be discussing EIA's '97 budget
9 with them, so I think I need to be there, John. But
10 I'll talk with you later.
11 MR. WOOD: Good luck.
12 MR. HAKES: I appreciate it. Thank you.
13 MR. MOUNT: So we move on now to the
14 first presentation on Restructuring the Oil and Gas
15 Crude Reserves Program. The presenter is John Wood,
16 Office of Oil and Gas.
17 MR. WOOD: Good morning. Due to budget
18 reductions, our resources in the Reserves Program
19 must be reduced. The annual reserves report is in
20 the EIA Flagship publication. It contains the
21 highest pro-rated data series in the Office of Oil
22 and Gas because of the uniqueness of the data.
NEAL R. GROSS
COURT REPORTERS AND TRANSCRIBERS
1323 RHODE ISLAND AVE., N.W.
(202) 234-4433 WASHINGTON, D.C. 20005-3701 (202) 234-4433
9
1 There's a 50-year history of the annual reserves
2 data. The last 18 years of this data has been
3 published by EIA, and we would like to retain as
4 high a quality annual program as resources permit.
5 And just some of the -- we have a 99.8
6 percent response rate from our respondents. Just
7 there are certain things that we do that we're very
8 proud of.
9 Next slide.
10 But we do have a problem. Due to budget
11 reductions, the resources devoted to the Reserves
12 Program must be reduced. At this time, we do not
13 plan to conduct a full reserve survey. That's a
14 3800 sample out of 23,000 operators for the 1996
15 data year. The 1995 data year's survey is in
16 progress.
17 Next slide.
18 We've asked the ASA Committee to comment
19 on three possible options. Option one: a complete
20 reserve survey every other year, with no reserves
21 report or survey made for intervening years. Two:
22 a complete reserve survey every other year, with
NEAL R. GROSS
COURT REPORTERS AND TRANSCRIBERS
1323 RHODE ISLAND AVE., N.W.
(202) 234-4433 WASHINGTON, D.C. 20005-3701 (202) 234-4433
10
1 published reserve estimates based upon model
2 reserve's behavior made for the intervening years.
3 Three: a complete reserve survey every other year,
4 with published reserve estimates based upon data
5 from a sample of the largest 150 operators. The
6 remaining reserves would be modeled and estimated.
7 We have two questions: (A) In light of
8 the reduced resources, what option would the
9 Committee recommend; and (B) Does the Committee have
10 any other ideas or proposals on how to implement
11 restructuring for the Oil and Gas Crude Reserve
12 Program?
13 Next slide.
14 EIA created the Reserves Program to
15 establish a unified, verifiable, comprehensive and
16 continuing statistical series for crude reserves and
17 crude oil and natural gas. The annual reserve
18 report provides the most accurate yearly estimates
19 of U.S. pre-reserves of crude oil, natural gas and
20 natural gas liquids.
21 These estimates were considered
22 essential in the development, implementation and
NEAL R. GROSS
COURT REPORTERS AND TRANSCRIBERS
1323 RHODE ISLAND AVE., N.W.
(202) 234-4433 WASHINGTON, D.C. 20005-3701 (202) 234-4433
11
1 evaluation of natural energy policy and legislation.
2 It is based upon data filed by operators of oil and
3 gas wells and the operators of natural gas
4 processing plants, customers, our Congress, federal
5 and state agencies, the oil and gas industry, the
6 financial community and the public.
7 The reserves data is used to inform
8 Congress and resource assessment, strategic planning
9 and modeling. In particular, it is the annual field
10 reserve data resulting from the survey -- this is
11 field-level data -- that have allowed EIA and other
12 groups, like United States Geologic Survey, to
13 reassess and enlarge the oil and gas resource-base
14 estimates. That data, then, fields the file -- the
15 oil and gas integrated field file which EIA has
16 built and maintains.
17 A substantial loss of detail in these
18 databases would hamper future resource assessment
19 and modeling efforts. The higher resource
20 assessments that have been made lately have allowed,
21 for example, U.S. production estimates for gas to
22 grow through the year 2015. The latest USGS
NEAL R. GROSS
COURT REPORTERS AND TRANSCRIBERS
1323 RHODE ISLAND AVE., N.W.
(202) 234-4433 WASHINGTON, D.C. 20005-3701 (202) 234-4433
12
1 resource assessment reflects future additions to
2 crude gas reserves from known fields of 322 trillion
3 cubic feet. That's 247 percent higher than the
4 previous estimate. Similarly, for crude oil, an
5 increase of 60 billion barrels, or 184 percent
6 higher as assessed. This is just the growth of
7 reserves in known fields.
8 Next slide.
9 Now what are crude reserves? EIA
10 defines crude reserves as those volumes of oil and
11 gas that geologic and engineering data demonstrate,
12 with reasonable certainty, to be recoverable in
13 future years from known reservoirs under existing
14 economic and operating conditions. Now over time,
15 increasing knowledge of an individual field, changes
16 in price, changes in technology all make a
17 difference in reserve estimates over time; and the
18 net effect has been generally for increases.
19 Next slide, please.
20 This slide shows the types of reserve
21 data collected at the field level: the data on new
22 field discoveries, new reservoirs and old fields,
NEAL R. GROSS
COURT REPORTERS AND TRANSCRIBERS
1323 RHODE ISLAND AVE., N.W.
(202) 234-4433 WASHINGTON, D.C. 20005-3701 (202) 234-4433
13
1 extensions to the reserves resulting from extending
2 the actual known boundaries of a given field, and
3 revisions and adjustments to reserves. The sum is
4 reserve additions.
5 Note that the rather small -- the first
6 positive stack there is the new field discoveries.
7 The volumetric contribution of the new field
8 discoveries is usually quite small in the individual
9 year. The sum of these various components is the
10 reserve additions.
11 It is the reserve additions in old
12 fields, as more of the oil in places recovered and
13 estimated to be recovered, that dominates the oil
14 reserve additions. That's the largest component.
15 Production has been larger than reserves
16 in most years. The red line shows the net reserve
17 change, and it's generally negative.
18 In 1994, there was a 2.2 percent decline
19 in the crude oil reserves, and this is the smallest
20 decline in four years. The current oil reserves are
21 22.5 billion barrels.
22 Next slide, please.
NEAL R. GROSS
COURT REPORTERS AND TRANSCRIBERS
1323 RHODE ISLAND AVE., N.W.
(202) 234-4433 WASHINGTON, D.C. 20005-3701 (202) 234-4433
14
1 Crude oil reserves change annually.
2 It's changes vary by region. You have plotted up
3 here, the blue line, the U.S. total. Texas, the
4 largest oil reserves state, and Wyoming, which is a
5 state with significant reserves plotted in green.
6 In 1986, as oil prices and drilling
7 cratered, all three dropped. But the U.S. decline
8 was 5 percent; Texas, 8 percent; and Wyoming, 11
9 percent.
10 In 1990, Texas was up 2 percent. The
11 U.S. total was down 1 percent, and Wyoming was down
12 4 percent.
13 And the last set of points plotted, in
14 1994, the U.S. total was down 2 percent; Texas was
15 down 5 percent; and Wyoming was down 9 percent.
16 I just put this up to show their
17 significant annual changes, and these changes are
18 not necessarily the same in any individual region.
19 Next slide.
20 The reserves data are presented
21 regionally. The reserves are regionally
22 concentrated, with five areas containing 64 percent
NEAL R. GROSS
COURT REPORTERS AND TRANSCRIBERS
1323 RHODE ISLAND AVE., N.W.
(202) 234-4433 WASHINGTON, D.C. 20005-3701 (202) 234-4433
15
1 of the natural gas reserves. Thirty-three states in
2 the federal off-short have oil and/or gas reserves.
3 In 1994, gas reserves were up for the
4 first time in four years. It was a 1 percent
5 increase and it's a good thing. There is a
6 relationship between crude reserves and production,
7 and you can't have increasing production for a long
8 period of time without stable or increasing crude
9 reserves. The U.S. is counting on increased gas
10 production for the year 2015. In 1994, the gas
11 reserves were 163.8 trillion cubic feet.
12 Go to the next slide, please.
13 EIA's Reserve Program depends
14 fundamentally on survey data gathered on Form EIA
15 23. We receive company confidential reserves data
16 that are provided to us on the survey each year.
17 This is a sample survey of 23,000 operators. This
18 represents operations in 45,000 fields. The larger
19 operators, the 650 or so companies, provide field-
20 level data. The smaller operators provide state-
21 level estimates for their reserves. In 1994, the
22 original sample was 4100 operators.
NEAL R. GROSS
COURT REPORTERS AND TRANSCRIBERS
1323 RHODE ISLAND AVE., N.W.
(202) 234-4433 WASHINGTON, D.C. 20005-3701 (202) 234-4433
16
1 Next slide, please.
2 Our current sampling strategy depends of
3 stratified sampling. There's a certain E-sample
4 stratum of Category One: large operators; Category
5 Two: intermediate operators -- They report at the
6 field level -- and Category Three: small operators
7 above production cutoff were selected with
8 certainty. There's a random-sample stratum where
9 there's an 8 percent sample collected. State by
10 state, there's a process run which minimizes the
11 total number of respondents and still meets our
12 target sampling area for that state.
13 Base-started sampling areas are: 1
14 percent for national estimates; 1 percent for each
15 of the five states having subdivisions: Alaska,
16 California, Louisiana, New Mexico and Texas; 2.5
17 percent for each state having 1 percent or more of
18 estimated U.S. reserves for production; 4 percent
19 for each state having less than 1 percent of
20 estimated U.S. production; and 8 percent for states
21 not published separately.
22 Could I have the next slide?
NEAL R. GROSS
COURT REPORTERS AND TRANSCRIBERS
1323 RHODE ISLAND AVE., N.W.
(202) 234-4433 WASHINGTON, D.C. 20005-3701 (202) 234-4433
17
1 Now the size of operators is very
2 skewed. In the 1994 frame, there were 161 Category
3 One operators, and they were sampled with certainty,
4 and those operators represented 87.5 percent of the
5 oil reserves and 83.6 percent of the crude gas
6 reserves in 1994.
7 The Category Two operators, 482 of them,
8 represented 5.1 percent of the crude oil reserves
9 and 10.5 percent on the gas reserves.
10 In the Category Three operators, again,
11 the frame had 22,211. Of those, we pulled a sample
12 of 3,431, about half and half with certainty and
13 random-sampled operators; and they represented 7.4
14 percent of the crude oil reserves and 5.9 percent of
15 the crude gas reserves.
16 Now it is this distribution that leaves
17 us to believe that a certain new sample of Category
18 One operators would provide a solid set of data
19 every other year that reserve estimates could be
20 based on.
21 Could I have the next slide?
22 Now I'd like to review the options we
NEAL R. GROSS
COURT REPORTERS AND TRANSCRIBERS
1323 RHODE ISLAND AVE., N.W.
(202) 234-4433 WASHINGTON, D.C. 20005-3701 (202) 234-4433
18
1 asked the Committee to consider. Option one is
2 basically no report, no survey, every other year.
3 The pros: there's no modeling required. There's no
4 expenditures for publishing data. The cons:
5 there's no data collected. The data series is
6 interrupted. The field-level data is lost. EIA
7 would be unable to provide annual national or
8 regional reserve information.
9 Could I have the next slide?
10 Option two would be a report of reserves
11 made during non-survey years based on model reserve
12 behavior. The pros: it would provide an estimate
13 of national reserves annually for customers. The
14 cons: there's no data collected. The data series
15 is interrupted. The field-level data would be lost.
16 The accuracy of the national reserves would be
17 questionable, and the regional reserve estimates
18 even more questionable.
19 Okay, the next slide.
20 Option three is a report of reserves
21 based on a smaller survey every other year. The
22 pros: the data series is less than complete, but is
NEAL R. GROSS
COURT REPORTERS AND TRANSCRIBERS
1323 RHODE ISLAND AVE., N.W.
(202) 234-4433 WASHINGTON, D.C. 20005-3701 (202) 234-4433
19
1 not interrupted. The field-level data for data
2 editing and resource studies would basically be
3 available. It would provide estimates of national
4 regional reserves annually that is based on data.
5 It's more reliable than option two in intervening
6 years. The cons: a sample of roughly 150 operators
7 would be surveyed instead of 3800. More estimation
8 would be required to expand the limited data sample,
9 and the accuracy of regional reserves would be
10 diminished.
11 Could I have the next slide?
12 I'd like to review the questions we
13 asked the Committee to consider, and they were: (A)
14 In light of reduced resources, what option would the
15 Committee recommend; and (B) Does the Committee have
16 any other ideas or proposals on how to implement
17 restructuring for the Oil and Gas Crude Reserve
18 Program?
19 Thank you.
20 MR. MOUNT: Thanks a lot.
21 First discussing it is Gordon Kaufman
22 for MIT.
NEAL R. GROSS
COURT REPORTERS AND TRANSCRIBERS
1323 RHODE ISLAND AVE., N.W.
(202) 234-4433 WASHINGTON, D.C. 20005-3701 (202) 234-4433
20
1 MR. KAUFMAN: Thank you. It's a
2 pleasure to be here.
3 MR. MOUNT: Do you want to come up here
4 or do you want to be there?
5 MR. KAUFMAN: I'm happy to sit here. I
6 have a drill sergeant voice. Everybody can here me.
7 I'm delighted to be here and, in
8 particular, to share with you some thoughts on this
9 set of issues. I have followed EIA since its
10 inception, and my heart has always been principally
11 in what John Wood has been talking about. Indeed,
12 EIA 23 is a flagship program, and it's universally
13 recognized to be so.
14 One of the questions that John asked is
15 a technical question conditional upon the budget
16 being recast, and we have to do this. What
17 particular steps can we take to do imputation, or
18 whatever it may be. You've got to cover the holes
19 that would be left by a substantive reduction in EIA
20 23. And as I thought about this, I said, "Well, we
21 could spend a lot of time, instead of fishing, just
22 talking about details and frames and structures and
NEAL R. GROSS
COURT REPORTERS AND TRANSCRIBERS
1323 RHODE ISLAND AVE., N.W.
(202) 234-4433 WASHINGTON, D.C. 20005-3701 (202) 234-4433
21
1 so on." But I would rather not do that at this
2 juncture, because I don't believe that there's a
3 clear-cut technical solution to the degradation of
4 quality of data that would follow a drastic axing of
5 pieces of EIA 23.
6 In particular, it's my personal belief
7 that the loss of field-level data will ultimately be
8 a disaster. And why do I believe that? It's
9 probably not true if you wish to compile, as you are
10 required to do, an historical record of the resource
11 base for purposes of statistical preview by those
12 who like to look at the historical record. But the
13 real power of this exercise comes in its ability to
14 provide data that is useful for detailed policy
15 analysis both within Government and outside of it.
16 John's presentation accents what I would
17 call Morry Adelman's dictum. What's most important
18 to measure are not stocks. They are flows, and
19 flows are changes. When you talk about inferred
20 reserves and proved reserves, if you back away from
21 a traditional view, you're really talking about
22 flows; and the proportional variations in flows --
NEAL R. GROSS
COURT REPORTERS AND TRANSCRIBERS
1323 RHODE ISLAND AVE., N.W.
(202) 234-4433 WASHINGTON, D.C. 20005-3701 (202) 234-4433
22
1 proportional to the mean or medium size of these
2 flows -- tends to be much larger than it is in
3 stocks.
4 The old saying that "the devil in the
5 details is in the details" is present here in this
6 activity, and I'd like to return to that thought
7 after asking some -- well, not so rhetorical
8 questions, although you started answering some of
9 them for us, John. What are the costs of ensuring
10 quality in this particular domain? It is always bad
11 politics to talk about trade-offs, but I'm neither a
12 politician nor a government administrator. But as
13 somebody's who's a concerned citizen, I feel it's
14 reasonable to talk about trade-offs.
15 There are alternatives to allocation of
16 funds, just as there are alternatives to deciding
17 how much "f" work to put into a sampling strata, a
18 sampling design with a fixed budget. And it's hard
19 for me, at least personally, to make a
20 recommendation as to which course of action, among
21 the ones that you have put down, is, in my personal
22 judgment, the best without having some kind of
NEAL R. GROSS
COURT REPORTERS AND TRANSCRIBERS
1323 RHODE ISLAND AVE., N.W.
(202) 234-4433 WASHINGTON, D.C. 20005-3701 (202) 234-4433
23
1 referent.
2 What I mean by that is that you said,
3 "Let's cut EIA 23," and presumably other things are
4 being cut; and I would hope that as you consider the
5 strategy, the kind of EIA corporate strategy, if you
6 like, that you might give some consideration to
7 reallocation. So what are the costs of ensuring
8 quality? I don't know the answer to that.
9 Renee is going to talk to us about
10 measuring data quality, which I think is quite a`
11 propos and relevant to this discussion, and she's
12 going to talk about, as I gather in your nice
13 summary of this, consistency and continuity. Can
14 policy analysis be done with the same degree of
15 rigor in precision as time goes on if the magnitude
16 of the cuts that you propose in EIA 23 take place?
17 Ultimately, if you do not maintain the
18 quality of the field-level data, it's my personal
19 belief that you will substantively damage your
20 ability to do rigorous policy analysis as the
21 current degree of precision the EIA has. It has
22 taken years to build up this particular program and
NEAL R. GROSS
COURT REPORTERS AND TRANSCRIBERS
1323 RHODE ISLAND AVE., N.W.
(202) 234-4433 WASHINGTON, D.C. 20005-3701 (202) 234-4433
24
1 system, and you have an enormous amount of in-house
2 expertise and knowledge ability. It gets back to
3 "the devil is in the details."
4 What's happened to the resource base --
5 the oil and gas resource base? It's self-evident
6 that the size distribution of fields has shifted and
7 that much more of the activity and contribution to
8 the flows here is coming from fields of smaller
9 magnitudes. And you've got to keep track of that if
10 you want precision in this domain.
11 So at this juncture, I will be happy, if
12 pushed, if you force me into the mold of saying,
13 "Well, we have Option One, Option Two and Option
14 Three, Gordon. Which do you choose?" But what I'm
15 arguing for here is a review of this activity to see
16 if there is not a way of absolutely ensuring the
17 quality of the field-level data; and we can move
18 later to talk about the technicalities about how you
19 might do that, but I'd like to just stop there.
20 MR. MOUNT: We have an extra discussant
21 from the Committee, John Grace.
22 MR. GRACE: Unfortunately, I don't have
NEAL R. GROSS
COURT REPORTERS AND TRANSCRIBERS
1323 RHODE ISLAND AVE., N.W.
(202) 234-4433 WASHINGTON, D.C. 20005-3701 (202) 234-4433
25
1 the eloquence or diplomacy of Gordon. I'll just say
2 straight out it's a stupid idea.
3 The database that's collected by EIA in
4 this activity is not only absolutely critical to
5 current policy studies, but its degree of detail and
6 ability as a database to be responsive to what will
7 be changing needs of analysts and policy makers is
8 going to grow dramatically.
9 As John mentioned, the USGS, when they
10 came out with the national assessment last year,
11 which I worked on for the last three or four years,
12 estimated that around 300 trillion cubic feet of
13 natural gas and 60 billion barrels of oil can be
14 expected to come as additions to reserves from
15 discovered fields. That's not new discoveries.
16 Compared to the proved resource base, it's over
17 double the amount of gas and triple the amount of
18 oil.
19 The thing that we need to know in order
20 to make those assessments is the detailed level of
21 data that we find in the OGIFF file, the oil and gas
22 field information file; and without that level of
NEAL R. GROSS
COURT REPORTERS AND TRANSCRIBERS
1323 RHODE ISLAND AVE., N.W.
(202) 234-4433 WASHINGTON, D.C. 20005-3701 (202) 234-4433
26
1 detail, we can't do even the volumetric estimates of
2 how much we can depend on in the future
3 volumetrically. And these quantities are the
4 backbone of the future production of oil and gas
5 over the next 15 or 20 years. They're not the
6 undiscovered resources that, in the survey, we spend
7 so much time and effort thinking about.
8 As John showed, the contribution of
9 undiscovered fields to the annual increase through
10 discoveries is very tiny compared to these, and this
11 is what we learn about in the annual survey done by
12 EIA.
13 Could I see the next slide?
14 Even though the OGIFF file gives us the
15 ability to understand now what the volumetric
16 contribution might be, the great gap that exists in
17 our understanding of the supply function of oil and
18 gas in the United States is now large and is getting
19 larger because we haven't mined the OGIFF file to
20 the degree we need to to understand the relationship
21 between the marginal costs of those inferred
22 resources and the volumes that will be brought to
NEAL R. GROSS
COURT REPORTERS AND TRANSCRIBERS
1323 RHODE ISLAND AVE., N.W.
(202) 234-4433 WASHINGTON, D.C. 20005-3701 (202) 234-4433
27
1 market as prices go up and down.
2 We have an idea of what the economic's
3 proved resources are by the definition of proved
4 resources. Necessarily, they're the resources that
5 are capable of being brought to market under
6 existing price and cost relationships.
7 We have a good idea now, as a result of
8 the 1995 assessment, of what the economic
9 relationships of undiscovered resources are -- that
10 tiny sliver which is added every year. But that big
11 whopping part of the bar graph in John's graph,
12 which is additions through revisions, extensions and
13 adjustments, is coming from that section of the
14 supply function in the middle, about which we now
15 have some estimates of volume, but have really no
16 understanding of the marginal cost structure. And
17 if we lose the resolution by reduction in sampling
18 size or other strategies -- which I understand the
19 budget problem, but then again, that's not my job --
20 if we lose the resolution that we have in that
21 middle segment, we will have lost the ability to
22 analyze and, therefore, understand the nature of
NEAL R. GROSS
COURT REPORTERS AND TRANSCRIBERS
1323 RHODE ISLAND AVE., N.W.
(202) 234-4433 WASHINGTON, D.C. 20005-3701 (202) 234-4433
28
1 supplies that are going to come from the most
2 important segment of the resource base.
3 Could you put the last slide up?
4 The role of revisions is clear. It will
5 undergird supply in the future, and it will
6 undergird the vast majority of supply -- of
7 additions to supply in the future.
8 As Gordon's already mentioned, the
9 nature of depletion in a fixed resource base is the
10 movement away from large fields to smaller fields;
11 and that also requires that not only smaller fields
12 be more carefully sampled, but smaller operators who
13 are the operators who operate smaller fields must be
14 sampled.
15 The changes in price and technology,
16 especially in an industry which is becoming increas-
17 ingly dominated in terms of the number of wells and
18 the number of fields by smaller operators, is some-
19 thing that's going to be much harder to understand
20 in the future just because of the diffusion of
21 technology over a large number of small operators as
22 opposed to calling Exxon and Chevron up on the phone
NEAL R. GROSS
COURT REPORTERS AND TRANSCRIBERS
1323 RHODE ISLAND AVE., N.W.
(202) 234-4433 WASHINGTON, D.C. 20005-3701 (202) 234-4433
29
1 and asking them how they feel about horizontal
2 drilling. I understand that's not quite the role.
3 The effect is the same.
4 Finally, we see a major level of
5 divestiture by the large oil companies and gas
6 companies as they sell off their smaller and
7 marginal assets which are, from their cost
8 structure, smaller and marginal assets, to smaller,
9 more nimble and adept operators who could profitably
10 produce them.
11 So in conclusion, we definitely need the
12 ability to see the fine texture in the resource base
13 in order to understand the most important component
14 of volumes that support future supplies. We need to
15 be able to be responsive to the dynamics of changes
16 in the resource base towards smaller fields and
17 changes in the industrial organization of its
18 production by the increasing role of smaller
19 operators and the ability to see, in this new rather
20 different world, what the role of prices and
21 technology will be.
22 I know that the questions that were
NEAL R. GROSS
COURT REPORTERS AND TRANSCRIBERS
1323 RHODE ISLAND AVE., N.W.
(202) 234-4433 WASHINGTON, D.C. 20005-3701 (202) 234-4433
30
1 posed were technical questions as to how to respond
2 to a bad situation. The message that I'm giving --
3 and I don't need to speak for him, but I believe the
4 message that Gordon's given is that the issue needs
5 to be raised to a different plane and addressed as a
6 matter of policy rather than optimization under a
7 constraint. Let's look at the constraint.
8 MR. MOUNT: Thank you, John.
9 The third discussant is Brenda Cox.
10 MS. COX: Well, in many ways I can
11 repeat what the two speakers said, knowing nothing
12 about oil and gas. But just looking at the
13 statistics coming out of this program, you see a
14 situation in which change can be rapid; and the
15 difference of a year or two could be important.
16 However, I approach this topic as -- I'm
17 a statistician. What's the best thing to do?
18 What's it mean? What's it going to cost? What are
19 the quality aspects? So that's how I looked at
20 this.
21 The first thing I did is look at the
22 basic design of the survey, regardless of how often
NEAL R. GROSS
COURT REPORTERS AND TRANSCRIBERS
1323 RHODE ISLAND AVE., N.W.
(202) 234-4433 WASHINGTON, D.C. 20005-3701 (202) 234-4433
31
1 it's done; and I found -- I think there are some
2 improvements needed in the basic design. In the
3 sampling plan that's being used, I think -- there
4 appears to me that there may be more certainties
5 than actually need to be. In business surveys, you
6 have a very skewed distribution. So a classic
7 business design is to have certainty selections and
8 then differing probabilities of selection based upon
9 size.
10 In this design, you're either certainty
11 or you're 8 percent, and I think there might be
12 something that could be done there that would
13 improve the efficiency.
14 Another thing I notice is that a
15 rotating design is not being used here. It's very
16 common when you're measuring change and when change
17 is highly important over time to use a rotating
18 panel approach in which some people are deliberately
19 kept in the sample from year to year and rotated out
20 so that a certain percentage of your sample is
21 common from one year to the next. And I think that
22 kind of approach should also be considered. So with
NEAL R. GROSS
COURT REPORTERS AND TRANSCRIBERS
1323 RHODE ISLAND AVE., N.W.
(202) 234-4433 WASHINGTON, D.C. 20005-3701 (202) 234-4433
32
1 that respect, I think there are approaches that
2 could be used to improve the basic design.
3 Then I wanted to comment on what's being
4 done in estimation from a statistical sense. This
5 design has a phenomenal response rate; just
6 absolutely phenomenal. Phenomenal is 100 percent
7 for the certainty and 99.something for the non-
8 certainty. That's just incredible. It says
9 something about the value of this data system that
10 people will respond at such a rate.
11 Now presently it doesn't appear to me
12 that any adjustment at all, though, is being done
13 for what non-response actually exists, and I would
14 recommend there are some fairly simple weighting
15 class adjustments that could be made. You don't
16 need to do an imputation the way it was mentioned in
17 the documentation here. But I would suggest a
18 weighting class adjustment there.
19 Then I was a little concerned that
20 there's a lot of imputation occurring. There's a
21 lot of imputation occurring because operators don't
22 have information that they can report to you. Some
NEAL R. GROSS
COURT REPORTERS AND TRANSCRIBERS
1323 RHODE ISLAND AVE., N.W.
(202) 234-4433 WASHINGTON, D.C. 20005-3701 (202) 234-4433
33
1 operators, particularly the very small ones, are
2 reporting production. They don't report reserves at
3 all, so reserves are estimated for them, as well as
4 adjustments, revisions, new discoveries, et cetera.
5 And that's a little scary, in fact, for me.
6 How those imputations, shall we say, are
7 being done is by developing a model for what their
8 reserves would be based upon their production. That
9 ratio, which is used to estimate that, is based upon
10 the people who actually report reserves. Well, the
11 people reporting tend to be the larger operators.
12 So you kind of have -- you're estimating what your
13 smaller operators are doing -- your very small
14 operators are doing -- based upon people who are
15 bigger than they are, and it just makes me feel very
16 uncomfortable. I don't know there's anything that
17 can be done about it, but it's something really that
18 needs to be recognized.
19 In reading the results of this study,
20 you don't see in actual survey estimates that they
21 make enough -- I don't think you make enough point
22 about the fact that, in fact, some of these
NEAL R. GROSS
COURT REPORTERS AND TRANSCRIBERS
1323 RHODE ISLAND AVE., N.W.
(202) 234-4433 WASHINGTON, D.C. 20005-3701 (202) 234-4433
34
1 estimates, even for the survey itself, are not
2 actual estimates. They're approximations or
3 guesses. And that's not to say that, in the
4 aggregate, they're not good. I couldn't respond to
5 that. It's just that a lot is being estimated.
6 Now to get to the basic questions we
7 were asked, I suspect there may be no choice but to
8 do something. A lot of programs are being cut now
9 and, frankly, the cuts in agency budgets don't
10 always flow from what agency has a fat budget,
11 unfortunately. So assuming something has to be
12 done, I think that we should look to the subject of
13 cost versus quality and, in particular, to say that
14 EIA will not produce estimates, approximations,
15 between-year projections if they cannot create
16 something of satisfactory quality.
17 Now EIA has a perfect opportunity to see
18 what it can do. It's got a past data series that
19 you can use to model, to say, "Well, what would have
20 happened if we had modeled for the intervening
21 years." In fact, you can develop an estimation
22 approach, use a certain portion of the data to
NEAL R. GROSS
COURT REPORTERS AND TRANSCRIBERS
1323 RHODE ISLAND AVE., N.W.
(202) 234-4433 WASHINGTON, D.C. 20005-3701 (202) 234-4433
35
1 develop your estimation approach, and then use
2 another portion to test it out and to say, "Okay, if
3 we had used this first ten years here to develop our
4 estimation approach, how well would we have done for
5 the next few years," just to see, you know, the
6 circumstances in which your -- you probably
7 shouldn't call it "estimation," by the way. We
8 probably should call it "projection."
9 Under what circumstances would your
10 projection approach work? What kind of
11 circumstances would lead to large variations in your
12 projections versus reality?
13 So that would be the first thing that I
14 would suggest: that you do a little pen and pencil.
15 I think you've got all the tools available to you
16 now to do that and determine that; and that might
17 answer one question, and that is, how well can you
18 do with a sample of 150? How much does 150 large
19 operators give you in terms of making those
20 projections versus none at all?
21 Now the other question I have to bring
22 up is the subject of your frames. Frames can go
NEAL R. GROSS
COURT REPORTERS AND TRANSCRIBERS
1323 RHODE ISLAND AVE., N.W.
(202) 234-4433 WASHINGTON, D.C. 20005-3701 (202) 234-4433
36
1 only so long without being updated, and I don't know
2 what the ideal update period would be for this
3 frame. I noticed in the documentation that it said
4 that some units were updated every two years. I
5 don't know what the normal range would be; but
6 definitely, if you skip a year, there's going to
7 have to be a full-frame updating before the next
8 survey.
9 In other words, some of the cost will be
10 greater for this every-other-year survey. You'll be
11 doing more updating. The updating may not be as
12 efficient of your frame. You may have to trace more
13 operators to find out what happened to them.
14 There's also a potential for coverage.
15 This survey is doing something that I would do, but
16 you have to recognize that it can be a problem -- or
17 I've done in the past for business surveys, and that
18 is, when you go out with an existing frame and you
19 start looking for people to find out they're there,
20 after you search and search and search, and you
21 can't find them, you conclude that they're out of
22 business and gone. But that may not be the case.
NEAL R. GROSS
COURT REPORTERS AND TRANSCRIBERS
1323 RHODE ISLAND AVE., N.W.
(202) 234-4433 WASHINGTON, D.C. 20005-3701 (202) 234-4433
37
1 It just may be that you can't find them.
2 Now I've made that same assumption
3 myself, but it's a potential source of under-
4 coverage that you have to worry about; and it could
5 lead to a potential continuing deterioration in
6 quality. There really wasn't a discussion of
7 coverage for this frame, so I don't know how well
8 the coverage is of the existing frame.
9 Then we get to the issue of how well can
10 you estimate; and then given, and then next, how
11 much better could you estimate with large units
12 here; and then what would be the cost of all of
13 this, because I think it's a trade-off between all
14 of these. Then, in addition, I'd say, "What would
15 EIA do personally were its own uses?" Forget about
16 users. For its own uses, what would EIA do if they
17 didn't have these estimates in the off-years? How
18 does it affect EIA's own internal programs,
19 procedures, your models, your decision making.
20 Thank you.
21 MR. MOUNT: Thank you, Brenda.
22 So we're open to the Committee,
NEAL R. GROSS
COURT REPORTERS AND TRANSCRIBERS
1323 RHODE ISLAND AVE., N.W.
(202) 234-4433 WASHINGTON, D.C. 20005-3701 (202) 234-4433
38
1 Campbell.
2 MR. WATKINS: Let me make three
3 comments. When I saw this item five on the agenda
4 that John had provided, as a euphemism I was going
5 to say, "Well, it's sad." But I'm actually shocked
6 with the situation in which you're placed.
7 I say that because, if anything, the
8 case can be made -- and I think it can be made
9 strongly -- that the reserve analysis and reserve
10 data, the United States, should be expanded rather
11 than contracted. The reason for that is that there
12 are still gaps, significant gaps, in knowledge about
13 what is happening to the United States' oil and gas
14 reserves.
15 You might argue that a country, for
16 example, like Saudi Arabia, where they really don't
17 know and don't care too much about what their
18 reserves are -- if you were in that position, maybe
19 this doesn't matter that much. But this is a
20 situation at least with respect to crude oil
21 reserves where the reserves are seemingly declining,
22 and it's that much more important that detailed
NEAL R. GROSS
COURT REPORTERS AND TRANSCRIBERS
1323 RHODE ISLAND AVE., N.W.
(202) 234-4433 WASHINGTON, D.C. 20005-3701 (202) 234-4433
39
1 reserve data be available.
2 I think I'm right -- and Gordon, you can
3 comment on this -- that perhaps the best and most
4 detailed data available are reserves for any country
5 is the information available in Canada.
6 MR. KAUFMAN: A western --
7 MR. WATKINS: A western sedimentary
8 base. Hopefully, the United States' information
9 would be moving in the direction of that sort of
10 detail, and the types of areas or the types of
11 information that would be useful to see further
12 extensions of U.S. data would be in the critical
13 areas of distinguishing affirmly between enhance
14 recovery reserve additions; that is, a gauge of
15 proportion of the recovery of the oil in place for a
16 given area -- distinguish between that and, say,
17 what I call aerial extensions in the reservoir by
18 developing drilling -- reserves added in that way --
19 distinguishing more between types of technology, and
20 John has mentioned horizontal drilling, and
21 providing finer detail on reserves by play, and
22 thereby voiding in terms of reserves analysis of
NEAL R. GROSS
COURT REPORTERS AND TRANSCRIBERS
1323 RHODE ISLAND AVE., N.W.
(202) 234-4433 WASHINGTON, D.C. 20005-3701 (202) 234-4433
40
1 what the term or what I can term "aggregation prob-
2 lems."
3 So the fact that you seem to be a place
4 with the situation, where you're going the opposite
5 direction, is, as I said at the start of my column,
6 really quite shocking.
7 The second comment is that the uses of
8 these data seem to be expanding rather than
9 contracting. For example, there are proposals to
10 augment the national accounts to reflect resource
11 depletion.
12 John, that's not on your list of
13 functions here, but I'll add to it.
14 Gordon is aware of the work that Morry
15 Adelman and myself have been doing on valuing in
16 situ reserves and the way that might be utilized to
17 address the problem of adjusting the national
18 accounts if, in fact, that was pursued. Any
19 application of that kind of analysis relies on
20 accurate reserve information and, in particular,
21 accurate information on the changes in the reserves.
22 So that the situation, in terms of the
NEAL R. GROSS
COURT REPORTERS AND TRANSCRIBERS
1323 RHODE ISLAND AVE., N.W.
(202) 234-4433 WASHINGTON, D.C. 20005-3701 (202) 234-4433
41
1 importance or the underlying importance of these
2 data to my mind, is increasing, and it's increasing
3 from desires to more properly accommodate and
4 reflect the question of the valuation of reserves in
5 a context of accounting for depletion or
6 appreciation, as the case may be.
7 My last comment is that everybody thus
8 far seems to have jointly avoided the questions that
9 John -- the options that John has put to the
10 Committee. So that may at least give you my
11 preference, which is clearly for option three. You
12 do not want to avoid publishing the reserves report,
13 make it every other year; and at least if you were
14 to have the sum of 150 as the distribution that
15 clearly shows in your own data, you are going to
16 pick up a lot of information by doing that. So my
17 vote, if you are forcing the situation -- I hope
18 that would not be the case -- would be for option
19 three.
20 MR. MOUNT: Cal?
21 MR. KENT: Can I begin with a very
22 mundane question; that is, how much money are we
NEAL R. GROSS
COURT REPORTERS AND TRANSCRIBERS
1323 RHODE ISLAND AVE., N.W.
(202) 234-4433 WASHINGTON, D.C. 20005-3701 (202) 234-4433
42
1 talking about? What's being spent every year -- I
2 have kind of a vague recollection -- and what are
3 the costs of the three options?
4 MR. WOOD: Maybe a three-year time we've
5 gone from about $1.7 million to $1.5 to $1.3 to what
6 we are targeting under out latest budget scenario is
7 more like $870,000 a year.
8 MR. KENT: That's to do the full --
9 MR. WOOD: Right. And then that is the
10 full cost of running the surveys, maintaining the
11 system, maintaining the frame activities, the
12 quality assurance activities.
13 MR. KENT: And what are the costs of
14 your options, then?
15 MR. WOOD: I'm sorry?
16 MR. KENT: What are the costs? If we
17 went with option one, option two and option three,
18 what would those -- obviously, option one costs us
19 almost nothing.
20 MR. WOOD: They're not, we hope, just
21 tremendously different. If you have a cost of
22 running a survey one year and you decide that you
NEAL R. GROSS
COURT REPORTERS AND TRANSCRIBERS
1323 RHODE ISLAND AVE., N.W.
(202) 234-4433 WASHINGTON, D.C. 20005-3701 (202) 234-4433
43
1 will run it every other year, but in fact try to
2 maintain the same qualities, the same frame quality
3 -- and the frame quality is almost the key to
4 getting a good set of reserve estimates. Over the
5 years, we've investigated lots and lots, more and
6 more sophisticated sampling to lower the sampling
7 rate and keep the same target rate, et cetera. But
8 it always comes back to: the frame has to be good
9 or you're not going to get anywhere.
10 So coming back to the question, you
11 can't simply take the cost of doing the survey every
12 year on a smooth ongoing basis and divide by two.
13 That just isn't the way it happens. We haven't
14 really gotten down to what we would call precise
15 estimates on, you know, dollar-by-dollar kind of
16 savings; but it is in the several hundred thousand
17 dollar range, in kind of marginal costs, to go get
18 the smaller operators.
19 MR. KENT: Well, I think it makes a
20 whale of a lot of difference as to which option one
21 recommends if one knows what the cost savings
22 actually are on these; and I don't know how you
NEAL R. GROSS
COURT REPORTERS AND TRANSCRIBERS
1323 RHODE ISLAND AVE., N.W.
(202) 234-4433 WASHINGTON, D.C. 20005-3701 (202) 234-4433
44
1 could make a decision, quite frankly, unless you've
2 got a good estimate of: if we go with No. 1, our
3 current budget is $800-plus thousand. It will only
4 be $200,000; or if we go with option three, we're
5 only going to save $100,000 if we went with option
6 three. Then for that $100,000, it may be an
7 entirely different conclusion than if you say
8 $500,000 by going with option three. Even making an
9 imprecise mental cost-benefit calculation, you know,
10 I'd need to know what the price tags, or at least
11 the approximate price tags, are here so that one
12 could make the evaluation, at least mentally, how
13 much are we going to lose under each one of these
14 compared to the cost savings that we would
15 experience under each one of these.
16 MR. WOOD: Okay. I would comment on the
17 option two. The first year or two costs may well be
18 higher if we executed it well in just the additional
19 modeling study.
20 MR. KENT: That's not costless.
21 MR. WOOD: What?
22 MR. KENT: That's not costless to do
NEAL R. GROSS
COURT REPORTERS AND TRANSCRIBERS
1323 RHODE ISLAND AVE., N.W.
(202) 234-4433 WASHINGTON, D.C. 20005-3701 (202) 234-4433
45
1 that.
2 MR. WOOD: So in fact, you know, on
3 several grounds option two doesn't look particularly
4 good.
5 On the option one, there are things that
6 we know are somehow qualitative in concept, but we
7 know are costly to avoid that we just haven't tried
8 to make assessments of, and that is that we know we
9 have to do some very serious frame maintenance to
10 make sure that, on an every-other-year cycle, we
11 really approach the same type of quality that we're
12 used to.
13 And two, the operators tend to have to
14 store the data in a slightly different way for us on
15 an operated basis than they do -- have to for their
16 own personal use. So if you skip a year, there's
17 enough turnover in the staffs of the companies that
18 we just anticipate a great deal more problems in
19 executing the survey.
20 So obviously I'm hesitating to try to
21 give a precise quantification of the difference
22 between, for example, option one and option three.
NEAL R. GROSS
COURT REPORTERS AND TRANSCRIBERS
1323 RHODE ISLAND AVE., N.W.
(202) 234-4433 WASHINGTON, D.C. 20005-3701 (202) 234-4433
46
1 It would, you know, probably be in the 10 percent of
2 the budget kind of options.
3 MR. KENT: Okay. And the second comment
4 that I would want to make, although I do think you
5 should generate more precise cost estimates of these
6 three options, would be to call attention to what
7 Brenda had previously said; and that is, with 50
8 years of good data, 18 of them are your own, if
9 you're going to use some sort of a modeling
10 technique, you've got enough data to be able to
11 back-cast or something like that and come up with --
12 to see really what you are going to lose of this
13 fine texture by just going back and saying, "If we
14 had done this."
15 Now, again, in the first year, that's
16 not going to be an inexpensive process. But you
17 could very easily establish whether or not this
18 approach for the intervening years is going to lead
19 to a significant diminution in quality just of your
20 data just by going back and testing this option in
21 the way that Brenda has suggested.
22 MR. MOUNT: Richard?
NEAL R. GROSS
COURT REPORTERS AND TRANSCRIBERS
1323 RHODE ISLAND AVE., N.W.
(202) 234-4433 WASHINGTON, D.C. 20005-3701 (202) 234-4433
47
1 MR. LOCKHART: I have a comment and a
2 question, I guess. The comment is to probably one
3 you've already thought about, but when you come to
4 1998, and it's the second time you haven't done a
5 survey or a full survey, or you're looking only at a
6 fragmentary survey, you won't have '96 data. So
7 when you go back and when you're developing modeling
8 methods for filling in the intervening years, you'll
9 be able to look at modeling methods which utilized
10 data from two years ago, as well as simple one-step-
11 ahead-type forecasts. And those modeling methods
12 will not be nearly as useful when the two-data --
13 two-step-behind data is missing. And so you'll want
14 to think about how well you'll be able to modify
15 your models when you don't have this two-year-old
16 data come, say, 2,000 or 1998.
17 The second -- that's just a remark.
18 The question I wanted to raise was that
19 you showed us two graphs: one which showed that
20 large operators were roughly responsible for 85
21 percent of reserves; and then you focused on a graph
22 which talked about things in reserve, and Gordon
NEAL R. GROSS
COURT REPORTERS AND TRANSCRIBERS
1323 RHODE ISLAND AVE., N.W.
(202) 234-4433 WASHINGTON, D.C. 20005-3701 (202) 234-4433
48
1 talked about the importance of flows. Large
2 operators are responsible for what fraction of
3 change of reserves? In other words, this certainty
4 sample, 150, you have 85 percent total reserves, but
5 we're really trying to measure changes. So how
6 important are they for that?
7 MR. WOOD: Well, actually I have an
8 overhead that would probably make that reasonably
9 clear. This is for the changes in percent of the
10 annual crude oil year by year, so we have 18 surveys
11 in here or 17 changes. The percent change is the
12 solid line for the total U.S. crude oil reserves,
13 and the dotted line is a category one reported crude
14 oil reserves.
15 So the flippant answer is that, you
16 know, they represent 85 percent of the change
17 because they represent 85 percent of the reserves,
18 and the changes tend to match.
19 It is kind of interesting, the biggest
20 mismatch -- for example, one of them is in 1986,
21 when the price of crude oil went from $25 a barrel
22 in December of 1985 to $9 a barrel in July of 1996,
NEAL R. GROSS
COURT REPORTERS AND TRANSCRIBERS
1323 RHODE ISLAND AVE., N.W.
(202) 234-4433 WASHINGTON, D.C. 20005-3701 (202) 234-4433
49
1 and the drilling dropped several thousand rigs in a
2 six- or seven-month period and there's tremendous
3 turmoil. So the idea of -- you know, there are
4 things that happen which may explain when there are
5 bigger changes.
6 Now we actually, in fact, looked at how
7 you might do simple projections of this at the
8 national level.
9 Bob, would you put that next overhead
10 on?
11 This is about the simplest thing you
12 could do for the annual national-level crude oil
13 reserves. What we did there was use the first 12
14 changes to calibrate that model and then projected
15 the next file; and the average absolute error in
16 those five years is about .4 percent, and the
17 largest one was about .6 percent.
18 Now obviously bigger changes than that
19 do happen, as it happened in 1986; but then again
20 there are other things that might help explain the
21 changes in certain years. Again, you would expect
22 to be a little better in oil because the percentage
NEAL R. GROSS
COURT REPORTERS AND TRANSCRIBERS
1323 RHODE ISLAND AVE., N.W.
(202) 234-4433 WASHINGTON, D.C. 20005-3701 (202) 234-4433
50
1 represented by the large operators is a little
2 higher. As you go to the regional level, the larger
3 states would probably not be too much different than
4 this. As you move into the smaller states, you
5 would expect certainty because often the larger
6 operators don't represent as big a piece of it.
7 I might comment on two or three threes
8 things as kind of a ready answer. We do have some
9 imputation processes designed that account for
10 response rate in the size and the expected behavior
11 of the very small operator. In the last two years,
12 we simply haven't implemented it because, you know,
13 there was no need to do it for the small certainty
14 operators, and there's a very small problem for the
15 category three.
16 Some of the imputation processes for the
17 small operator reserves, they are significantly
18 different than for the large operators. On a
19 national level, the reserve-to-production ratio
20 might be 9 or 10 to 1. In New Mexico for gas, it's
21 like 15 to 1. For the smaller operators, it's more
22 like 6 to 1. So that sample of more comparably
NEAL R. GROSS
COURT REPORTERS AND TRANSCRIBERS
1323 RHODE ISLAND AVE., N.W.
(202) 234-4433 WASHINGTON, D.C. 20005-3701 (202) 234-4433
51
1 sized operators who make their estimates in kind of
2 similar ways, you know, there's a lot of difference
3 in the way the smaller operators are actually
4 estimating compared to the reported data for the
5 larger ones.
6 MR. MOUNT: Why don't you leave that up
7 there, because I think it will stimulate some more
8 discussion.
9 John?
10 MR. GRACE: I would just suggest, though
11 that graph itself does provide some confidence in
12 the national-level aggregate, the greatest concern
13 that I have is not in the ability to peg the
14 national-level aggregates or even state-level. It's
15 in the field data. And it's by the analysis of the
16 field data that we're going to get to an
17 understanding of how much of these volumes of
18 adjustments, revisions and extensions are going to
19 be contributed to supply; and that ultimately is
20 what we care about.
21 I mean, even more to Morry Adelman's
22 point of the flows are what matter, changes in the
NEAL R. GROSS
COURT REPORTERS AND TRANSCRIBERS
1323 RHODE ISLAND AVE., N.W.
(202) 234-4433 WASHINGTON, D.C. 20005-3701 (202) 234-4433
52
1 stock can be conceived as flows, but really what we
2 all care about is the flows that come out of the
3 ground not moving categories between reserves. And
4 it's the flows that come out of the ground that are
5 affected by the structure of marginal cost, and our
6 ability to understand marginal cost is dependent on
7 that fine texture that's available in the data that
8 never shows up in the book.
9 MR. MOUNT: Does anybody else from the
10 Committee want to make comments?
11 MS. COX: Just the emphasis on field-
12 level data. Field-level data isn't being obtained
13 for the small operators; is that correct?
14 MR. WOOD: That's correct. Probably 5
15 percent of the data is at the state level, and that
16 data is very carefully matched painstakingly by Bill
17 Monroe, Office of Oil and Gas, back to the fields
18 where we appear to not have the coverage; but we do,
19 from supplementary data sources, get to dimension.
20 The reserve data, then, is distributed mostly over
21 the small fields -- very small percent back over
22 those. So it's accounted for in the system. There
NEAL R. GROSS
COURT REPORTERS AND TRANSCRIBERS
1323 RHODE ISLAND AVE., N.W.
(202) 234-4433 WASHINGTON, D.C. 20005-3701 (202) 234-4433
53
1 is always at the margin, there's data you don't know
2 at the field level.
3 MR. MOUNT: Is there anybody from the
4 public who would like to make comments?
5 I have a few. Basically, I would like
6 to endorse comments that have already been made:
7 that I think that the importance of the costs and
8 the trade-offs that Cal talked about for the
9 different options is clearly important; the value of
10 a rolling sample that Brenda talked about for
11 understanding a very dynamic situation, and the
12 ability to do the sort of modeling that you have
13 just described with data that have already been
14 gathered -- that a number of Committee members have
15 mentioned. But I want to add one more thing, and
16 that is that these data really do matter, and the
17 cost of acquiring them is really relatively small
18 compared to the cost of many other components of
19 government.
20 I think it would be very unfortunate if
21 we got into the situation where we're looking at one
22 data set versus another data set within EIA. I
NEAL R. GROSS
COURT REPORTERS AND TRANSCRIBERS
1323 RHODE ISLAND AVE., N.W.
(202) 234-4433 WASHINGTON, D.C. 20005-3701 (202) 234-4433
54
1 think the important thing is to recognize that in a
2 situation where things are changing, and here the
3 ability to extract oil has improved dramatically
4 over the last few years, sort of understanding these
5 things better is absolutely essential really for the
6 benefit of the country.
7 This is a highly strategic commodity,
8 and the same thing can be said for the issues that
9 we discussed yesterday about the changing structure
10 of the electric utility industry. This is a very
11 dynamic situation and that we cannot understand it
12 without having good data. The role of EIA is
13 absolutely essential in this sort of endeavor.
14 So if you want to have anymore remarks
15 -- have you covered all of your responses?
16 MR. WOOD: Well, other than that we'll
17 certainly look into the panel approaches and, then
18 again, would also allow for a certain amount of
19 certainty sampling in various panels and potentially
20 the sampling according to size. But I think the
21 driver of what we have arrived at over the years as
22 the sampling procedures is, you know, we have kept
NEAL R. GROSS
COURT REPORTERS AND TRANSCRIBERS
1323 RHODE ISLAND AVE., N.W.
(202) 234-4433 WASHINGTON, D.C. 20005-3701 (202) 234-4433
55
1 refining the sample size for a given target.
2 Two, there is such a turnover in the
3 companies -- you know, where a 15 percent change in
4 the frame or more is not uncommon -- that, you
5 know, we spend a lot of time keeping track of that;
6 and two, any given company can acquire a very large
7 proportion of the reserve base, I mean, half percent
8 and things, even though they're not a particularly
9 large company, through acquisition or something else
10 and the company owner of it.
11 There's been a tremendous movement of
12 the fields operatorship amongst the companies during
13 the last seven years. In fact, we've documented and
14 tracked that. So there's been at least a 50 percent
15 turnover inside the companies on their field
16 operatorship. What that always makes me worry about
17 is that if they'll be such potential for erratic
18 change in a given year, that if you missed any of
19 the sample size and if you don't sample, you know,
20 the larger groups, basically we'll sample. And I
21 thank you for your comment.
22 MR. MOUNT: So it looks as though we're
NEAL R. GROSS
COURT REPORTERS AND TRANSCRIBERS
1323 RHODE ISLAND AVE., N.W.
(202) 234-4433 WASHINGTON, D.C. 20005-3701 (202) 234-4433
56
1 a little bit ahead of schedule, which is very good.
2 I propose that we take a break now and meet back in
3 15 minutes, because our last discussant, Dan Relles,
4 wants to rush off at the end here, and that will
5 give him a better chance of getting to his next
6 meeting.
7 So I thank all of the contributors this
8 morning. I think this has been a very important
9 discussion.
10 (Whereupon, a short recess was taken.)
11 MR. MOUNT: So the first presentation,
12 we're revisiting a topic that the Committee has been
13 concerned about for a number of past sessions. This
14 is an update on issues pertaining to the
15 restructuring of the natural gas industry. Roy
16 Kass, Office of Oil and Gas.
17 MR. KASS: Hi. Good morning. I'm Roy
18 Kass, from Reserves in Natural Gas Division of the
19 Office of Oil and Gas; and I'm going to talk about a
20 sub-part of what has been distributed to you in the
21 past, and I understand came along in a most recent
22 package of material: the results of an EIA Process
NEAL R. GROSS
COURT REPORTERS AND TRANSCRIBERS
1323 RHODE ISLAND AVE., N.W.
(202) 234-4433 WASHINGTON, D.C. 20005-3701 (202) 234-4433
57
1 Improvement Team that was looking at gaps in our
2 data. Specifically, they focused on gaps in the gas
3 data. We've got a lot of them.
4 The two that are most important are,
5 first, there were missing information that would
6 enable separate identification of what it costs to
7 transport gas. There is no way that any reasonable
8 analyst can parse apart the stuff in our data series
9 and get an estimate that they would be confident in;
10 that this is a transportation cost specifically for
11 a given leg or to get gas from one place to another.
12 The second problem that they focused on
13 was coverage in the measurement of prices. This has
14 to do with the changes in the industry that have
15 happened over the past 10 or 15 years. I remember
16 coming and speaking to the ASA Committee two years
17 ago when we had noticed the problem.
18 The problem really comes from the fact
19 that all of our surveys go to companies that
20 physically move the gas. Our monthly survey and our
21 annual survey go to those companies, and we asked
22 them essentially two questions.
NEAL R. GROSS
COURT REPORTERS AND TRANSCRIBERS
1323 RHODE ISLAND AVE., N.W.
(202) 234-4433 WASHINGTON, D.C. 20005-3701 (202) 234-4433
58
1 The first question is: How much gas do
2 you sell to customers that you deliver gas to, and
3 what did you get for it? What's your revenue
4 stream?
5 The second question is: How much gas do
6 you -- as we pointedly put it -- transport for the
7 account of others?
8 In the second case, our respondents
9 obviously have absolutely no idea what the cost was.
10 All they can tell us is what they generated as the
11 revenues for their transportation function. Because
12 of that, we do what we call "shade in" the revenue
13 thing. We don't ask anything about that. Keep that
14 thought in the back of your mind. It's going to
15 come up later.
16 As the industry has changed over the
17 past several years, more and more gas has been
18 transferred to what we call transportation gas.
19 Customers, especially large customers, can cut deals
20 from all sorts of sources to buy gas for less cost
21 than if they went to their friendly local
22 distribution company. As a result, we are getting,
NEAL R. GROSS
COURT REPORTERS AND TRANSCRIBERS
1323 RHODE ISLAND AVE., N.W.
(202) 234-4433 WASHINGTON, D.C. 20005-3701 (202) 234-4433
59
1 as I said, less and less of the gas.
2 We know that. We try to be up-front
3 about it. We publish a representation of how much
4 gas, by state, is covered in our price series. But
5 people tend to not look at footnotes. They tend to
6 interpret table titles the way they want, and a lot
7 of folks say that the number we're publishing -- for
8 instance, for industrial gas -- is the cost of gas
9 for industrial customers in a state.
10 This shows the representation of the
11 past several years. The earlier data series is
12 industrial. We've been monitoring industrial
13 transportation versus sales for longer than we've
14 been monitoring commercial. And you can see that
15 the industrial is in pretty poor shape.
16 Nationally, in 1994, we caught about 25
17 percent of the gas; '95 trended down even more; and
18 we figure our preliminary estimates are something
19 like 20 to 22 percent of the gas nationally. There
20 are some states where we capture 100 percent; there
21 were some states we capture substantially less than
22 5 percent. If anybody is working for information in
NEAL R. GROSS
COURT REPORTERS AND TRANSCRIBERS
1323 RHODE ISLAND AVE., N.W.
(202) 234-4433 WASHINGTON, D.C. 20005-3701 (202) 234-4433
60
1 the state of New Mexico, for instance, and wants to
2 do price, beware. It is glitchy as all get out;
3 okay? And that's because we're capturing
4 approximately 3 percent of New Mexico gas. I got a
5 call on that a couple of weeks ago.
6 Well, in terms of the coverage issue and
7 the misinterpretation issue, there was an article in
8 NG magazine in the last session that mentioned me
9 and it came to my attention, and I'll quote from it:
10
11 "It seems that the U.S. Government has no idea what
12 industriales really pay for natural gas.
13 The prices that me and my group
14 painstakingly release every month in
15 EIA's natural gas monthly publication
16 under the rubric average price for
17 natural gas sold to industrial consumers
18 by state are nothing of the kind, and
19 the Government readily admits it. The
20 question is, what do we do about it."
21 What I'm doing here today is really a progress
22 report. It's a heads-up on what we are doing about
NEAL R. GROSS
COURT REPORTERS AND TRANSCRIBERS
1323 RHODE ISLAND AVE., N.W.
(202) 234-4433 WASHINGTON, D.C. 20005-3701 (202) 234-4433
61
1 it, what we intend to do about it, and it's
2 something to keep in mind. I have some concerns
3 that I'll voice at the end about this.
4 Okay. So as I said, we monitor natural
5 gas delivered to end-users. By doing the
6 measurement on the molecules, we're missing more and
7 more of the gas. We have a basic approach that we
8 came up with in terms of how could we fill in this
9 gap, and this really has two-sided parts -- two
10 parts.
11 One is: go to the people who pay the
12 bills. They know what the bills are. The other is:
13 go to the people who send the bills. They know
14 what the bills are. On the one hand, we've got the
15 consumers. There are, in the industrial sector,
16 about 20,000 transportation consumers. That's off-
17 system sales. Their average consumption a year is
18 something on the order of 300 million cubic feet of
19 gas. That's a lot of gas. They are big users,
20 comparing them to the 185,000 on-system users that
21 have an average of about 11 million a year, okay?
22 So we're talking a big difference.
NEAL R. GROSS
COURT REPORTERS AND TRANSCRIBERS
1323 RHODE ISLAND AVE., N.W.
(202) 234-4433 WASHINGTON, D.C. 20005-3701 (202) 234-4433
62
1 That proportional difference is
2 approximately the same as commercial sector as well.
3 We're talking really big commercial users that cut
4 their own deals; relatively small commercial users
5 who don't.
6 Well, if we've got a frame of 20,000 and
7 we wanted estimates by state, we're talking about a
8 daunting sampling effort. If all things were equal,
9 I would opt for doing something other than that on a
10 monthly survey.
11 There is the other than that. Those are
12 the guys who pay the bills. Go to the guys who send
13 the bills; okay? There is a marketing industry that
14 has developed and consolidated over the last few
15 years. About three years ago, what it took to be a
16 marketer was a business card and a phone. But each
17 of them have bought each other out. It's gotten
18 bigger and bigger. They have gotten bigger an
19 bigger, rather; and now we have a pretty stable
20 group of marketers.
21 There's an organization up in Bethesda
22 called Ben Schlesinger and Associates that each year
NEAL R. GROSS
COURT REPORTERS AND TRANSCRIBERS
1323 RHODE ISLAND AVE., N.W.
(202) 234-4433 WASHINGTON, D.C. 20005-3701 (202) 234-4433
63
1 puts together a directory. It is the only source
2 that we have located that represents itself as a
3 comprehensive listing of marketers. It has
4 approximately 450 entries. The top 65 to 70 of them
5 account for about 90 percent of the gas. So that
6 represents, you know, a target of opportunity that
7 we're trying to take advantage of.
8 We're going to go to the folks who send
9 the bills, the marketers. We're going to enumerate
10 the top 65, a cut-off sample; all right?
11 Among the advantages of going to the
12 marketers are that they represent the set of actors
13 who sell the gas. They could potentially capture
14 sales to commercial and residential consumers. The
15 residential off-system sale component is small now,
16 but there have recently been a lot of changes in
17 state regulation and state rules; and looming on the
18 horizon over the next five years is an increase.
19 There are a lot of experiments going on in several
20 states. There are a lot of marketing affiliates set
21 up by OBC's to cap that market -- and, as I said,
22 it's a small target.
NEAL R. GROSS
COURT REPORTERS AND TRANSCRIBERS
1323 RHODE ISLAND AVE., N.W.
(202) 234-4433 WASHINGTON, D.C. 20005-3701 (202) 234-4433
64
1 Disadvantages are that they're not used
2 to EIA. They are not used to regulation. They are
3 not used to filling in our silly forms. And we
4 frankly do not know how, you know, the impact of
5 that is going to play out.
6 Conversely, EIA is not used to them. We
7 really don't know what they have in their data
8 system; okay? We really don't know, on a day-to-day
9 basis, what they need to do their jobs. One thing I
10 learned when I came into the statistics business is
11 that if you ask people something, if you ask
12 companies something, that something better be in
13 their routine business records, or you are not going
14 to get anything worthwhile.
15 So the proposed solution is: go to the
16 folks who send the bills. We have a draft survey
17 form.
18 Could I have No. 3? Oh, let me have No.
19 4.
20 This is really drafty. It's last week's
21 version and it's changing day to day. It's the
22 operational part.
NEAL R. GROSS
COURT REPORTERS AND TRANSCRIBERS
1323 RHODE ISLAND AVE., N.W.
(202) 234-4433 WASHINGTON, D.C. 20005-3701 (202) 234-4433
65
1 What we're going to ask them, by state,
2 is how much the gas they deliver, f.o.b. burner tip,
3 and how much gas they deliver -- they sell, rather,
4 f.o.b. burner tip, and how much gas they sell f.o.b.
5 citygate; and for the citygate price, that just gets
6 it into the local distribution system. We're not
7 going to ask them this thing called "distribution
8 charges," because that's what the local distribution
9 system charges to get it from the pipeline to the
10 customer's company.
11 There are anecdotes that say that in
12 some cases that last ten miles has a confiscatory
13 rate. Why it's important is that in the case where
14 we got the burner tip sales, we want to unduplicate.
15 In the case where we have the f.o.b. citygate
16 sales, we want to add in order to get a
17 representation of what the true retail price might
18 be; okay?
19 Now may we have the old three?
20 In terms of the whole flow of gas,
21 that's a real simplified schematic. Every place
22 that a line hits the box or a line hits a circle is
NEAL R. GROSS
COURT REPORTERS AND TRANSCRIBERS
1323 RHODE ISLAND AVE., N.W.
(202) 234-4433 WASHINGTON, D.C. 20005-3701 (202) 234-4433
66
1 a place where a sale can take place. A marketer can
2 sell at the well head. A marketer can sell at a
3 market hub. A marketer can sell at a storage
4 facility, at an interconnect, at a place where one
5 pipeline meets another. We do not know how much of
6 a marketer's sale they're going to be able to tell
7 us took place to a consumer in a given state. This
8 is something that we're going to go out and find
9 out. So we have that as a caveat.
10 Another thing we have as a caveat is
11 that we're interested in states to consumers -- I'm
12 sorry, sales to consumers by state, and we have
13 anecdotal stories from a couple of the marketers
14 already that say they sell to a big customer. The
15 customer takes it up in, let's say, the state of
16 Ohio and moves it to his plants that are located in
17 Ohio, Kentucky and Michigan. They know this, but
18 they don't know how much gas goes to Ohio, to
19 Kentucky and to Michigan. So again, a problem that
20 might end up really being problematic.
21 Okay. We're in a pretest mode now.
22 We've sent a draft to OMB for clearance in our
NEAL R. GROSS
COURT REPORTERS AND TRANSCRIBERS
1323 RHODE ISLAND AVE., N.W.
(202) 234-4433 WASHINGTON, D.C. 20005-3701 (202) 234-4433
67
1 pretest; we intend to mail out to the companies next
2 week. We're going to ask them to fill in a form for
3 one state, get it back; and mostly we're doing this
4 for two reasons: first, to find out whether they
5 can fill it in from their company records. I
6 imagine if they can't, we will hear very quickly.
7 Second, to get entree to them. We've tried
8 telephone contacts, and frankly we ran into a lot of
9 stone walls. They won't return our calls. We are
10 not business for them. As I said earlier, they're
11 not used to doing business with EIA; EIA is not used
12 to doing business with them.
13 Okay. Additional questions which we
14 can't do in a pretest because it wasn't in the
15 original OMB clearance.
16 Jerry -- is Jerry still here?
17 MR. MOUNT: Just gone.
18 MR. KASS: Just gone; okay. He, I was
19 hoping, could make me straight about what we can and
20 cannot do at the OMB connection.
21 But with the OMB clearance process now,
22 there are two iterations of federal register
NEAL R. GROSS
COURT REPORTERS AND TRANSCRIBERS
1323 RHODE ISLAND AVE., N.W.
(202) 234-4433 WASHINGTON, D.C. 20005-3701 (202) 234-4433
68
1 notices. In the first federal register notice, we
2 intend to find out what states the marketers do
3 business in and the stage of the gas flow, in which
4 the transaction takes place. If we find that we're
5 going to be able to capture a lot of gas at those
6 places with two "x's," that's different from where
7 the transactions take place in other of those
8 potential places.
9 Okay. We see some upcoming problems,
10 and this is where, oh, maybe next year we'll come to
11 the Committee and ask for some advice. But this is
12 really a heads-up.
13 What if we determine that we can expect
14 to fill in some of the gap in coverage? Right now,
15 we're missing 80 percent; okay? Let's say that we
16 can fill in, in a best-case analysis, 80 percent of
17 that; okay? That's different from if we can fill in
18 10 or 15 percent of that. This could happen because
19 trades that are out of scope. It could happen
20 because of insufficient data in corporate records.
21 It could happen for a lot of reasons. But we've
22 got, on the one hand, we could fill in a lot of the
NEAL R. GROSS
COURT REPORTERS AND TRANSCRIBERS
1323 RHODE ISLAND AVE., N.W.
(202) 234-4433 WASHINGTON, D.C. 20005-3701 (202) 234-4433
69
1 gap. On the other hand, we can't fill in a whole
2 lot.
3 Then we've got to make some decisions.
4 The first is, at what point is it worthwhile to
5 continue with the survey? We call it the "901"
6 because we gave it a name. Now with this survey
7 will come corresponding modifications to the monthly
8 857. We're going to unshade what our 400
9 respondents, through our monthly form, report.
10 They're going to have to report the transportation
11 revenue for their transportation gas. We're going
12 to unshade our annual report transportation thing.
13 So on an annual basis, all companies are going to
14 report a transportation revenue.
15 At what point is it worthwhile to go to
16 that effort? Basically, the question, I think,
17 comes to what's going to be the most beneficial to
18 our customers: if we go forward with the 901 or if
19 we do something else.
20 And the something elses that we can do,
21 just to give you three examples: one, we can
22 continue as we do. We publish all reported prices.
NEAL R. GROSS
COURT REPORTERS AND TRANSCRIBERS
1323 RHODE ISLAND AVE., N.W.
(202) 234-4433 WASHINGTON, D.C. 20005-3701 (202) 234-4433
70
1 We give a caveat table saying this is the coverage.
2 We let our customers do with it as they will. A
3 lot of customers misinterpret, misrepresent what our
4 numbers are. This is especially problematic when
5 you get part of the industry press picking up the
6 price table, not mentioning at all what the coverage
7 is, and going forward saying the price in state X is
8 $10 this month. Well, it is for a very little bit
9 of gas.
10 The second thing we could do is report a
11 price for state only if the coverage exceeds some
12 threshold. For instance, I said that there are
13 states with less than 5 percent. Are we doing
14 anybody any real benefit by reporting out that
15 state? Well, if the 5 percent is too little, what's
16 enough; okay? What's the threshold going to be?
17 Finally, we could drop the price series
18 from publication altogether. Are we doing a service
19 for publishing a 20 percent coverage price? The
20 belief has been that the prices we publish, the
21 prices we gather, are a biased estimate of what a
22 true price for that category of customer is, but it
NEAL R. GROSS
COURT REPORTERS AND TRANSCRIBERS
1323 RHODE ISLAND AVE., N.W.
(202) 234-4433 WASHINGTON, D.C. 20005-3701 (202) 234-4433
71
1 tracks well with what the true price is. This is
2 really an article of faith, and it's been the
3 religion that I've been working with for a while.
4 But we don't know; and if people share that article
5 of faith, that's one thing. If they think that it's
6 a true data point, they're clearly wrong. We know
7 it's a bias. We're assuming that it's a constant
8 bias. We have no idea what the magnitude to the
9 bias is.
10 Thank you.
11 MR. MOUNT: We have no formal discussant
12 for this presentation, but hopefully there are some
13 comments from the Committee.
14 MR. KENT: Well, this is really off the
15 cuff, but will the end of the world really come if
16 you quit publishing the price data? And I ask that
17 really, I guess, as a serious question, because
18 years ago, every time everybody said they wanted us
19 to publish prices, so we published prices so they
20 could beat us up about the prices we published.
21 And, I mean, this almost seems like it's self-
22 flagellation or something on our part. You know,
NEAL R. GROSS
COURT REPORTERS AND TRANSCRIBERS
1323 RHODE ISLAND AVE., N.W.
(202) 234-4433 WASHINGTON, D.C. 20005-3701 (202) 234-4433
72
1 why do we need this grief? I mean, unlike what the
2 presentation we heard earlier on reserves, you know,
3 where I think we all understand the absolute
4 essential nature of that, what would we lose if we
5 quit doing it?
6 MR. KASS: Okay. I can give you
7 anecdotes; okay? I get calls from time to time from
8 people who are in contracts that were written years
9 ago, where a price they paid is geared to a
10 published price. That's one use of our data. I
11 understand with the advent of the spot market and
12 with the advent of the future's market, more and
13 more contracts are not relying on EIA, and that's
14 probably as it should be. But we don't know how our
15 customers are using our data; okay? The one way to
16 find out is to not do it and see who squeals
17 loudest.
18 MR. MOUNT: Campbell?
19 MR. WATKINS: Let me make a couple of
20 comments and suggestions. First, have you looked at
21 the possibility of collecting information,
22 particularly on the transportation rates from the
NEAL R. GROSS
COURT REPORTERS AND TRANSCRIBERS
1323 RHODE ISLAND AVE., N.W.
(202) 234-4433 WASHINGTON, D.C. 20005-3701 (202) 234-4433
73
1 EBB's or electric bulletin boards and the lifelines?
2 MR. KASS: Extensively. It does not
3 fill the entire transportation classification. The
4 EBB's are at the margin of the rate.
5 MS. BISHOP: Use the mic.
6 MR. KASS: I'm sorry. The EBB's, to the
7 extent that they cover a price, cover the price at
8 the margin; not the long-term commitments. That's
9 one thing that we looked at early on.
10 MR. WATKINS: But an increasing
11 proportion of the gas is transported so-called at
12 the margin.
13 MR. KENT: Yeah.
14 MR. WATKINS: So that is not
15 insignificant information by any means.
16 The second idea is -- you touched on the
17 contracts -- whether you could, by a sampling
18 process, acquire information from contracts on the
19 pricing provisions, transportation provisions.
20 There are some data that are in the public domain in
21 any case. I think I'm right in saying that any gas
22 that crosses international borders, that those
NEAL R. GROSS
COURT REPORTERS AND TRANSCRIBERS
1323 RHODE ISLAND AVE., N.W.
(202) 234-4433 WASHINGTON, D.C. 20005-3701 (202) 234-4433
74
1 contracts have to be filed in the public domain in
2 the United States, and there is some information or
3 source of information on prices and transportation.
4 MR. KASS: That's interesting.
5 MS. BISHOP: People are saying they
6 cannot hear.
7 MR. WATKINS: Sorry?
8 MS. BISHOP: Talk louder.
9 MR. WATKINS: Sorry. Let me repeat what
10 I was saying about use of the contract's
11 information, where it can be in the public domain,
12 or obtained by satellite. A lot of the contract --
13 or some contract information for international flows
14 is in the public domain in any case. They offer --
15 I mean, I was really surprised, and so must you be
16 very uneasy about the amount of information you have
17 on the industrial sector. I mean, if your average
18 is 25 percent, as you pointed out, with respect to
19 New Mexico, you're very, very low.
20 Now since all that international gas
21 does reach a lot of market areas -- I mean, it's not
22 as if you're talking about a rim prospect. I
NEAL R. GROSS
COURT REPORTERS AND TRANSCRIBERS
1323 RHODE ISLAND AVE., N.W.
(202) 234-4433 WASHINGTON, D.C. 20005-3701 (202) 234-4433
75
1 suggest you could at least look at that.
2 Could you put up the flow of natural gas
3 graph, again, because I had a couple of comments on
4 that.
5 It goes to the question, a point of
6 clarification: I take it where you had the "X" with
7 the LDC, that indicates a citygate process.
8 MR. KASS: Right.
9 MR. WATKINS: The omission, or what
10 seemed to be an omission, is why you wouldn't want
11 to have an "X" either in the well head or the gas
12 plant as well.
13 MR. KASS: The way the survey was
14 designed, we're really aiming at getting a retail
15 price. The further away from the site of
16 consumption the sale takes place, the more
17 transportation costs we're missing, okay, which
18 flows back to the fact that we can't monitor
19 transportation costs.
20 MR. WATKINS: But perhaps related to my
21 earlier comment, if you did collect some of the,
22 let's say, point of -- whether the sale at the --
NEAL R. GROSS
COURT REPORTERS AND TRANSCRIBERS
1323 RHODE ISLAND AVE., N.W.
(202) 234-4433 WASHINGTON, D.C. 20005-3701 (202) 234-4433
76
1 MR. KENT: Let's say the point of "X" is
2 the gas pump.
3 MR. WATKINS: -- which is the base for a
4 lot of it, and again going back to the facts that
5 you may have some information in the public domain,
6 the transportation to the point of the price, that
7 could give you, say, a citygate price somewhere down
8 the line --
9 MR. KENT: That's what I thought they
10 did.
11 MR. WATKINS: -- if you had the well
12 head or gas plant exit price as well.
13 I'm just trying and groping around for
14 ways you could try and fill some of these gaps.
15 That's all I have.
16 MR. MOUNT: Anybody else on the
17 Committee?
18 Well, I suppose I would like to say the
19 obvious: the price does matter for analysis,
20 particularly of the industrial sector, and I find
21 the idea of trying to get information from the
22 brokers on the supply side very interesting. I'm
NEAL R. GROSS
COURT REPORTERS AND TRANSCRIBERS
1323 RHODE ISLAND AVE., N.W.
(202) 234-4433 WASHINGTON, D.C. 20005-3701 (202) 234-4433
77
1 sorry that the Committee hasn't commented on that.
2 Maybe I could encourage somebody to say something.
3 I feel that, in addition, this is the
4 sort of thing that may have to be considered for the
5 electric industry. It seems inevitable that there
6 are going to be brokers who both represent groups of
7 suppliers and brokers who represent groups of
8 consumers who were involved in competitive markets;
9 and I think that if there is one group that I would
10 like to know the price of gas about, it is
11 independent power producers, and presumably they're
12 one of the ones that you're missing.
13 Are there any comments from the public?
14 Oh, sorry, Campbell.
15 MR. WATKINS: Let me just make another
16 comment on what you just raised. I should say there
17 is a lot of information in the trade press, at least
18 on spot prices, hub centers and various points along
19 the system. So all is not completely bleak on the
20 gas pricing outlook, and the trade press
21 publications do provide some key information.
22 MR. MOUNT: Cal?
NEAL R. GROSS
COURT REPORTERS AND TRANSCRIBERS
1323 RHODE ISLAND AVE., N.W.
(202) 234-4433 WASHINGTON, D.C. 20005-3701 (202) 234-4433
78
1 MR. KENT: Well, let me just raise an
2 issue about who should be collecting this price
3 data, since you brought up the electric people.
4 What role should FERC be playing in this? And I
5 throw it out as a theoretical question because
6 somebody's going to have to have the clout.
7 I'm not that much familiar with other
8 gas markets, other than one in West Virginia anyway,
9 but I do think that what Roy said, if you go to
10 these brokers, they've got no incentive to report to
11 you. They may not really have the data in the way
12 that you want it. The way they keep data is
13 basically the way they're accountant wants it for
14 their tax purposes, and I just really begin to
15 wonder if we've got a problem here.
16 Certainly what you pointed out about the
17 IPP's and their consumer of gas is that if there's a
18 real regulatory issue here, that we're just going to
19 spend a lot of time beating our heads against the
20 wall, trying to get information that if somebody
21 doesn't have a bit of a stick, we're not going to be
22 able to get.
NEAL R. GROSS
COURT REPORTERS AND TRANSCRIBERS
1323 RHODE ISLAND AVE., N.W.
(202) 234-4433 WASHINGTON, D.C. 20005-3701 (202) 234-4433
79
1 That was all I was saying, because I
2 think you're absolutely right. As the electricity
3 goes the same path that natural gas went, we're
4 going to have more and more difficulty; and this
5 even comes back to the confidentiality issue that
6 we've had before, because brokers are certainly not
7 going to want to have their data put out there where
8 it might become public record for their competitors.
9 MR. MOUNT: Well, to offer a carrot in
10 this process, certainly a lot of the things that are
11 done in land grant universities gather information
12 that individuals might be reluctant to release, but
13 that they're actually very interested in looking at
14 reports that contain all the data, obviously without
15 the individual entities identified. So this is an
16 advantage to the people who are providing the data
17 on this, you know, as a long-standing tradition with
18 agriculture commodities.
19 Have you any final comments?
20 MR. KASS: No. I think I would like to
21 follow up with Campbell about the reporting of
22 important places.
NEAL R. GROSS
COURT REPORTERS AND TRANSCRIBERS
1323 RHODE ISLAND AVE., N.W.
(202) 234-4433 WASHINGTON, D.C. 20005-3701 (202) 234-4433
80
1 MR. MOUNT: So thank you all for
2 presentation.
3 We can move on to the next topic and
4 pick up some time for Dan. So we're moving now on
5 to an issue that I think many of us are interested
6 in, statistical issues pertaining to re-engineering
7 at EIA. All I can say as an introduction is I hope
8 that re-engineering is going better here than it is
9 at Cornell.
10 The first presentation is Measurement
11 Model for Information Management Processes by Nancy
12 Leach, Energy Markets and End Use.
13 MS. LEACH: As Tim said, I'm Nancy
14 Leach. I'm reporting today as a member of EIA's
15 business re-engineering or BR team. For those of
16 you who have handouts -- it looks like this -- the
17 last three pages are details on each measure. Oh, I
18 see Renee is giving them out right now. It provides
19 the measure, the target, the collection frequency
20 and the data collection method. As I talk, I may
21 not give every single piece of those, but you do
22 have that in the back.
NEAL R. GROSS
COURT REPORTERS AND TRANSCRIBERS
1323 RHODE ISLAND AVE., N.W.
(202) 234-4433 WASHINGTON, D.C. 20005-3701 (202) 234-4433
81
1 What I'm going to do this morning is
2 provide an overview of the performance measures for
3 only the information management.
4 Okay, the next slide -- or the IM
5 sections of the BR.
6 Want to do the next slide?
7 Essentially, these are the survey-
8 related processes: how to design the survey,
9 collect the data, process the survey, and then put
10 the data out. As you see, develop the design,
11 create and update the frame, request respondent
12 data, receive respondent data.
13 Next.
14 Then we import information. This is
15 when we use non-EIA data sources. We call it
16 imported data. We clean up the data, and we protect
17 the confidential data, and then we create what we
18 call the information components, or these are your
19 graphs, your charts, your tables, your explanatory
20 text.
21 The first set of measures are for survey
22 information. The first two refer to the
NEAL R. GROSS
COURT REPORTERS AND TRANSCRIBERS
1323 RHODE ISLAND AVE., N.W.
(202) 234-4433 WASHINGTON, D.C. 20005-3701 (202) 234-4433
82
1 availability time for the survey estimates and for
2 the survey data. The idea is to get the data out
3 faster than we currently do.
4 The third measure is to track the change
5 in respondent burden. This is to keep up with our
6 legislative mandate of -- I think it's a 10 percent
7 reduction in burden each year.
8 Our fourth is for edit performance
9 statistics.
10 You want to bring up the next chart?
11 The idea here is to identify good edits;
12 that is, edits that track truly erroneous data with
13 a minimum of respondent burden and processing
14 burden. A good edit changes bad data and then
15 provides better data, better quality.
16 Our performance statistics we defined
17 were frequency of errors detected, the number of
18 erroneous records identified, the number of true
19 errors identified -- again, trying to get at was
20 this valid data? Impact of identified errors. Did
21 it make a difference?
22 Sure, you've identified all of these
NEAL R. GROSS
COURT REPORTERS AND TRANSCRIBERS
1323 RHODE ISLAND AVE., N.W.
(202) 234-4433 WASHINGTON, D.C. 20005-3701 (202) 234-4433
83
1 errors. You went back to respondent. Respondent
2 said, "It's fine with me. That's exactly what I
3 told you, and I still agree that that's the correct
4 answer." Well, we don't change the record. We have
5 wasted our time by calling back and irritate a
6 respondent. So maybe that's an error that we would
7 want to drop. And then the number of changes made
8 as a result of validation.
9 Again, sure, identified all these
10 errors, but did I change anything because of it?
11 The emphasis -- let me just say -- can
12 we go back to the one with the yellow tag on it?
13 The emphasis from the BR Team was to
14 make these automatic; to try to put all of these
15 flags, these counts, within the edit and imputation
16 programs themselves. Right now a lot of this work
17 is done manually. You go to your error, your error
18 list, and you say, "Oh, yeah, I got five errors
19 flagged." It's more likely 500 usually -- and then
20 checking it all.
21 So the idea is to get all of this
22 created automatically so we don't have to sit up
NEAL R. GROSS
COURT REPORTERS AND TRANSCRIBERS
1323 RHODE ISLAND AVE., N.W.
(202) 234-4433 WASHINGTON, D.C. 20005-3701 (202) 234-4433
84
1 there and keep track of all of these edits.
2 The fifth survey information performance
3 measure is imputation performance statistics. Now I
4 was told -- now we can go to the imputation slide --
5 that some of you all might not be used to dealing
6 with the real world where we have recalcitrant and
7 imperfect respondents. So we do have to do some
8 imputation, and that's essentially the filling in of
9 the missing data.
10 We have unit non-response, where the
11 respondent has refused to complete the form at all,
12 or item response where somewhere in the midst of the
13 form, someone just doesn't answer a question. It's
14 skipped or whatever.
15 Do you have the next one?
16 So the imputation statistics that the
17 group came up with: the number of total data items
18 imputed, the percent of one particular data item
19 imputed, percent of the total data set imputed, and
20 the effect on the file data. Does imputation make a
21 difference? Is it worth the time and effort?
22 We identified two measures associated
NEAL R. GROSS
COURT REPORTERS AND TRANSCRIBERS
1323 RHODE ISLAND AVE., N.W.
(202) 234-4433 WASHINGTON, D.C. 20005-3701 (202) 234-4433
85
1 with data requests. Essentially, number one is your
2 postal returns. In the old world, where we mailed
3 forms as would be your postal return, how many bad
4 addresses did you have?
5 Number two is the cost. How much did
6 you have to spend to get the report out? And that's
7 the cost fer form. Again, the idea is to minimize
8 these. Let's get fewer returns and for less money.
9 Data receipt measures. We had four of
10 those. It's your percent data entry error by method
11 of entry. Your percent response is received by the
12 due date. This is a critical one for many of our
13 surveys. Our respondents don't respond on time.
14 Therefore, it throws everything off. So our target
15 here is to try to minimize that.
16 The cost per data item sell. How much
17 does it cost to get a particular form keyed? Is OCR
18 more expensive, as direct as key, whatever.
19 Then the fourth is time from deadline to
20 continue the next process. How long does it take to
21 get from one step to the next. In general, again,
22 the idea is to kind of cut down on the amount of
NEAL R. GROSS
COURT REPORTERS AND TRANSCRIBERS
1323 RHODE ISLAND AVE., N.W.
(202) 234-4433 WASHINGTON, D.C. 20005-3701 (202) 234-4433
86
1 time spent between all of these steps.
2 Next one.
3 We also looked at a performance measure
4 for imported -- again, this is your non-EIA data.
5 We often bring non-EIA data in to benchmark our own
6 data or to check sets of our data sources. Here,
7 the measure is the amount of time spent cleaning the
8 imported data. You know, do you spend all of your
9 time cleaning the data, so it's really not worth
10 bringing it in?
11 We also defined a frame or sample
12 measure. Again, trying to use our resources most
13 efficiently, we're looking at the number of births,
14 deaths, and updates categorized by source and
15 volume. Is this particular source worth the amount
16 of time or effort and money that we put in? Do we
17 update our form? Is it really worth our frame
18 updates?
19 And our last measure is on the
20 information component. Again, this is where you're
21 creating your graphs, your tables, your explanatory
22 text, reports. Here, the measure was designed to be
NEAL R. GROSS
COURT REPORTERS AND TRANSCRIBERS
1323 RHODE ISLAND AVE., N.W.
(202) 234-4433 WASHINGTON, D.C. 20005-3701 (202) 234-4433
87
1 number of accesses or hits in the electronic data
2 basis and transmittal of virtual and/or actual hard
3 copies. Again, the idea is to maximize those.
4 And Renee will continue on.
5 MR. MOUNT: Thank you.
6 The second presentation: Is There a
7 Summary Measure of Data Quality, Renee Miller,
8 Office of Statistical Standards.
9 MS. MILLER: Good morning. Can you all
10 hear me okay? Okay. Nancy told you about all kinds
11 of measures that we developed during our business
12 re-engineering process. But the one thing that we
13 don't have is a summary measure of data quality.
14 Now this issue came up several times,
15 and several times we concluded that it was very
16 important, particularly if we make our data more
17 timely. But we also concluded that we didn't know
18 how to do it. So we're now turning it over to you
19 and have the following questions.
20 Is it worthwhile having a summary
21 measure? Do you have ideas on what it should be?
22 Or maybe we should look at this from a different
NEAL R. GROSS
COURT REPORTERS AND TRANSCRIBERS
1323 RHODE ISLAND AVE., N.W.
(202) 234-4433 WASHINGTON, D.C. 20005-3701 (202) 234-4433
88
1 point of view and concentrate on the explanatory
2 notes in our publications. Should we do more to
3 standardize the descriptive material?
4 Not to worry. We're not asking you to
5 look at this from scratch. I'm going to describe
6 available measures, give background on previous
7 efforts, and give you an outline of a proposal that
8 Dwight French and I worked on during the business
9 re-engineering process, but that wasn't adopted.
10 Okay. To start with what's available,
11 we compute and publish revision error, the
12 difference between our preliminary and final
13 estimates. Now this measure in particular has been
14 criticized as not being a good measure of data
15 quality because it doesn't tell us about the final
16 estimates. How do we know that the final estimates
17 are any good?
18 There's an additional problem. Suppose
19 we don't revise the data? Does that mean there's no
20 error?
21 We also compute and publish sampling
22 error and response rates; and while these are
NEAL R. GROSS
COURT REPORTERS AND TRANSCRIBERS
1323 RHODE ISLAND AVE., N.W.
(202) 234-4433 WASHINGTON, D.C. 20005-3701 (202) 234-4433
89
1 considered important measures, the general feeling
2 was they don't tell the whole story. We also
3 discuss non-sampling error in our publications, but
4 it's a discussion. It's not a particular measure.
5 Going back in time, EIA used to conduct
6 what we called validation studies. They were
7 cradle-to-grave examinations of the data; and as I
8 describe in the paper, they proved to be both
9 extensive and very expensive and were eventually
10 discontinued when our budget was reduced in the
11 1980's. But they did provide information on the
12 major sources on non-sampling error.
13 For example, they included an audit of
14 company records which gave us information on
15 measurement error. They also included a search for
16 deficiencies in the frame which gave us information
17 on coverage, and they also included a comparison of
18 hard copy with the automated file which gave us
19 information on processing error. And sometimes the
20 information we got was actually quantifiable. But
21 what we didn't have was a way of adding it all up to
22 get total survey error because sometimes the errors
NEAL R. GROSS
COURT REPORTERS AND TRANSCRIBERS
1323 RHODE ISLAND AVE., N.W.
(202) 234-4433 WASHINGTON, D.C. 20005-3701 (202) 234-4433
90
1 were offsetting.
2 We also, in the early days, did a lot of
3 comparisons with other series; and in the early
4 days, the comparative series were plentiful. We
5 considered the series that we were interested in the
6 "reference" series, and we computed the comparative
7 series as a percentage of the reference series.
8 So let's take crude oil imports, for
9 example. We had three comparative series and we had
10 data for three years. So we considered it nine
11 independent estimates. In other words, we had nine
12 independent estimates of the ratio described in the
13 first bullet. Well, with nine estimates, you can
14 compute a mean, a standard deviation of the mean, at
15 a 95 percent confidence interval. And that's what
16 we did. So for crude oil imports, the 95 percent
17 confidence interval was 99.2 to 100.8, and we
18 therefore concluded that our data on imports were
19 accurate to within 1 percent.
20 Well, as you might imagine, these
21 conclusions about accuracy were not well received,
22 because we didn't have all that much information
NEAL R. GROSS
COURT REPORTERS AND TRANSCRIBERS
1323 RHODE ISLAND AVE., N.W.
(202) 234-4433 WASHINGTON, D.C. 20005-3701 (202) 234-4433
91
1 about the comparative series. In some cases, we
2 didn't really even have a clear idea of how the
3 comparative series were obtained.
4 So although we continued to perform
5 comparisons, we stopped coming to conclusions about
6 data quality based on them, and we shifted our focus
7 to explaining and resolving discrepancies.
8 Okay. Moving on to more recent times,
9 we've looked at elements of data quality, and this
10 may look familiar to Dr. Kent. It's from his 1991
11 presentation at the annual ASA meeting.
12 We identified four elements:
13 timeliness, customer satisfaction, consistency and
14 continuity. This approach differs from the work
15 that was performed in the validation studies where
16 we were trying to measure total survey error. Here
17 we were looking at the fitness of use of our
18 statistical products. And since 1991, we've made a
19 lot of progress in measuring timeliness and customer
20 satisfaction, but we don't have measures for
21 consistency or continuity. By consistency, we meant
22 how well our data compare with other series. Were
NEAL R. GROSS
COURT REPORTERS AND TRANSCRIBERS
1323 RHODE ISLAND AVE., N.W.
(202) 234-4433 WASHINGTON, D.C. 20005-3701 (202) 234-4433
92
1 they consistent over time -- or rather, were they
2 internally consistent; and for continuity, we meant:
3 are we measuring the same thing over time?
4 Okay. Moving on to even more recent
5 developments, some of you may recall at a few
6 meetings ago, Paul Biemer suggested that we prepare
7 quality profiles where we describe what's known
8 about each source of non-sampling error. And this
9 came up when we were discussing data on imports and
10 exports.
11 Well, we didn't prepare a quality
12 profile for imports and exports, but we recently
13 completed one for the Residential Energy Consumption
14 Survey. It's this purple publication. This was a
15 joint effort between the Office of Energy Markets
16 and End Use and Office of Statistical Standards, and
17 Tom Jabine, a former committee member, was the
18 principal author.
19 Another development also in the energy
20 consumption area -- which, by the way, is Nancy's
21 area here -- is a succinct set of notes for the
22 Commercial Building Energy Consumption Survey, or
NEAL R. GROSS
COURT REPORTERS AND TRANSCRIBERS
1323 RHODE ISLAND AVE., N.W.
(202) 234-4433 WASHINGTON, D.C. 20005-3701 (202) 234-4433
93
1 CBECS. And that's what you have in attachment one
2 of the paper. These notes describe the survey
3 methodology, sampling and non-sampling error, in a
4 very concise way.
5 So now that I've given you an idea of
6 what we've looked at in the past, let me move on to
7 the business re-engineering proposal. As I said,
8 this issue came up several times, and several times
9 we thought it was important, but that it wasn't
10 doable. But Dwight French and I went ahead and
11 worked on it anyway.
12 We realized the reason that it seems
13 undoable was that there were several dimensions of
14 data quality, and we listed some of them. There's
15 sampling error, measurement, coverage, non-response,
16 what we called "methodological consistency, which is
17 really the same thing that Dr. Kent called "continu-
18 ity." And these dimensions differ in the ease in
19 which they can be quantified. Sampling error, for
20 instance, can be computed directly from the data.
21 Now methodological consistency, we didn't know how
22 to quantify that.
NEAL R. GROSS
COURT REPORTERS AND TRANSCRIBERS
1323 RHODE ISLAND AVE., N.W.
(202) 234-4433 WASHINGTON, D.C. 20005-3701 (202) 234-4433
94
1 Ideally, you would like a series to be
2 stable over time -- not to have any breaks -- but
3 sometimes, due to changes in the industry, it's
4 inevitable that you revise your data collection; and
5 the issue is: Does the series get penalized for
6 having breaks?
7 Now some of these other dimensions -- it
8 sounds like they should be easily quantified, you
9 know, such as measurement error. But we don't have
10 information for each survey on an ongoing basis.
11 This is the type of information that we obtained
12 from the validation studies which have been
13 discontinued. So the question is: what to do?
14 Well, we thought that since we did have
15 some information from each survey, what we should do
16 is gather the information together and then rate
17 each survey on each of the dimensions. And we would
18 have two categories: the level of knowledge we have
19 and the quality level. We thought we would use a 1-
20 to-5 scale since that seems to work in our customer
21 satisfaction surveys, where 5 meant things were just
22 wonderful. On the other hand, 1 meant they were not
NEAL R. GROSS
COURT REPORTERS AND TRANSCRIBERS
1323 RHODE ISLAND AVE., N.W.
(202) 234-4433 WASHINGTON, D.C. 20005-3701 (202) 234-4433
95
1 quite so wonderful.
2 So to give an example of how this would
3 work, let's take non-response. A survey might get a
4 score of 5 on the level of knowledge for a non-
5 response if, say, there was documentation available
6 on the response rate, on our follow-up and
7 imputation procedures, and the key information was
8 presented in the publications. A survey may get a 5
9 on the quality level for a non-response if, say, the
10 response rate was 98 percent in terms of both number
11 of respondents and volumes reported. And in
12 attachment two of the paper, there are lots of other
13 examples of how this scale might work.
14 Well, as you can see, attachment two,
15 you know, was getting kind of complicated. As a
16 result of these complications and other unresolved
17 issues, this Business Re-engineering Team decided
18 not to pursue this procedure.
19 Some of the issues were: who would do
20 the rating? Now the Committee might be thinking,
21 "Well, you know, this is kind of an interesting
22 abstract issue." But one of the possibilities was
NEAL R. GROSS
COURT REPORTERS AND TRANSCRIBERS
1323 RHODE ISLAND AVE., N.W.
(202) 234-4433 WASHINGTON, D.C. 20005-3701 (202) 234-4433
96
1 we were thinking that maybe the Committee would help
2 us do the rating.
3 One of the other issues was the time
4 involved. The perception was that this would be
5 very time consuming.
6 The third issue was: Could we really
7 ensure consistency? A general feeling was if we
8 could be precise enough that we could ensure
9 consistency, would we really be giving any more
10 information than you could obtain from the
11 explanatory notes?
12 So that brings us back to the original
13 questions. Is it worthwhile having a summary
14 measure? Are there ideas on what it should be? Or
15 should we take a different approach and try to do
16 more to standardize the descriptive material.
17 Thank you.
18 MR. MOUNT: Discussant Dan Relles.
19 MR. RELLES: Regarding the question:
20 "Should you have a summary measure," yesterday my
21 answer was no, but today it's yes. So I'll say why.
22 The "no" was a knee-jerk reaction based
NEAL R. GROSS
COURT REPORTERS AND TRANSCRIBERS
1323 RHODE ISLAND AVE., N.W.
(202) 234-4433 WASHINGTON, D.C. 20005-3701 (202) 234-4433
97
1 on two observations. One is the summary quality
2 measure that has come out of statistics related to
3 modeling is one of the most misused objects I know
4 of; namely, R-squared. R-squared is interpreted by
5 most people as being sort of a measure of the
6 quality of a regression model. It's equal to 1
7 minus the sum of square's error over the sum of
8 square's total; and the sum of square's error indeed
9 is a measure of quality in the sense that the
10 smaller that gets, the better you are. But the
11 denominator is a measure of diversity of the
12 original population, and that's something you
13 couldn't do anything about.
14 So if, for example, I took the amount of
15 heating oil consumed during the winter from RECS,
16 and I kind of took the homogeneous population of
17 people who lived in, I don't know, Tennessee, and
18 regressed that on income, I'd probably get an R-
19 square that was around .15 maybe. But if I took the
20 entire country, I think the regression would
21 rediscover that the south and the west are a little
22 warmer than the northeast, and I'd probably get a
NEAL R. GROSS
COURT REPORTERS AND TRANSCRIBERS
1323 RHODE ISLAND AVE., N.W.
(202) 234-4433 WASHINGTON, D.C. 20005-3701 (202) 234-4433
98
1 regression in the neighborhood of .90.
2 Nevertheless, R-squared is highly touted
3 by people as a measure of quality; and if you talk
4 to social scientists, the first question they'll ask
5 you is: Was it over 20 percent? If the answer's
6 no, they don't want to talk to you anymore even
7 though you can have R-squares of .05 that are
8 tremendously important for policy purposes, because
9 they can measure fairly steep relationships.
10 So, okay, the potential for misuse is
11 one thing to recognize. The other thing to
12 recognize is that any measure of quality is even
13 more complicated than what you described because it
14 depends on the uses to which it will be put.
15 So let's take a number like 1.0, which
16 many of you will recognize. But I don't think that
17 many of you know that it's the sine of 89 degrees to
18 one significant digit. Now is that a quality
19 number? Well, I calculated it last night on my
20 calculator, and I assure you -- and I did it three
21 times and I got the same answer each time. So I
22 assure you it's a quality number.
NEAL R. GROSS
COURT REPORTERS AND TRANSCRIBERS
1323 RHODE ISLAND AVE., N.W.
(202) 234-4433 WASHINGTON, D.C. 20005-3701 (202) 234-4433
99
1 But now ask me, "Okay, well, what am I
2 going to use that number for?" Well, if I were to
3 try to use it to plot a sin curve, it would
4 correctly get me up to close to one. So that would
5 be good. If I wanted to compute a cosine, it's 1
6 minus the sin-squared, square rooted, I think, and
7 that would get me close to zero. But if I wanted to
8 compute a tangent, it would point me to infinity.
9 So it's pretty lousy for that. So what was
10 apparently a quality number is no good for that
11 particular application.
12 By the way, that quality number has one
13 attribute that I think you all wish the reserve
14 numbers would have; namely, the value of the number
15 doesn't degrade over time. I mean, the sin of 1 is
16 going to be the same thing tomorrow. So you can ask
17 the question, "Well, okay, that's a quality number.
18 But how can I get an even higher quality number?"
19 The answer is, "Well, let's compute it to two
20 significant digits." And the answer there is .98.
21 That's still probably not good enough for computing
22 tangents, but it's better than 1.0 for many other
NEAL R. GROSS
COURT REPORTERS AND TRANSCRIBERS
1323 RHODE ISLAND AVE., N.W.
(202) 234-4433 WASHINGTON, D.C. 20005-3701 (202) 234-4433
100
1 things.
2 I could go on and perhaps produce -- do
3 the best job I can and produce it out to 100
4 significant digits but the main point that I'm
5 trying to get across is that you really have to ask
6 what are you going to use that number for?
7 I guess another number that comes up a
8 lot in statistics is the cumulative normal
9 probabilities. Let's say at 3 or 4 the value is
10 1.0000, which is fine for a lot of things, like how
11 significant is this regression coefficient. But
12 it's pretty lousy for things like computing hazard
13 functions or likelihood functions or other kinds of
14 things that depend critically on, not the cumulative
15 CDF itself, but sort of 1 minus the cumulative.
16 So with that preamble, I was really sure
17 that the idea of a single summary measure was really
18 a bad idea.
19 But on the other hand, you look at how
20 many smart people have wrestled with this, and you
21 have to say, "Well, there must be something there."
22 It's kind of like ravioli's a really
NEAL R. GROSS
COURT REPORTERS AND TRANSCRIBERS
1323 RHODE ISLAND AVE., N.W.
(202) 234-4433 WASHINGTON, D.C. 20005-3701 (202) 234-4433
101
1 good idea, because if you look at how many cultures
2 have adopted something similar to ravioli, like the
3 Chinese have dim sum and the Jews have kreplach, it
4 must be a good idea.
5 So what I tried to do is look for kind
6 of a thread that would unite a lot of the stuff that
7 I saw today. The idea, I believe, is that quality
8 really is a number that you ought to attach to a
9 process that data have gone through; and I want to
10 propose a process where the numbers are 1, 2, 3 and
11 4. I guess I would assert that the higher the
12 number, the higher the quality.
13 The quality, what it's really trying to
14 measure, is the degree to which you've empowered the
15 user of the number to deal intelligently with the
16 problems of the numbers. So by telling you that the
17 sign of 89 degrees is 1.0 to two significant digits
18 -- or to one significant digit, I've given you all
19 the information you need to deal intelligently with
20 that number.
21 And that's what I want to try to do with
22 regard to the things you collect in surveys. My
NEAL R. GROSS
COURT REPORTERS AND TRANSCRIBERS
1323 RHODE ISLAND AVE., N.W.
(202) 234-4433 WASHINGTON, D.C. 20005-3701 (202) 234-4433
102
1 intention is to sort of find a place to put each of
2 the things we saw today, as well as Calvin's earlier
3 remarks. So I'm going to propose a 4 point scale.
4 The higher you go on the scale, the higher the
5 quality of the data or, equivalently, sort of the
6 more of the data are understood so you can deal with
7 its problems.
8 On the first level, I think timeliness,
9 consistency -- oh, and these things also tend to
10 correspond to the order in which you do these things
11 and the order in which you might actually chop off
12 doing them if your budget comes into play. I think
13 everything that EIA tries to do, sort of achieves at
14 least level one, namely, it's timely, consistent and
15 continuous. I think over the presentations I've
16 seen over the last couple of years, I mean, there's
17 every effort given to getting the data in faster,
18 higher quality, pay attention to whether or not it's
19 consistently coded over time. If not, alert the
20 user of that. And I believe these are the natural
21 initial steps that one takes when compiling any data
22 set. So I'll call those sort of the prima facie
NEAL R. GROSS
COURT REPORTERS AND TRANSCRIBERS
1323 RHODE ISLAND AVE., N.W.
(202) 234-4433 WASHINGTON, D.C. 20005-3701 (202) 234-4433
103
1 validity observations. I believe they're satisfied
2 with virtually all data that we see here.
3 The second level I would call
4 "metadata," which means data about data, and that's
5 where I'd put a lot of the stuff that Nancy and
6 Renee showed us this morning. First, you have to
7 know what values the data can take; and what I think
8 of there is building a database that describes the
9 data you have. I think of things like value the
10 data can take on. What are the values of null
11 values? What is the frequency of imputing
12 something? How often are the data missing? What's
13 the time rate of decay of the information?
14 It's sort of all the stuff about a
15 variable or a data set you'll need to kind of make
16 initial judgments about it. I think of that as
17 perhaps something that could exist on the Web.
18 Wherever you see a number on -- if you had a
19 database that described all of your variables, a
20 consistent metadata base -- when I was on the Web
21 before I came here, I thought how nice it might have
22 been to be able to click on a data value and kind of
NEAL R. GROSS
COURT REPORTERS AND TRANSCRIBERS
1323 RHODE ISLAND AVE., N.W.
(202) 234-4433 WASHINGTON, D.C. 20005-3701 (202) 234-4433
104
1 open up something that explained more about that
2 than I could see. As a matter of fact, I couldn't
3 see any metadata on the Web when I went looking for
4 it.
5 But again, if you had a database
6 organized like that, it would be easy for you, I
7 believe, to click on any number and get sort of the
8 standardized description out of that. I believe
9 your lists today that include errors of various
10 kinds and everything else would be valuable things
11 to pop up when I clicked on that number.
12 The other thing might be as a management
13 tool, because when EIA is having to face
14 information, face its cuts, it would be really nice
15 to be able to have kind of a common format for
16 descriptions of databases and variables in it so
17 that if, for example, you decided it was really
18 important to look at, in eliminating a survey,
19 whether timeliness was really that critical in it,
20 that you could just kind of grab the time dimension
21 descriptors of the data and at least be reminded now
22 that this reserves data set is really time important
NEAL R. GROSS
COURT REPORTERS AND TRANSCRIBERS
1323 RHODE ISLAND AVE., N.W.
(202) 234-4433 WASHINGTON, D.C. 20005-3701 (202) 234-4433
105
1 and perhaps another data set is not.
2 So I think metadata building is really
3 an important activity and that all of the things
4 that you and Nancy -- Renee and Nancy discussed this
5 morning really ought to go into metadata
6 descriptions. So anything that had full metadata on
7 it, I would give it two.
8 Then there's another level, which I'll
9 call level three, which is -- I think what the user
10 would need in addition to be able to deal
11 intelligently with the errors, and that's a document
12 like this, which I think is a marvelous document.
13 It describes a survey in a way that if you gave me
14 the data and gave me this book, I think I wouldn't
15 shoot myself in the foot. It talks about the
16 design, the coverage, non-response, measurement
17 error, data processing and imputation, how to do
18 estimation and sampling error -- which includes a
19 discussion of weights -- comparing it with other
20 estimators, and some suggestions for data users.
21 I really think this is a terrific model
22 of how to document your surveys, and I believe that
NEAL R. GROSS
COURT REPORTERS AND TRANSCRIBERS
1323 RHODE ISLAND AVE., N.W.
(202) 234-4433 WASHINGTON, D.C. 20005-3701 (202) 234-4433
106
1 -- oh, yeah, it's the thing that Renee waved too.
2 It's the residential energy consumption survey
3 quality profile. There was a similar one in a much
4 earlier draft stage for the commercial -- what was
5 it? CBEC, Commercial Buildings Energy Consumption.
6 But RECS is definitely at least a three in quality
7 given that, and the other one isn't quite done yet.
8 Again, that's likely to come after you
9 assert your timeliness and you build your metadata.
10 It's a lot of effort to produce one of these
11 things, and I fear that in the budget cutbacks,
12 you're not going to be able to do too many of these.
13 But that's okay. You just advertise your data as a
14 two instead of a three and blame it on the budget.
15 How do you make it a four? Well, I
16 think making it a four is building still more
17 metadata, and that's sort of user-oriented metadata.
18 It's about the users' use of your data; and again
19 I'd like to see databases developed that have that
20 information, although I recognize that's even more
21 expensive.
22 But a good example would be yesterday we
NEAL R. GROSS
COURT REPORTERS AND TRANSCRIBERS
1323 RHODE ISLAND AVE., N.W.
(202) 234-4433 WASHINGTON, D.C. 20005-3701 (202) 234-4433
107
1 heard about NEMS, and there's certain data that goes
2 into NEMS, like RECS, and is RECS quality data?
3 Well, I guess the question I'd want to ask is:
4 Suppose the RECS data -- what sort of overall
5 contribution of RECS to the total error produced at
6 the end-user result by NEMS -- I guess I'd want to
7 know the degree to which RECS, even though it's
8 already a three, whether it's contributing 1 percent
9 of the error, 10 percent of the error, 50 percent of
10 the error; and I'd like to bring that information up
11 front. You know, the user that I presume needs the
12 document, that would be the NEMS people. It's
13 another plug to get them to look at random error and
14 error in general. I guess I'd like to -- since RECS
15 is a survey, I'd kind of like to see them boot-strap
16 RECS and, for various boot-strap samples, look at
17 the variations in outcomes from NEMS of key
18 quantities and, from that, get a sense of what the
19 contribution of RECS is to their uncertainty.
20 I think in discussing reserves this
21 morning, I must apologize for listening with one ear
22 because I was trying to think about what I was going
NEAL R. GROSS
COURT REPORTERS AND TRANSCRIBERS
1323 RHODE ISLAND AVE., N.W.
(202) 234-4433 WASHINGTON, D.C. 20005-3701 (202) 234-4433
108
1 to say, but it was pretty clear that one didn't need
2 a fancy model like NEMS to argue that the time decay
3 rate of reserve estimates was so sensitive that
4 there's all these incredible decisions that are
5 being made that are going to incur tremendous error
6 if RECS doesn't -- if the reserves go to once every
7 two years. Again, that's the kind of thing that I
8 think needs to find its way into metadata bases that
9 describe things like the reserve series so that it
10 doesn't escape management attention when budgets are
11 being looked at, if for no other reason.
12 But again, as a user, not only are you
13 giving me a three, but you're also giving me a
14 database of what other users are doing, and I think
15 that would be, you know, really valuable for me to
16 make even more sense of things.
17 So I guess at level four, I would say if
18 a data set has been augmented by a metadata set that
19 describes user requirements -- not user
20 satisfaction, because I don't care about user
21 satisfaction as much as I care about how much --
22 well, because users are never -- it doesn't cost
NEAL R. GROSS
COURT REPORTERS AND TRANSCRIBERS
1323 RHODE ISLAND AVE., N.W.
(202) 234-4433 WASHINGTON, D.C. 20005-3701 (202) 234-4433
109
1 them anything, so they're never going to be
2 satisfied. But if I understood how much the data
3 was impacting the precision in what the user cares
4 about, then I'd be able to make an intelligent
5 decision about whether it's worth spending more
6 money to increase the quality of the data. And I
7 just don't think that information is known at this
8 point.
9 It says here I should pontificate, so I
10 guess I'll do that at this point. It seems like
11 we're always coming back to the same issue. Quality
12 gets mentioned everywhere, and quality issues are
13 just always to nest. It's kind of like if you
14 mention quality enough times, you've shown your
15 awareness of it, so you don't have to deal with it.
16 That's a little extreme, but I even saw it in the
17 AEO which I got sent a couple of days ago. Right in
18 the front, there was all this preamble about how
19 important quality is and how uncertain we were about
20 certain things, and then, you know, there's all this
21 discussion out to the year 2015 that doesn't mention
22 any uncertainty the rest of the time.
NEAL R. GROSS
COURT REPORTERS AND TRANSCRIBERS
1323 RHODE ISLAND AVE., N.W.
(202) 234-4433 WASHINGTON, D.C. 20005-3701 (202) 234-4433
110
1 So I guess I want to pontificate more on
2 the need to deal with uncertainty and to deal with
3 error and to try to be more imaginative on doing
4 that. I know that in the past -- and I think
5 statistics have come a long way on that dimension,
6 too. It used to be the way to deal with that was
7 develop more complicated models. I remember
8 earlier, you know, we were hearing about uncertainty
9 in NEMS, and there was a whole team trying to deal
10 with those models on an analytical level; and those
11 are hard problems -- especially when you've got a
12 black-box computer program that you're not going to
13 reopen no matter what because it runs to completion
14 without error.
15 But I think statistics has developed
16 techniques over the years for looking at uncertainty
17 -- things like the boot strap, things like multiple
18 imputation, things like just seat-of-the-pants-
19 perturbing parameters that basically argue there's
20 enough compute power out there. We can afford to
21 re-run our models many times and see how much
22 variation there is there. And I believe that, you
NEAL R. GROSS
COURT REPORTERS AND TRANSCRIBERS
1323 RHODE ISLAND AVE., N.W.
(202) 234-4433 WASHINGTON, D.C. 20005-3701 (202) 234-4433
111
1 know, quality ultimately has to be answered in terms
2 of how much does it impact the predictions that we
3 really care about, and my feeling is that we haven't
4 seen enough of that here.
5 So Tim's been asking me, before I run
6 off, what do I think I would like to see?
7 Obviously, a continuation of my discussion for next
8 time. I think that's all I really want to say.
9 Thank you.
10 MR. MOUNT: So, thank you. Samprit?
11 MR. CHATTERJEE: I have no wisdom, or
12 special comments to add. No special wisdom, but
13 just some comments to add and maybe a few questions.
14 The first thing is I think to liven up
15 the proceeding, I disagree with Dan that quality can
16 be summed up in one number. You know, it's much too
17 complex a process to be summed up in a number,
18 although I think we should, in your publication,
19 indicate the quality of the data which you're
20 presenting; and from that point of view, one of the
21 points which is mentioned, which I like very much,
22 is for each data series or data surveys to present
NEAL R. GROSS
COURT REPORTERS AND TRANSCRIBERS
1323 RHODE ISLAND AVE., N.W.
(202) 234-4433 WASHINGTON, D.C. 20005-3701 (202) 234-4433
112
1 what you call a quality profile, which is a multi-
2 attribute, multi-variate characteristic which would
3 give the person, the user, roughly the kind of
4 things which they need to know in using the data
5 analysis. And some of the points which you also
6 showed in your transparency No. 7, sampling error,
7 measurement error, coverage, non-response,
8 proportion imputed and so forth, I think we'll give
9 the person using it as much information as they need
10 to know in order to use the data for the analysis.
11 The second point which I had is among
12 the elements of data quality, I think Dan said about
13 that, too, is customer satisfaction. I think the
14 customer satisfaction is a function of the data
15 quality. You know, what else the customer should be
16 satisfied with if the quality of data which he or
17 she is getting is not very good. So I think
18 elements of customer satisfaction is a function of
19 elements of data quality, and I know in that I might
20 be stepping on toes, but anyway.
21 The other point which I had was in
22 thinking about quality, if you look at a quality
NEAL R. GROSS
COURT REPORTERS AND TRANSCRIBERS
1323 RHODE ISLAND AVE., N.W.
(202) 234-4433 WASHINGTON, D.C. 20005-3701 (202) 234-4433
113
1 profile, it's not just a time -- you know, a one-
2 time thing. I think quality can be understood if,
3 for successive rounds, successive years, see how the
4 numbers are changing. It's the trend in these
5 dimensions which will indicate improvement of
6 quality or stagnation or decrease in quality. So I
7 think it's the quality measured when one point of
8 time is very meaningless, because it is -- what do
9 you call it -- a standard; and how it does is shown
10 by, I think, the trend over time.
11 MR. MOUNT: Bradley?
12 MR. SKARPNESS: You know, Dan brought up
13 many very good points about data quality and what's
14 really important, and I just wanted to reemphasize
15 that, that the biggest thing that I get in trouble
16 with with a data set is that I do an analysis and
17 then I find out later that I didn't have some
18 critical piece of information, and I really am
19 making a statement and I didn't understand the data
20 as well as I should have in the first place. And
21 the reason for that is that I'm using other people's
22 data and that I have to read their documentation and
NEAL R. GROSS
COURT REPORTERS AND TRANSCRIBERS
1323 RHODE ISLAND AVE., N.W.
(202) 234-4433 WASHINGTON, D.C. 20005-3701 (202) 234-4433
114
1 depend on that; and as I delve into it more, they
2 either omitted a critical step or a critical part of
3 how they collected the data, or I wasn't informed of
4 that, and then until, you know, it's too late
5 sometimes. And so the documentation of how the data
6 was collected and what were the circumstances in
7 those types of pieces of information are real
8 critical and should be carried on here.
9 So there was one of these -- is the
10 descriptive material and the quality, but I would
11 like to say, you know, not only the summary stuff,
12 but actually some of the effort in things that
13 occurred while you were collecting the data is also
14 very important to try to understand what its
15 usefulness is and its consistency, too.
16 On another note, the customer
17 satisfaction part I would break out a little bit.
18 Because you do have this data out on the Web or on
19 the Internet or in a Web site, the transportability
20 of the data becomes a real pain sometimes. It costs
21 a lot of effort to get the data to us, do something
22 with it and get it into SAS, basically.
NEAL R. GROSS
COURT REPORTERS AND TRANSCRIBERS
1323 RHODE ISLAND AVE., N.W.
(202) 234-4433 WASHINGTON, D.C. 20005-3701 (202) 234-4433
115
1 I would like people, you know, to think
2 about that part of it, and then the utility and
3 usefulness ultimately -- sort of the two extremes of
4 what you were saying. You've got this timeliness.
5 Get it in, but then how useful is it in trying to
6 quantify that. That's all.
7 MR. MOUNT: Brenda?
8 MS. COX: I also wanted to comment on
9 Nancy's paper, which, by the way, I thought was
10 excellent. Actually, I thought both papers were
11 excellent, but Nancy's I'm going to send to someone
12 at MPR because we've been trying to develop the same
13 methods. Do you have all these in there?
14 I think at a minimum we should at least
15 try to describe the aspect of quality. You're
16 coming at it more the quality of what we're doing,
17 physically producing -- not looking at timeliness so
18 much as this product here. Once you finally get it,
19 what's it like? I would say that at a minimum you
20 want to describe and quantify your quality so that
21 at least your sophisticated users, which would
22 include yourself, can understand the quality of what
NEAL R. GROSS
COURT REPORTERS AND TRANSCRIBERS
1323 RHODE ISLAND AVE., N.W.
(202) 234-4433 WASHINGTON, D.C. 20005-3701 (202) 234-4433
116
1 you've got. And you've got measures here that are
2 going in that direction. Like the percent of data
3 that's imputed, I think at least descriptions of the
4 surveys should include things like this.
5 Now I would say -- I've noticed this in
6 the EIA publications, too, by the way. I would
7 suggest that you think about a standard outline for
8 your methodology discussions and a standard outline
9 where you actually tell people, "Here's the kind of
10 things you should be discussing." You should give
11 an explicit definition for your target population.
12 You should give the objectives of the survey. You
13 should tell what is the unit from which you're
14 collecting data and who are they reporting for.
15 In the Reserves Program, it wasn't
16 totally clear to me. Some of those items weren't
17 totally clear to me from the discussion in the back.
18
19 And then things like what percent of the
20 data are imputed and items like this. So a general
21 -- so that people actually know what they should be
22 going for would be helpful, I think.
NEAL R. GROSS
COURT REPORTERS AND TRANSCRIBERS
1323 RHODE ISLAND AVE., N.W.
(202) 234-4433 WASHINGTON, D.C. 20005-3701 (202) 234-4433
117
1 At least, then, with measures like this
2 and a description that would let me get a feel for
3 the whole thing, then I could say, "Well, gee, I
4 felt pretty confident here. Ninety-nine percent
5 response rate. Great." But then when you go in and
6 you look and you go, "Wait a minute. A lot of this
7 data is being estimated." I don't know how much, by
8 the way. The report didn't say -- or maybe I didn't
9 look closely enough. I will say that. I might not
10 have looked closely enough.
11 There's some other areas where you might
12 want to think about quality measures, like sampling
13 and weighting. Just some measures like design
14 effects or the design effect running away. Maybe
15 some clustering effects and things like that, so
16 that you get a sense of what the data are like.
17 They're more useful for us as designers and doers in
18 the sense that you look at them and we say, "Gee,
19 look at that design effect. Maybe you'd better look
20 a little further."
21 I was looking at a design where they
22 were over sampling intentionally, but I got down in
NEAL R. GROSS
COURT REPORTERS AND TRANSCRIBERS
1323 RHODE ISLAND AVE., N.W.
(202) 234-4433 WASHINGTON, D.C. 20005-3701 (202) 234-4433
118
1 the cell that was defining the ultimate level of
2 over sampling -- calculated design effects running
3 away, and I was seeing design effects of nine and I
4 went "Wait a minute." You know exactly what that
5 means. Bad design. Bad design. But those kind of
6 measures would be very helpful, I think.
7 So that would be what I suggest is a
8 simpler technique.
9 MR. MOUNT: Anybody else with comments
10 from the Committee?
11 I wanted to say that I was entirely
12 persuaded by Dan, and I also started off as a major
13 skeptic about being able to come up with a measure.
14 I think that really what you've come up with is a
15 sort of categorization and that the value of this,
16 particularly for people trying to use data
17 electronically, of having sort of a compact summary
18 of what you can expect to be able to find about a
19 data set, I think is very valuable.
20 However, it obviously isn't a substitute
21 for metadata itself; and I think that there, as I've
22 argued before, there really are opportunities to
NEAL R. GROSS
COURT REPORTERS AND TRANSCRIBERS
1323 RHODE ISLAND AVE., N.W.
(202) 234-4433 WASHINGTON, D.C. 20005-3701 (202) 234-4433
119
1 standardize the way that data are documented
2 electrically, and there ought to be much more
3 consistency not just across different data sets,
4 data files, but also across federal agencies.
5 I think that it's still early enough to
6 really do something to avoid getting a very
7 fragmented way of doing this. So I think these are
8 very important things that are being discussed
9 today. I don't know if that's going to stimulate
10 any more comments.
11 Gordon?
12 MR. KAUFMAN: Well, I can't resist the
13 comment. In Conan Doyle's "The Hound of
14 Baskervilles," Watson remarks that the hound did not
15 bay that night, and Holmes replies to the effect,
16 "Exactly, my dear Watson. That's what's really
17 important."
18 On this issue of data quality, a key
19 dimension of data quality would seem to be to me
20 identification of what the data does not provide and
21 that's necessary to meet the objectives as to why it
22 was gathered in the first place. And when you begin
NEAL R. GROSS
COURT REPORTERS AND TRANSCRIBERS
1323 RHODE ISLAND AVE., N.W.
(202) 234-4433 WASHINGTON, D.C. 20005-3701 (202) 234-4433
120
1 to think along those lines, away from numerical
2 measures of quality, you begin to think of scenarios
3 with which many of us are familiar.
4 If you go back to the oil and gas
5 upstream data, it's possible to provide an
6 aggregated projection time series for oil and gas at
7 various levels that is extremely timely. It's
8 extremely accurate. It's consistent. It's got a
9 lot of continuity to it. The problem with it is
10 when a Senator from the Hill calls and says, "I
11 would like in two months an answer to the following
12 policy question," the data isn't there to meet that
13 customer need.
14 What's the bottom line with respect to
15 measuring customer satisfaction? Some animals are
16 equals, but some are more equal than others; okay?
17 From EIA's point of view, as a legislative service
18 as well as servicing a wide domain of users, this
19 notion of accurate measurement of consumer
20 satisfaction for the purpose of providing the
21 quality of the data that it provides is going to be
22 an awfully elusive thing, and I would hope that
NEAL R. GROSS
COURT REPORTERS AND TRANSCRIBERS
1323 RHODE ISLAND AVE., N.W.
(202) 234-4433 WASHINGTON, D.C. 20005-3701 (202) 234-4433
121
1 doing quantitative performance measure wouldn't
2 detract from careful attention to that dimension to
3 data quality.
4 MR. MOUNT: So any other final comments?
5 Any comments from the public?
6 Dan?
7 MR. RELLES: Since I got hit three times
8 on short shrift of customer satisfaction, let me say
9 that I agree with everybody on that issue. I simply
10 was suggesting that it would be a byproduct of
11 minimizing or at least understanding the amount of
12 error that they're incurring, rather than a direct
13 goal that you can pursue, because I frankly don't
14 know how to measure it. I think I have a chance of
15 measuring the amount of error that a user is
16 incurring by the problems that are in the data.
17 MR. MOUNT: So I'd like to thank Nancy,
18 Renee and Dan for the presentations and the other
19 people who have contributed; thank EIA for being a
20 good host; and I think that we can call the meeting
21 to an end early.
22 I would like to ask the Committee, could
NEAL R. GROSS
COURT REPORTERS AND TRANSCRIBERS
1323 RHODE ISLAND AVE., N.W.
(202) 234-4433 WASHINGTON, D.C. 20005-3701 (202) 234-4433
122
1 we meet at 12:30 p.m. for lunch? That gives us a
2 half an hour to check out for those who want to do
3 it and meet at 12:30 rather than 12:45 p.m. Is that
4 okay?
5 (Whereupon, at 12:05 p.m., the meeting
6 was concluded.)
7
8
9
10
11
12
13
14
15
16
17
NEAL R. GROSS
COURT REPORTERS AND TRANSCRIBERS
1323 RHODE ISLAND AVE., N.W.
(202) 234-4433 WASHINGTON, D.C. 20005-3701 (202) 234-4433
