Get documents about "Executive Summary VersionTime Team Members
Document Sample
Executive Summary: Injury Surveillance Pilot Project
Evaluation Report
Version/Time: June 2002
Team Members: Parminder Raina, Hassan Soubhi, Kate
Turcotte, Mariana Brussoni, Patti Janssen, and Alan Hotte.
Prepared by:
Abstract
The purpose of this evaluation was to assess the implementation of the Injury
Surveillance Pilot Project (ISPP) in its mission to describe the burden of injury
among persons presenting at selected Emergency Departments (ED) in British
Columbia. Overall, the ISPP with the Emergency Department Injury Surveillance
System (EDISS) at its core was able to collect data that were for the most part
error-free with a capture rate of over 82 percent. Recommendations include the
need for continued funding to allow for data collection to continue, continued
training of Health Records personnel to improve the coding of selected data
elements, and continued automation of the database. Obstacles encountered
include an overall provincial shortage of Health Records staff, the adoption of
ICD-10 injury codes, and various job actions and contract negotiations with the
provincial government. Overall, the evaluation provides a positive outlook for the
ISPP.
Executive summary: Injury Surveillance Pilot Project Evaluation Report
Version/date: June 2002
Authors: P. Raina, H. Soubhi, K. Turcotte, M. Brussoni, P. Janssen & A. Hotte
1 of 8
Introduction
The Injury Surveillance Pilot Project (ISPP) worked to implement the Emergency
Department Injury Surveillance System (EDISS) in ten hospitals within three
regions of British Columbia (BC) between December 2000 and March 2002. The
three regions were the South Fraser Health Region (SFHR), the North West
Health Services Society (NWHSS), and the Thompson Health Region (THR).1
An evaluation of the ISPP was undertaken from January to March 2002,
approximately one year into the project.
The purpose of the evaluation was to assess the implementation of the ISPP in
its mission to describe the burden of injury among persons presenting at
Emergency Departments (EDs) in selected hospitals in BC. Specifically, the
evaluation assessed the following project goals:
• Data collection implementation, including provision of training and support to
Emergency Department Admitting, Nursing and Physician staff and Health
Records personnel to chart, abstract and code a minimum data set for ED
patients presenting for the first treatment of an injury;
• Establishment and management of electronic injury data storage;
• Regular analysis and interpretation of data;
• Dissemination of information; and
• Use of information for injury prevention.
The stakeholders in the ISPP evaluation are the groups or individuals who are
interested in the results or who will be using the evaluation information. Internal
and external stakeholders are listed below:
Internal Stakeholders External Stakeholders
§ Hospital Emergency § Ministry of Health Planning – Deputy Medical Officer of Health
Departments § Ministry of Health Planning – Office of Injury Prevention
§ Hospital Health Records § Ministry of Health Planning – Regional Interface for Health Data
Departments § Health Region – Medical Health Officers
§ Hospital IS Departments § Health Region – Injury Prevention Coordinators
§ BC Injury Research and § Hospital CEOs
Prevention Unit (BCIRPU)
§ Injury Prevention organizations and advocates
§ Health Canada
1
Restructuring of these Health Regions has led to the creation of the Fraser, Northern and
Interior Health Authorities. This restructuring is not reflected in this report.
Executive summary: Injury Surveillance Pilot Project Evaluation Report
Version/date: June 2002
Authors: P. Raina, H. Soubhi, K. Turcotte, M. Brussoni, P. Janssen & A. Hotte
2 of 8
The ISPP evaluation considered 26 questions addressing:
• Data Collection -implementation costs, staffing, charting completeness,
Minimum Data Set versus NACRS, data quality, sensitivity & specificity,
continuation of data collection, and plans to seek internal & external funding.
• Data Integration - scrambling procedure, system availability, and future data
linkage.
• Data Analysis & Interpretation - database manual.
• Dissemination - schedule for data reporting, presentations, availability of
aggregated data, production of data reports, data report feedback, engaging
new regions, external interest in EDISS, EDISS newsletters, use of data in
injury prevention, use of data in hospital resource allocation, new research
proposals and local champions of EDISS.
Support for the ISPP
This evaluation of the ISPP has demonstrated:
• Most of the participating hospitals and regions desire to continue the data
collection for EDISS.
• The limiting factor of continued data collection beyond funding is the
availability of Health Records staff.
• EDISS newsletters have been well received and are helping to raise
awareness around injury prevention and the role of data collection.
• There is the potential to expand EDISS within BC, as indicated by the
interest reported from other hospitals and regions.
• Further work is required regarding the changing political climate and
structure of the health systems to determine where EDISS might fall within
the management of the new Health Authorities.
• The sustainability and future expansion of EDISS beyond the ISPP will
primarily be determined by the availability of resources dedicated to injury
surveillance activities.
Evaluation Summary
Data Collection
• Start-up and annual operating costs have been determined at the hospital
level based on size and resources.
• Sensitivity of the system was determined to be 82.6 percent (77.8% - 87.3%).
• Inappropriate inclusion of non-injuries in the database was negligible with a
system specificity of approximately 99 percent, a result comparable to other
studies.
• Reliability of data abstraction and coding was reasonably high across data
elements, with high similarities between the selected sites.
• All sites combined, reliability decreased with the level of detail required for
proper coding of a given data element. Alternatively, reliability decreased for
Executive summary: Injury Surveillance Pilot Project Evaluation Report
Version/date: June 2002
Authors: P. Raina, H. Soubhi, K. Turcotte, M. Brussoni, P. Janssen & A. Hotte
3 of 8
the injury-event data elements which the ED staff may not have been
routinely documenting prior to the ISPP.
• Completeness was found to be between 96 and 100 percent for the socio-
demographic variables.
• Diagnosis and Cause of Injury reached a level of completeness2 of 89 and 92
percent respectively.
• The lowest level of completion was for Activity When Injured, where only 48
percent of the audited charts had complete information.
• Analyses of completion rates pre- and post-training periods showed slight
differences that did not reach statistical significance.
• None of these differences reached the 20 percent threshold assumed,
therefore a larger sample size would be required to provide adequate power
for assessing the statistical significance of the differences found.
Improvements in the level of completeness of Main Diagnosis and Cause of
Injury would have to be addressed relative to the ICD coding requirements.
There is some indication that completeness of charting may be easier to achieve
at smaller sites, possibly influenced by fewer injury cases seen over the same
period of time as compared to larger sites and therefore the perception of more
time to document injury-event information. Smaller sites also have fewer ED staff
members, therefore it may be easier to achieve commitment from the ED staff to
chart more injury-event details. Finally, budgetary pressures may be felt
differently at the smaller sites.
Two of the three sites participating in the chart audit adopted ICD-10 for the
2001/2002 fiscal year, therefore reliability could have been affected by the
learning curve for this new coding system of Diagnosis and Cause of Injury.
Place of Occurrence and Activity When Injured are injury-event data elements
that ED staff are not as likely to chart in great detail. Furthermore, the coding
scheme for Activity When Injured is particularly stringent, and requires a high
level of detail to code appropriately.
Results indicate the need for improving the completeness of Activity When
Injured. Difficulties with this data element stem from the level of detail required
for the coding system. Activity When Injured requires not only the literal activity,
such as riding a bicycle, but also an explanation of the motivation for that activity,
for example (e.g. travelling to work).
Injury information collected by the EDISS Minimum Data Set provides a reliable
assessment of the injury events seen in the participating EDs. Increased
attention must be paid to the data elements that describe Place of Occurrence
2
Based on the BCIRPU perspective, referring to the level of detail required to appropriately code
a data element according to the level of aggregation used to report the EDISS data.
Executive summary: Injury Surveillance Pilot Project Evaluation Report
Version/date: June 2002
Authors: P. Raina, H. Soubhi, K. Turcotte, M. Brussoni, P. Janssen & A. Hotte
4 of 8
and Activity When Injured. Potential improvements to sensitivity may be expected
with a review of the protocol for identifying injury ED charts and targeted training
on the inclusion and exclusion criteria used in EDISS.
This evaluation has demonstrated that the data collected and submitted by the
participating hospitals are error-free, and that the system has achieved good
sensitivity and very high specificity. Charting by ED staff is complete for the
socio-demographic data elements, diagnoses and causes of injury, while needing
more attention for the injury-specific data elements, such as Place of Occurrence
and particularly Activity When Injured. Abstraction and coding by the Health
Records staff is reliable, again with future attention directed at Place of
Occurrence and Activity When Injured.
Data Integration
• Privacy is ensured at the hospital level by scrambling Chart Number and
PHN.
• Only the four hospitals of the SFHR were able to submit data between
September and October 2001 for the first round of Hospital Data Reports. All
hospitals successfully submitted data for the second round of reports.
• Potential to link EDISS data with other databases for research has been
explored with the BC Linked Database.
Data Analysis & Interpretation
• EDISS Database Manual details aspects of data management and analysis.
Dissemination
• Access to aggregated data received a positive response from the Health
Data Warehouse, Ministry of Health Planning.
• Preliminary Data Reports have been favourably received.
Obstacles Encountered
The implementation of the ISPP involving ten Hospitals within three regions of
BC has encountered several obstacles over the past 16 months.
Specifically these include:
§ Provincial shortage of Health Records staff for external hiring;
§ Mandated adoption of ICD-10 for coding inpatient charts at all sites;
§ Optional adoption of ICD-10 for coding ambulatory charts at nine of the ten
sites;
§ Delay in receiving hospital software updated for ICD-10;
§ Job actions involving the BC Nurses' Union, the Hospital Employees' Union
and the BC Health Sciences Association;
§ Contract negotiations between the three unions and the provincial
government.
Executive summary: Injury Surveillance Pilot Project Evaluation Report
Version/date: June 2002
Authors: P. Raina, H. Soubhi, K. Turcotte, M. Brussoni, P. Janssen & A. Hotte
5 of 8
The obstacle with the greatest impact has been the adoption of ICD-10 and the
resulting delay in receiving the new hospital software. This created a backlog of
not only the EDISS data abstraction and coding, but also of all the hospital charts
processed through the Health Records departments.
The provincial shortage of Health Records staff has been a chronic challenge
throughout the implementation of the ISPP, and has also played a role in
delaying the abstraction and coding of injury ED charts for EDISS at some of the
sites.
Despite all of these obstacles, data are coming out of the system. Furthermore,
many of these obstacles are one-time events, and their effect on the ISPP will be
diminished over time. The timeliness for data abstraction, coding and submission
will improve as these challenges are addressed.
Limitations of the Evaluation
The one all-encompassing limitation of the current evaluation of the ISPP and
EDISS is the timing of the evaluation in relation to the progress attained by the
participating hospitals. Many obstacles were encountered over the past year
resulting in the delay of data abstraction, coding and submission to BCIRPU.
Many of the evaluation questions could not be fully addressed as hospitals are
just now finalizing their internal procedures for data submission and the data has
yet to be widely disseminated.
Low response rates to both the Key Informant Survey and the Data Report
Evaluation and Needs Assessment are considered to be a result of the recent
reorganization of BC Health Regions into new Health Authorities, and the related
budget cuts imposed by the provincial government.
Other factors influencing the response rate of these tools include the workload
related to the fiscal year-end activities and vacation times spanning the course of
the Key Informant Survey and the Data Report Evaluation and Needs
Assessment.
Finally, an identified limitation of this evaluation is that chart audits were based
on the information documented on the ED forms only. This limitation may have
lead to low estimates of charting completeness, reliability and the sensitivity of
EDISS.
Executive summary: Injury Surveillance Pilot Project Evaluation Report
Version/date: June 2002
Authors: P. Raina, H. Soubhi, K. Turcotte, M. Brussoni, P. Janssen & A. Hotte
6 of 8
Recommendations for the ISPP
In response to the potential uses of the EDISS data by hospitals and regions, it is
recommended that each of the three participating regions create committees to
address:
§ The use of these data and explore the role of community partners in injury
prevention.
§ The need for continued funding if the data collection for EDISS is to be
sustained beyond the scope of the ISPP.
In response to the completeness of charting by the ED staff, it is recommended
that the structure and content of the training sessions be reviewed with a focus
on the data elements of Place of Occurrence and Activity When Injured.
In response to the reliability of data submitted to BCIRPU, further investigation
may reveal that reliability is directly influenced by the completeness of charting,
particularly for the Place of Occurrence and Activity When Injured data elements.
Training sessions for both ED and Health Records staff may affect the reliability
of the data.
In response to the sensitivity of EDISS, strategies to reduce the number of real
injury cases missed by the Health Records staff should be explored. The first
step is to review the False Negative cases as identified in the chart audit to
ensure that these cases are indeed missed injuries. Further exploration of the
patient charts beyond the ED form may provide reasons for excluding these
cases from EDISS, thus revealing a higher sensitivity than is reported here. The
second step is to explore the different systems currently in use at the different
sites for identifying injury cases, along with a comparison based on the size of
EDs. The third step is to address ongoing training of Health Records staff in the
identification of injuries.
In response to the dissemination of data reports, an updated Dissemination Plan
is warranted for Period 3 of the ISPP, with a focus on specialized data reports
and systematic evaluations of these reports. The list of recipients should be
reviewed, along with their roles in the new political climate and creation of Health
Authorities. Furthermore the means of distribution (electronic vs. hard copy)
should be reviewed.
In response to the EDISS Newsletter, content for upcoming newsletters for
Period 3 of the ISPP should be drafted with themes identified, taking into account
suggestions from this evaluation. The distribution list should be reviewed and
expanded, as well as reviewing the means of distribution.
Executive summary: Injury Surveillance Pilot Project Evaluation Report
Version/date: June 2002
Authors: P. Raina, H. Soubhi, K. Turcotte, M. Brussoni, P. Janssen & A. Hotte
7 of 8
Furthermore, it is recommended that resources be identified to conduct a second
stage surveillance of a particular injury trend as identified by the first year of
EDISS data with a six month component for data collection and a three month
component for data entry and analysis.
2002-2003 Period 3 Evaluation
The focus of the upcoming Period 3 Evaluation is as follows:
1. Data collection
§ Compile qualitative data through semi-structured focus groups of
Emergency Department staff of selected participating hospitals concerning
barriers regarding charting completeness of the Minimum Data Set.
§ Continue to monitor data collection and data quality through assessing
levels of charting completeness, reliability of Health Records staff, and
sensitivity and specificity of the system of selected Health Records
Departments.
2. Data Integration
§ Monitor ability of the system to report in a timely fashion.
3. Data Analysis
§ Report on the uses of EDISS data in injury prevention.
4. Diffusion, Dissemination, and Implementation
§ Report on the effectiveness of dissemination framework and BCIRPU’s
support of participating regions in the use and interpretation of EDISS
data.
§ Report on the effectiveness of dissemination products in enhancing the
awareness of the identified target audiences concerning the project.
§ Outline perceptions of recipients on the content and format of data reports
produced by BCIRPU.
§ Evaluate current uses of EDISS data in planning or evaluating injury
prevention.
§ Evaluate the ability of EDISS, participating hospitals, and Health
Authorities to establish community partnerships and linkages.
§ Evaluate longer-term dissemination and uptake of information coming out
of EDISS at the project level.
Executive summary: Injury Surveillance Pilot Project Evaluation Report
Version/date: June 2002
Authors: P. Raina, H. Soubhi, K. Turcotte, M. Brussoni, P. Janssen & A. Hotte
8 of 8
Related docs
