surveillance data to public
Anne Marie Parkinson
Community Health & Epidemiology
Department of Public Health Sciences
Faculty of Medicine
University of Toronto
The following people have contributed to this project:
Durham Region Health Department (DRHD)
Philippa Holowaty, Epidemiologist
Donna Reynolds, Associate Medical Officer of Health
Disseminating on-going surveillance data to public
The Rapid Risk Factor Surveillance System (RRFSS) is an on-going survey
of adult attitudes and health behaviours to enable public health professionals to
prioritize health issues, direct operational plans, measure effectiveness and
monitor progress at the local level.
A data dissemination model is described that enables integration of data into
public health program planning, implementation and evaluation. Qualitative
research methods were used to obtain information related to data understanding,
dissemination and utilization. Using existing frameworks, values, attitudes,
access, supports, skills and settings were examined. The dissemination model,
implementation process and evaluation plan were developed as part of the
RRFSS initiative at Durham Region Health Department.
The Rapid Risk Factor Surveillance System (RRFSS) is an on-going
survey of adult attitudes and health behaviours to enable public health
professionals to prioritize health issues, direct operational plans, measure
effectiveness and monitor progress at the local level. A need was identified to
create an environment where RRFSS data would be effectively used in
Evidence-based decision making is increasingly valued within public health.
However effective incorporation of evidence into practice is complex and remains
inconsistent [Coyle & Sokop (1990), Funk et al. (1991)]. Therefore, the RRFSS
which enables public health professionals to base programming decisions on
data, could not be more timely.
Literature directly related to effective incorporation of health surveillance data
into practice is lacking, however the broader research utilization literature has
described factors that were expected to be applicable to data utilization.
Research utilization is defined as the process by which relevant research
is critically examined and applied to validate current practice or to change
practice [Royle et al. (1996), McCurren (1995), Camiletti (1998)]. Many authors
have described the mediating factors and barriers to implementing research
findings into practice [Simpson (1996) , Funk et al. (1995a, 1995b), Stolk & Mayo
(1995), Camiletti (1998), Cabana (1999, 2000), Carter et al. (1995), Parahoo
(2000), Retsas, (2000)].
Strategies to increase research-based practice
There are many ways that have been identified that can increase an
organization’s potential for research-based practice. These include: 1) Creating
an environment where administrators show the value of evidence-based practice
by example and by expectation, by formalizing the expectation for others through
inclusion in staff responsibilities and performance appraisals [Funk et al. (1991,
1995a,1995b)]; 2) Enhancing administrative support and improving availability
and accessibility to research findings [Funk et al. (1991), Parahoo (2000)]; 3)
Supporting research utilization by providing mechanisms for colleague support
through group activities [Funk et al. (1991, 1995b), Parahoo (2000)]; 4) Active
dissemination of research findings through informing and educating health
professionals about content; 5) Increasing time available for reviewing and
implementing research findings [Funk et al. (1991), Parahoo (2000)]; 6)
Involving health professionals in the selection of research topics [Burns et al.
(1992), Anderson (1993), SIGN (1999)] and 7) Regular reminders relating directly
to practice activities [Cabana (1999), Anderson (1993), SIGN (1999)]. These
strategies relate to research utilization but may be applicable in promoting the
integration of data into public health program planning, implementation and
The barriers include a lack of positive attitude toward research, lack of
time to read research and implement new ideas, lack of availability of pertinent
research findings, resistance to change, limited access to journals, limited ability
to understand and interpret research reports, limited exposure to strategies of
how to use research, lack of administrative support and concerns about
reallocation of staff time [Funk et al. (1995a, 1995b), Stolk and Mayo (1995)
Camiletti (1998), Cabana (1999, 2000), Carter et al. (1995), Retsas (2000),
Parahoo (2000) ].
It is important to acknowledge however that factors affecting research
utilization may not all be directly relevant for data utilization, although the broader
categories identified in the literature, that affect research utilization in practice,
were expected to be transferable to the use of data for similar purposes. The
barriers to research utilization have been classified in the literature into various
categories. For the purpose of this paper, health data utilization is examined
using a combination of existing research utilization frameworks in terms of values
and attitude, access and support, skill, and setting as described by Camiletti et
al. (1998) and Champion and Leach (1986, 1989).
Rapid Risk Factor Surveillance System
The RRFSS is an on-going survey of adult attitudes and health behaviours
that are directly related to programming efforts of public health professionals.
RRFSS enables public health professionals to prioritize health issues, direct
operational plans, measure effectiveness and monitor progress at the local level.
The surveillance system is a telephone survey that provides data suited for
detecting temporal changes, seasonal variation, the effects of a program
overtime and data that will allow for before-and-after comparisons when a
program is initiated or when other potential impacts occur [MacNeill (1997)].
The questionnaire has three components: 1) core module, a core of
questions asked by all participating health units in Ontario; 2) optional modules,
sets of questions chosen by individual health units to meet specific needs and 3)
health unit specific questions.
As part of the RRFSS initiative, the Epidemiology & Evaluation Unit at
Durham Region Health Department, Whitby Ontario identified the need for a
system of data dissemination that would assist program managers and staff to
begin regularly integrating health data from RRFSS into the program planning,
implementation and evaluation cycle. This report is aimed at exploring the
research utilization culture of public health professionals and describing a data
dissemination model that promotes utilization of data in public health practice.
Qualitative research methods were used to explore program managers
and staff’s attitude toward using on-going health risk factor surveillance data
(RRFSS) in relation to program planning and evaluation, their perceived access
to relevant health data and support for the implementation of findings. The stages
of the design are shown in Figure 1.
Three focus groups were conducted, one with management and two with
program staff. The management focus group participants were purposively
selected as representatives from different Health Department program areas.
These same managers also formed an Advisory Working Group to oversee the
data dissemination and utilization model development.
For the staff focus groups, the health department was stratified by
divisions and programs: Environmental Health, Health Promotion-Disease
Prevention, Parenting, Injury Prevention, Clinical Services, and Comm unicable
Disease Control and Prevention Program. Dental Health, Infant Development,
and Emergency Medical Services were excluded due to availability.
Administrative staff were also excluded. Focus group participants were then
randomly selected without replacement from each program, proportionate to the
number of FTE’s in that area.
The selection criteria for staff focus group participants was defined as
individuals who: i) Occupied a full-time position at Durham Public Health
Department in a specific program; ii) Were available for participation on the focus
group date; iii) Had management approval, if required.
Six managers and fourteen staff were selected for the focus groups; 5 managers
and 11 staff were present. The job titles of focus group participants are presented
in Table 1. The focus group questions used are illustrated in Table 2.
The qualitative data obtained in the focus groups was analyzed by hand
using a method adapted from the Public Health Research, Education and
Development (PHRED) E valuation Tool Kit [Porteous et al. 1997]. All focus
groups were tape-recorded and transcribed. The completed transcripts were read
independently by researchers 1 and 2. The focus group results were analyzed in
the context of research utilization frameworks from the literature [Camiletti et al.
(1998),Champion & Leach (1986,1989)]. Highlighters were used - a different
colour for each of the following four categories that guided the analysis: 1) Value
and attitude 2) access and support 3) skills and 4) setting. Value & Attitude
refers to the perceived level of commitment to data utilization at the health
department and individual’s feelings about incorporating data into program
planning. Access & Support refers to perceptions of availability of informa tion
and perceptions of support in understanding and effectively using data. Skill
refers to the ability to find, read, analyze and apply research to practice and
Setting refers to the workplace environment, including time, access to technology
and staffing. The highlighters were used to mark the comments that dealt with
each category. Using the above categories, opinions, ideas, or feelings
expressed in the focus groups were tallied and clustered into themes.
The Advisory Working Group of managers along with the researchers
used the focus group results for the development of the data dissemination and
utilization model. The model was developed using a logic model format adapted
from the PHRED Evaluation Tool Kit [Porteous et al. (1997)].
The long-term outcome objective of the model was to ensure the use of
community health status information in assessing local health needs and in the
planning and evaluation of programs as set by the Ontario Ministry of Health &
Long-term Care Mandatory Requirements and Standards for program planning
and evaluation. The intermediate outcome objective was the foundation for this
project, that RRFSS data would be used for program planning, implementation
and/or evaluation, that is, evidence -based action at the program level. The
themes identified in the focus groups were used to determine the activities
required in the model to increase utilization of RRFSS data in the programming
cycle and short-term outcome objectives formed the link between the activities or
process objectives and the intermediate objectives of data utilization.
Public Health Unit
5 Managers 11 Program Staff
1 focus group 2 focus groups
Transcription of tapes
Researchers 1 & 2
Figure 1: Research design
Table 1: Job title of participants
Program Staff Managers
Focus group 1 Focus group 1
Senior Public Health Inspector
Public Health Nurse-Parenting 3 Managers-Public Health
Public Health Nurse-Reproductive Nursing & Nutrition
Public Health Nurse-Injury Program Coordinator- Public
Prevention Health Nurse
Public Health Nurse-Clinical
Services Manager – Environmental Health
Focus group 2 Division
Public Health Inspector-General
Public Health Nurse-Health
Promotion Disease Prevention
Public Health Nurse-Health
Promotion Disease Prevention
Public Health Nurse –Injury
Community Health Nurse–
Communicable Disease Control and
Public Health Nurse – Parenting
Table 2: Focus Group Questions
How do we use health data for planning and evaluation, and how can the
Epidemiology & Evaluation Unit make this data most available to you?
1. What do you think about when I say data?
2. What do you think the Health Department’s commitment is to using health
data for planning?
3. How do you feel about using data, yourself, in your program planning?
4. How do you get health data at the moment?
5. If this information were available from several places and in different forms,
how would you most prefer to receive it?
6. Once you have the data, do you feel comfortable using it?
7. Would you like there to be regular discussions of RRFSS data and its
implications for programming?
8. How frequently should the surveillance data go out to you?
The themes identified in the focus groups show that most participants
valued data as part of the program planning, implementation and evaluation
cycle, however, barriers such as lack of time, lack of access to computers, and a
skill deficit in data interpretation and applyi ng findings in practice compromise
data utilization. The key findings from the focus groups that were applicable to
disseminating RRFSS data are presented (Table 4).
There were no comments in the focus group discussions that fell outside
the four predetermined categories, however this may have been due to the
investigators previous knowledge of the research utilization literature preceding
focus group question development. The focus group results were independently
analysed by researchers 1 and 2 and minimal to no deviation in agreement was
identified between the two analyses.
The data dissemination and utilization model summarizes the key
elements of data dissemination, shows which activities are expected to lead to
which outcomes and identifies the critical questions for evaluation [Porteous et
al. 1997]. The evaluation component of the dissemination process is embedded
in the intermediate and short-term outcome objectives of the logic model. The
intermediate outcome objective measures utilization of the RRFSS data and the
short-term outcome objectives measure the outcomes from processes of
dissemination that should lead to utilization. The supporting documentation for
the model includes the tools necessary for evaluation of the RRFSS
dissemination and utilization model.
The Question Feedback Form is the evaluation tool to measure utilization
of the RRFSS data at the programming level. The question feedback form
collects information on how the data has been used (intermediate outcome
objective) and what actions (ie. program changes) have been made as a result of
the data. The feedback form is also used to determine which questions are
useful and should remain on the questionnaire.
The RRFSS Internal Evaluation Survey is the evaluation tool to measure
the dissemination process. The survey measures the short-term outcome
objectives of the data dissemination and utilization model. The data
dissemination and utilization model with supporting documentation is presented
in Appendix A.
Table 3: Focus Group Results
Value & Attitude Access & Support Skills Setting
There is a positive attitude Support from the Staff competence with There is a lack of computers
to using data and the value Epidemiology & Evaluation computers and looking for and intra/internet access.
of data is increasing. Unit (E&E) is needed, to resources varies vastly.
answer questions and clarify Lack of time.
There is a desire to be interpretation. Comfort level with
evidence-based. computers and the RRFSS results need to be
There is confusion on how intra/internet is also an discussed at program
Data can ensure we are on to access E&E. issue. meetings; it should be a
target with program plans, standing item on the
otherwise changes may The intranet is easy to Skills vary in data agenda, regular discussion
need to be made to plans. access but there must be a interpretation and how to is needed.
hard copy as well. apply it to programming in
Results in relation to the
objectives would be helpful.
A text explanation is also
The research utilization frameworks used to investigate data utilization
were very useful to our investigation. The themes that we identified for data
utilization within each category were similar to those described in the research
Most of the public health professionals in this study valued data as part of
the program planning, implementation and evaluation cycle and expressed a
desire to be more evidence-based. However, barriers such as lack of time, lack
of access to research findings and a skill deficit in data interpretation and
applying findings in practice compromise data utilization. Similar themes are
reported by Rodgers (1994) and May (1998) who used semi-structured
interviews to explore the research culture of practitioners. A number of surveys
have attempted to determine the factors which affect research utilization among
practitioners and also reported similar results [Hatcher (1997), Camiletti (1998),
Funk et al. (1991) (1995a) (1995b), Retsas (2000), Parahoo (2000) ].
This suggests that for Durham Region Health Department, the barriers identified
in the research utilization literature are interchangeable with the barriers
identified in the utilization of data in program planning , implementation and
evaluation. Therefore, the strategies that have been identified in the literature to
increase an organization’s potential for research-based practice may also be
useful in integrating data into public health programming.
Based upon the findings of the focus groups and the literature, the
reasons why public heath professionals do or do not use health data for program
planning, implementation and /or evaluation are complex and involve a web of
individual and organizational factors. Therefore, an effective intervention to
increase data utilization in practice must address both the individual and
The data dissemination and utilization model presented here addresses a
number of individual and organizational barriers that were identified in the focus
groups and that are consistent with the research utilization literature. The model:
1) acknowledges that implementing evidence into practice is an organizational
rather than individual issue; 2) acknowledges the need for collaboration between
those that use the data and those that generate it; 3) recognizes the skills
needed to complete the cycle of combining evidence and practice; 4) recognizes
the importance of disseminating data in a variety of ways; 5) acknowledges the
need for access to findings; 6) recognizes the need for training and education in
research i.e./ interpretation and analyses and 7) recognizes the need for regular
Public health professionals have a diverse knowledge, interest and ability
in integrating health data into practice. We have described the beginning
implementation of a strategic plan to promote data utilization in a public health
unit in support of evidence-based practice. The data dissemination and utilization
model presents the key elements of dissemination and shows which activities are
expected to lead to utilization.
This type of research is not designed to be generalizable, however the
study has provided a glance at the complex nature of research utilization among
public health professionals and has lead to a data dissemination and utilization
model that has elements that would be applicable to other RRFSS participating
health units across Ontario. It can serve as a guide for other health units
conducting their evaluation of RRFSS utilization for planning, implementing, and
evaluating public health interventions and programs [Klaucke et al.,1988]. The
model also has the potential to provide valuable information in measuring
whether the surveillance system is meeting the system’s objectives by serving a
useful public health function [Klaucke et al., 1988].
Anderson, G. (1993). Implementing practice guidelines. Canadian Medical
Association Journal. Vol. 148, no. 5, pp.753-55.
Burns, L.R., Denton, M, Goldfein S. et al. (1992). The use of continuous quality
improvement methods in the development and dissemination of medical
practice guidelines. QRB, pp. 434-39. December.
Camiletti Y.A., Huffman, M.C. (1998). Research Utilization: evaluation of
initiatives in a public health nursing division. Canadian Journal of Nursing
Adminstration. Vol. 11, no. 2, pp. 59-77.
Carter, A.O., Renaldo, N., Hodge M.J. et al. (1995) Report on activities and
attitude of organizations active in the clinical practice guideline field.
Canadian Medical Association Journal, Vol. 153, no. 7. pp.901-06.
Champion , V. & Leach, A. (1986). The relationship of support, availability, and
attitude to research utilization. Journal of Nursing Administration, Vol. 16,
no. 3, pp. 19-37.
Champion , V. & Leach, A. (1989). Variables related to research utilization in
nursing: an empirical investigation. Journal of Advanced Nursing, Vol. 14,
Coyle, L. & Sokop, A. (1990). Innovative adoption behaviour among nurses.
Nursing Research, Vol. 39, pp. 176-180.
Funk, S., Champagne, M., Wiese, R. & Tornquist, E. (1991). Barriers to using
research findings in practice: a clinician's perspective. Applied Nursing
Research, Vol. 4, no. 3, pp. 90-95.
Funk, S., Champagne, M., Tornquist, E., & Wiese, R. (1995a). Clinical Methods:
Administrator's views n barriers to research utilization. Applied Nursing
Research, Vol. 8, no. 1, pp. 44-49.
Funk, S., Tornquist, E. and Champagne, M. (1995b). Barriers and facilitators of
research utilization: an integrative review. Nursing Clinic of North
America, Vol. 30, no. 3, pp.395-407.
Hatcher, S. & Tranmer, J. (1997). A survey of variables related to research
utilization in nursing practice in the acute care setting. Canadian Journal
of Nursing Administration, pp. 31-53, September-October.
Klaucke D.N., Buehler, J.W., Thacker, S.B. et al. (1988). Guidelines for
evaluating surveillance systems. MMWR, 37(S-5), pp. 1-18.
May, A.L., Mulhall, A., Alexander, C. (1998). Bridging the research-practice gap:
exploring the research culture of pracitioners and managers. Journal of
Advanced Nursing, Vol. 28, no. 2, pp. 428-37.
MacNeill, I.B., Umphrey, G.J. (1997). Concept paper on Canadian health
surveys and risk factor surveillance. LCDC, Health Canada.
McCurren, C. (1995). Research utilization: Meeting the challenge. Geriatric
Nursing, Vol. 16, no. 3, pp. 132-135.
Ontario Ministry of Health & Long-term Care (1997). Mandatory Health
Programs and Services Guidelines. Minister of Health, Health Promotion
and Protection Act, Section 7, R.S.O. 1990, c. H.7.
Parahoo, K. (2000). Barriers to, and facilitators of, research utilization among
nurses in Northern Ireland. Journal of Advanced Nursing, Vol. 31, no. 1,
Pepler, C.J. (1995). Using research to improve nursing practice. In L.A. Talbot
(Ed.). Principles and practice of nursing research, pp.535-567. Toronto.
Porteous, N.L., Seldrick, B.J., and Stewart, P.J. (1997). Program Evaluation Tool
Kit-a blueprint for public health management. Public Health Research,
Education and Development Program. Ottawa-Carleton Health
Retsas, A. (2000). Barriers to using research evidence in nursing practice.
Journal of Advanced Nursing, Vol. 31, no. 3, pp. 599-606.
Rodgers, S. (1994). An exploratory study of research utilisation by nurses in
general medical and surgical wards. Journal of Advanced Nursing, Vol.
20, pp. 904-11.
Royle, J. A., Blythe, J., Ingram, C., DiCenso, A. Bhatnager, N., & Potvin, C.
(1996). The research utilization process: The use of guided imagery to
reduce anxiety. Canadian Oncology Nursing Journal, Vol. 6, no.1, pp.20-
Scottish Intercollegiate Guidelines Network. (1999). An introduction to SIGN
methodology for the development of evidence-based clinical guidelines.
Publication number 39 (pp.25-27). Edinburgh.
Simpson, B. (1996). Evidence-based nursing practice: the state of the art. The
Canadian Nurse. pp. 22-25. November.
Stolk, B. & Mayo, E. (1995). Barriers to research utilization perceived by staff
public health nurses. Unpublished master's thesis, University of Western