Sustainability Plan for djhelp by ift26461

VIEWS: 6 PAGES: 56

									HYSTEMS
 EALTH
 S AB
 L                                       University of Wisconsin-Madison
                                          UW Madison School of Nursing




                Internet Access to Digital Libraries
                    Library of Medicine Grant LM 7781-01


         Final Report
         Prototype Evaluation
         Business Sustainability Plan
              Patricia Flatley Brennan, R.N., Ph.D., Principal Investigator
              David J Grindrod, Project Coordinator
              Debra Gatzke, Executive Director, DJHCP



                                                                      August 10, 2004




The Dodge-Jefferson Healthier Communities Partnership
                                                                                                          i


Acknowledgements

         In addition to the principal authors and personnel listed on the cover of this document, many
         other individuals contributed a great deal of work to this project. We wish to acknowledge the
         hard work and significant contributions of:

         Amy Lutzke (Fort Atkinson Dwight Foster Public Library), Dan McCrea (United Way of
         Jefferson and North Walworth Counties), Joy Shong (Watertown Memorial Hospital), and Pat
         Trapp (Beaver Dam Hospital) — members of the DJHCP Technology Planning Committee

         Natalie Norcross, Assistant Director, UW-Madison Middleton Health Sciences Library

         Stephen Douglas, Graduate Assistant, UW-Madison Health Systems Lab

         Marti J. Sopher, Outreach Specialist, UW-Madison Health Systems Lab.


         For all of these and the many other members of the Dodge Jefferson Healthier Community
         Partnership, and of the communities of Dodge and Jefferson counties involved in this project,
         we are grateful.
                                                                         H
                                                                                          1
                                                                              EALTH
                                                                          S    YSTEMS
                                                                          L     AB
                                  Final Report
                               DJHCP/UW Madison
                          Digital Libraries Project
                                 June 24, 2004

I. Summary

The University of Wisconsin-Madison Health Systems Laboratory (UWHSL) assisted the
Dodge Jefferson Healthier Communities Partnership (DJHCP) to develop and evaluate
djhelp.info, a prototype web-based health and human services information and referral
service. With funding from a grant from the National Library of Medicine (LM007781-
01), DJHCP and UWHSL jointly developed djhelp.info to provide a locally-controlled
Web portal to regional and national information selected to meet the identified health
information needs of residents of Dodge and Jefferson counties. The Web portal,
djhelp.info, implemented through an arrangement with the United Way of Dane County
(for database development) and NorthLight’s Resource House, became active in March,
2004. djhelp.info provides access to information about health concerns and local health
resources through two navigation strategies: a keyword search and a guided search
approach.

Design Process Overview:

The project was originally to run from September 30, 2002 to September 29, 2003. At
the end of that period UWHSL and DJHCP requested, and NLM granted a no-cost
extension to September 29, 2004. The goal of the Community Digital Health Library
project was to design, build, test and evaluate a prototype community-based digital
consumer health library that would provide federal, statewide, and local health
information to community citizens directly, as well as to provide health information
evaluation services to health institutions in the community.

Steps in this process included:

       1. Assessment of community needs for various types of information, capacity for
              seeking, receiving, processing information in a variety of formats
       2. Assessment of existing resources in community
       3. Prioritizing information needs, delivery format requirements
       4. Design of the prototype resource model
       5. Building links to community resources and organizations needed to validate
              and sustain the model.
       6. Resolving conflicts in design needs driven by various community interests,
              organizations, and resources
       7. Implementing the model
       8. Testing the model
       9. Developing plan for refining, sustaining, and expanding to full service
                                                                                           2


During the course of assessing and prioritizing community needs and objectives, the
Technology Planning Committee, delegated by the DJHCP board, identified a strong
need for information on locally-available service resources. At the same time, the
Committee discovered an overlap of interests with a statewide project to develop a
telephone- and internet-based ―2-1-1‖ Information and Referral service. The Planning
Committee elected to incorporate the development of a 2-1-1 local referral service over
the two-county area into their vision of a digital community health information library,
creating a unique hybrid resource that would provide access both to trusted national and
state-level health information resources, and to local health care and human service
providers.

After reviewing a number of different design approaches to the hybrid community health
information library, the committee ultimately decided to use a commercially-available
web application front end, Resource House, running over a locally-developed database of
healthcare and human service providers, as the base of their model. Resource House is
designed to provide public and professional access to local and regional information and
referral databases that are based on the Info-Line taxonomy of the national Alliance of
Information and Referral Systems (AIRS). The design strategy was to enhance the local
information and referral website model with links (on the website as well as built into the
database) to lead users to evaluated and trusted external sites for general health
information. Health Systems Lab staff, in consultation with DJHCP Planning Committee
members, developed a functional map of the website design, shown in Appendix A at the
end of this report.

A prototype of the website design was completed in March of 2004, subjected to bench
performance testing at the University of Wisconsin-Madison Health Systems
Laboratories, and offered to community members in Dodge and Jefferson counties for
testing and evaluation. Community members recruited to try out the site included
members of the lay public, information and referral professionals such as public
librarians and social service provider agencies, and health care providers.

The evaluation results indicated that the prototype, dubbed djhelp.info, was generally
viewed as acceptable by users for usability and usefulness. There was wide anticipation
of and appreciation for the value of a product that would provide trustable health
information as well as assistance in finding local services. However, users did not find
navigating the functions of the website to be intuitive, and many were frustrated with the
incomplete state of the local database.

Section II of this report summarizes the results of the testing and community evaluation
of the website prototype, in order to draw conclusions about directions to be taken in
order to develop djhelp.info into a fully-functioning digital community health library.
(The full evaluation report is attached as Appendix B to this document.)

Section III presents a Business Plan for the organizational development and maintenance,
community support, and sustainable financial support for development and operations of
the djhelp.info website and database.
                                                                                          3




II. Evaluation.

The University of Wisconsin-Madison Health Systems Laboratory (UWHSL) assisted the
Dodge Jefferson Healthier Communities Partnership (DJHCP) to develop and evaluate
djhelp.info, a prototype web-based health and human services information and referral
service. With funding from a grant from the National Library of Medicine (LM007781-
01), DJHCP and UWHSL jointly developed djhelp.info to provide a locally-controlled
Web portal to regional and national information selected to the identified health
information needs of residents of Dodge and Jefferson counties. The Web portal,
djhelp.info, implemented through an arrangement with the United Way of Dane County
(for database development) and NorthLight’s Resource House, became active in March,
2004. djhelp.info provides access to information about health concerns and local health
resources through two navigation strategies: a keyword search and a guided search
approach. This report summarizes the evaluation of djhelp.info.

UWHSL and DJHCP personnel performed laboratory performance testing and field-
testing with Dodge and Jefferson community members. Performance testing results
revealed that the Web portal functioned as expected, although navigation strategies and
screen-based instruction were insufficient for naïve users to make full use of the resource.
Detailed observations made during the laboratory assessment revealed that the interface
was not intuitive. The absence of a site map or some overall explanation of all available
resources left many users confused as to the exact nature of the resource.

The field evaluation illuminated the general acceptability of the service to the community
and elicited strong expressions of interest in the availability of this service. It also
brought to light a number of areas of need for improvement that correlated with and
reinforced the concerns noted in the laboratory performance testing. Most participants
were able to complete searching tasks, and found djhelp.info retrieved useful results. The
degree of acceptability was directly related to the richness of the database included in the
version of the djhelp.info employed in the test, with higher ratings being associated with
the more extensive regional database.

Next-step recommendations for DJCHP to improve the value and function of djhelp.info
include:
       1. Develop a sustainable strategy to complete and maintain the database of local
           resources
       2. Manage user expectations by clarifying site functions with on-screen
           directions
       3. Provide on-screen explanations of the key word search functions
       4. Develop a brief user training module accessible from the main screen

Long-term development should focus on creating a more customized opening screen that
highlights the DJHCP/United Way ownership.
                                                                                          4




III. Business Sustainability Plan for djhelp.info

Strategic Analysis

The DJHCP Technology Planning Committee and UW-Madison team members
conducted a strategic analysis exercise on the position of the prototype digital health
information library to assist in business planning. A strategic (―SWOT‖) analysis
considers the strengths and weaknesses of an organization or business, and the
opportunities and threats it faces in its environment.

Strengths: The partnership between the University and the component members of
DJHCP creates a number of organizational strengths that can foster growth of djhelp.info,
blending the efforts, resources and visions of people and organizations who want to make
good things happen for the community. A partnership such as DJHCP can enhance the
potential for cooperative leadership and broad-based support of a community resource,
which would be more difficult for a single provider entity to achieve.

Combining trusted health information sources with local 2-1-1 health- and human
services information serves many community members at an appropriate level of service
and technology they choose for themselves. This model offers ―one-stop‖ shopping for
information, both digitally and by telephone. It is not aimed only at the poor or
underserved, but serves also the bulk of people in the middle, as well as those who are
technology-savvy and well informed.

Weaknesses: Connecting the digital library with the formative state and regional 2-1-1
system creates a number of constraints and restrictions. Ideally, the djhelp.info database
should mesh with other information systems across Wisconsin. However, dependence on
decisions of the Dane County and Wisconsin United Way 2-1-1 implementation efforts
hampered planning and decision-making in the Dodge and Jefferson county project,
which needs to move ahead on a shorter time frame.

The complex objectives identified for djhelp.info (as a resource for both local and state/
national information) create a potential stumbling block for user acceptance. The guided
search interface the consumer sees as the front end of the resource needs to work well
independently of any other functions. Need to have someone associated with project who
can do successful grant applications.

The organization structure of DJHCP and its Technology Planning Committee contain
some inherent potential weaknesses. Constituent organizations such as area hospitals and
clinics are in economic competition, as are the individual local communities in some
views. This may generate frictions and reduce cooperation. Active participation in the
planning committee has tended to devolve primarily to information provider
professionals (librarians and IT staff. This may limit the projects scope of vision and
community buy-in.
                                                                                            5



Opportunities: The convergence of state and federal bio-terrorism and bio-surveillance
efforts and development of a 2-1-1 information and referral service enhanced the
opportunity to create a complete database of up-to-date, easily-maintained locally
available health- and human services. In addition, local health care provider institutions
may be more supportive of this project because they perceive a need to enhance access to
and strengthen local services in order to fend off the encroachment of big-box corporate
health care. Conversely, Aurora Health Care may see a need to support local services in
order to demonstrate membership and goodwill in the community. However, involving
Aurora in the project could create conflicts and weaken current local supporting
organizations; this strategy should be weighed carefully for potential good and bad
consequences.

The djhelp.info database of local services will help people and organizations involved in
providing or directing those services, as well as community members. If djhelp.info is
well-marketed and becomes well-known and used, agencies and civic groups will see
advantages in continuing to fund it. Information that goes beyond the local service area
but that presents local services as a first choice is an inherent advantage. Local healthcare
providers, the counties, individuals and service groups are potential partners or
supporters. Djhelp.info could be something for which all the providers can take credit; it
then becomes a way to be a community booster.

There is a need to periodically inform or remind community members about djhelp.info,
since they may forget it exists if they have not needed it in the last year. It can be
included in orientation packets to community and church, marketed to churches through
ministers and councils, to schools through newsletters, and to employers and employees
through human resource departments (who have a local professional group). It may also
be possible to get a list of employers in Dodge and Jefferson counties from the state.
Providing information to teachers and instructors in school systems and technical schools
could help build the next generation of users, who would already be familiar with the
online community resource. Annual notices to all the outlets and organizations could be
brief, snappy, and ready to use rather than elaborate letters or articles requiring editing
and condensing for distribution.

Websites and online services may currently enjoy a certain glamorous cachet that can
attract community support and sponsorship. However there is also a risk underlying the
appeal of the ―new thing:‖ Websites can readily go stale, so that it is important to
provide continuous updating and checking for broken links and outdated information, in
order to avoid conveying an impression of shabbiness and neglect.

Threats: As djhelp.info matures, the present committed participants in DJHCP may
withdraw or move on and be replaced. It is important to recruit new members who are
not narrow-minded or self interested, but interested in the greater good.

There is a risk in the two counties of continuing economic downturn due to job layoffs
and company closings, that may impact future donations. Many individual service
                                                                                          6


agencies in the community are struggling to hold their heads above water. This presents
a potential problem if they don’t take the time to maintain current program information in
the database, and it becomes fragmented and out of date. There is also a challenge in
adding another project that competes for funding with existing ones. The threat of
competition with existing agencies may be eased by emphasizing that djhelp.info is
designed to improve access to local services by making consumers more aware of what is
available in the community, or making the community more aware of what needs to be
available.

There are no other agencies or businesses competing to provide services similar to
djhelp.info. There was a potential for competition or conflicting interests between
djhelp.info as a component of 2-1-1 service, and the United Way of Jefferson and North
Walworth Counties (UWJNWC), which operates a ―First Call for Help‖ telephone
information and referral service. However, this conflict was resolved by an agreement in
principal between DJHCP and UWJNWC to jointly sponsor the djhelp.info website and
database, and when the time arrives, operate a 2-1-1 call center. A draft Memorandum of
Understanding (MOU) between DJHCP and UWJNWC (see Appendix B) lays out a
mechanism for sharing the assets and responsibilities for these community resources.
Partnering with UWJNWC, which is also an organizational member of DJHCP, will
allow djhelp.info to be integrated into a major funding stream for the area, rather than
standing alone and competing with other agencies for donor dollars. Furthermore, the
partnership provides djhelp.info with the benefits of the United Way agencies business
contacts and expertise, which can help to broaden the vision of and support for the
website and database beyond the present core of concerned information professionals.

United Way of Jefferson and North Walworth Counties volunteered to partner with
DJHCP as a primary sponsor of the djhelp.info database and website because there is an
overlap of interest between this community service and the mission of UWJNWC. This
is particularly the case with the proposed 2-1-1 telephone referral service, which is being
developed statewide in Wisconsin as a project of the state United Way organization.
UWJNWC contributed its First Call for Help database, which forms a substantial part of
the djhelp.info local services base. This is a mutually beneficial partnership, which gives
DJHCP access to the local United Way’s business and community connections and
powerful funding expertise, and gives the United Way agency the ability to expand its
local information and referral service into a more powerful and enhanced 2-1-1 service
and web presence that takes advantage of DJHCP’s health care and information provider
expertise.

The draft MOU provides what amounts to a prenuptial agreement between DJHCP and
UWJNWC regarding the dispersal of the assets (website, database, and 2-1-1 call center)
in the event of termination of the agreement. The purpose of this agreement is to allow
each organization to proceed with confidence that it is not risking loss of control of its
most cherished objectives in this project, so that the parties can develop and maintain the
level of trust and cooperation necessary to jointly maintain the resource.
                                                                                          7



Budget and Funding.

(A summary budget narrative is presented in Appendix C. For a detailed budget analysis
document, see Appendix D.)

The estimated cost for licensing, training, operator certification and development
necessary to implement and continue full djhelp.info operations for the rest of the 2004
calendar year is $12,965.00, or $16,715 to allow operations for a full calendar year
(through July 2005). [These costs are in addition to expected donations of in-kind labor
and equipment services of $3,500 from UWJNWC and $400 from employers represented
on the DJCHP Planning Committee for startup, and $600 from United Way and $1,000
from DJHCP for 5-month operating costs.] After December 2004, expected annual
operating costs would average $8115 per year, in addition to donated labor and
equipment services of $1,265 from United Way and $2,400 from Oversight Committee
members.

DJHCP Technology Planning Committee members agreed with a number of other
sources indicating that a broad base of funding support is the most desirable situation for
a community service or resource. However, funding support by UWJNWC can be the
vital element that will allow djhelp.info to start up and go forward with a reasonable
modicum of financial stability. Broad-based community support can be encouraged and
maintained either by allowing directed giving to the United Way, specifically for support
of djhelp.info and 2-1-1, or by soliciting and accepting donations directly to djhelp.info
its management committee.

The financial structure of djhelp.info is envisioned as a two-tiered arrangement, with
DJHCP and UWJNWC the primary partners, sponsors and joint owners of the website,
database, and future call center. Both organizations would be identified on the ―brought
to you by‖ banner on the website, and possibly in a relevant telephone message.
Additional individual or organizational partners, donors, or sponsors may be solicited,
accepted, and acknowledged in yet-to-be-determined fashion, but DJHCP and UWJNWC
will remain the sole primary partners and owners, unless both parties agree to add another
primary partner some time in the future.

Also yet to be determined is the role of the other area United Way agencies, based in
Dodge and Jefferson counties. This is a matter that can be discussed and agreed upon
among the three local United Way agencies, but it appears that only the UWJNWC has
the resources and capacity to act as a primary partner, so that the other agencies, who also
want to participate in the project, will act as sponsors or donors.

After an initial period of database development and website enhancement, UWJNWC
support could probably be best applied to the known, ongoing operating costs of the
website and database (and eventually, call center). As additional functional
enhancements are identified and prioritized, the partners could apply for support from a
variety of foundation, corporate and governmental grant programs, for community health
                                                                                           8


and wellness, bioterrorism, digital library development, and technology development,
among others.

Discussion:

With or without the partnership and support of the United Way, the board and member
organizations of DJHCP will want to know that djhelp.info (as well as the associated
2-1-1 project) have a secure, stable and predictable funding base to support the ongoing
developmental and operational costs of the digital library. Further, the board and
executive director of UWJNWC and other potential partner organizations will want to see
a sound and realistic financial plan for djhelp.info and for DJHCP as a whole, in order to
be willing to commit resources and effort to this project.

It is important to recognize that the current project (that is, the Library of Medicine-
funded Internet Access to Digital Libraries project now being completed) was not
designed to provide substantial support for the 2-1-1 implementation, nor for DJHCP’s
program activities in general. These entities are so closely intertwined and interrelated,
however, that it is probably impossible to completely discuss one without considering the
others.

There are essentially four funding mechanisms that can serve to support djhelp.info into
the future, and allow it to develop to its fullest potential:
     Selling services
     Selling advertising
     Partnership support
     Government or private grants

Each mechanism has distinct advantages and drawbacks or limitations. These
mechanisms are not mutually exclusive, and could be combined in different measures to
take maximum advantage of the advantages and avoid drawbacks so far as possible.

Selling advertising may be the most effective way to generate support for a public
medium. It is probably the least desirable alternative, however, because it will potentially
subject djhelp.info to commercial pressures that may distort, crowd, or drown out the
intended public service benefits and messages of the website. Furthermore, the
credibility of the information in the digital library with consumers may be undermined if
they perceive that the information providers have a financial interest in shaping consumer
choices.

Partnership support shares some similar characteristics with advertising, in that partners
may be interested in opportunities to promote their own organizational interests through
the website. Partner interests may range from advancing common public interest goals,
to gaining visibility and good name on the website, to having logos, small blurbs, or links
on the website. There is no clear boundary between partner promotion and advertising,
and the same risks may entail, of crowding out the primary message, undermining
credibility, and encountering market pressure to modify the primary health message. For
                                                                                           9


this reason it is most preferable to develop partnerships based on common public interest
objectives. To the extent that promotional considerations are employed, they should be
carefully circumscribed by clear policies that limit the size, number, format and content
of promotional messages.

The primary partnership opportunity, which has already been discussed, is the offer of
cooperation with UWJNWC, and possibly with other area United Way organizations.
Another partnership opportunity that seems like a natural development that has not yet
been fully explored, is additional direct support from DJHCP member organizations,
particularly the area hospitals and clinics.

Selling services is an option that is built into the Resource House model on which
djhelp.info is currently based. Resource House provides options for sub-database
development, organizational client and service management, and other features tailored
for information/referral and service provider agencies, that can be licensed and
distributed by the license holder (DJHCP) to other organizations. This mechanism also
has some overlap with partnership funding, in that service provider organizations could
be solicited to provide some payment of support in exchange for being listed as resource
providers in the djhelp.info database.

All three mechanisms just discussed may likely be subject to a critical mass effect: a
certain degree of success in recruitment of users and resources, functionality and
availability of the digital library may be necessary before advertisers, service client
organizations, or potential partners may become convinced that it is worthwhile to invest
in djhelp.info. However, funding support is necessary to develop users and service
resource listings, functionality and availability.

For this reason, continuing grant support is appropriate and may be necessary for further
development of the digital library. It is important to recognize that grant support alone is
probably not a sustainable funding resource in the long term. Grant funding should be
well focused on the objectives and directions in which DJHCP (and partners) want to
develop the digital library/ health information resource, and the organization as a whole.
They should provide time and direction in which to develop a long term sustainable
business model of partnership support and revenue-generating services. Grant
development should be conducted concomitant with a strong marketing effort, defined in
the broadest sense as informing and educating as well as recruiting potential users,
service provider agencies, partners and donors.

The UW Health Systems Lab team have identified three potential research areas for
development of djhelp.info that could be explored in partnership with DJHCP, with a
potential for federal funding. We have identified federal grant programs that appear to
offer potential support for each research direction. Pursuit of a particular grant or
combination of grants would correspond to a particular choice of approach or emphasis in
development of the website, and perhaps of the organization as a whole. In addition,
there may be federal grant programs are outside of our area of research interest that might
be of interest to DJHCP in the pursuit and development of its organizational mission.
                                                                                        10


Although we do not envision a research role for our group in these programs, we might
be able to support DJHCP in a subcontractor relationship in developing program
proposals. (DJHCP would also be eligible to pursue these grants on its own, or in
partnership with other organizational entities.)

The research questions we propose, with supporting grants, are listed below. Fuller
elaborations of the grant funding opportunities are attached as appendices to this
document.

Question 1: How can we effectively obtain, develop, sustain and use community input in
developing regional digital health information resources?
Funding availability: NIH PAR-04-117, Understanding and Promoting Health Literacy
(Appendix F)
              Supports goals of ―Healthy People 2010.‖
              Letter of Intent deadline September 13, 2004
              Application deadline October 13, 2004

Question 2: Developing locally-relevant health content to be distributed via the
djhelp.info.
Funding:
        Understanding and Promoting Health Literacy (Appendix F)
        NIEHS/ NIOSH RFA-ES-04-007, Environmental Justice: Partnerships for
        Communication (Appendix G)
              Encourages community outreach, training, research, education and
              evaluation efforts to reduce exposure to and reduce health impact of
              environmental and occupational stressors in underserved populations.
              Application deadline November 17, 2004
        NLM/ NIMH PA 03-178, Informatics for Disaster Management Grants (Appendix
              H).
              Supports development of systems for collection and dissemination of
              information aiding in prevention, advance warning, early detection,
              notification of the public and authorities, mobilization of response,
              delivery of relief and medical care natural or technological disaster
              situations.
              Could involve working with Wisconsin Health Alert Network (HAN) and
              2-1-1.

Question 3: Assessment and enhancement of the capacity and utility to consumers of
djhelp.info for secure storage of personal health information.
Funding:
        NLM Information System Grants, PA 92-093
               Support the use of information technology to bring usable, useful health-
               related information to end users including health professionals,
               researchers, educators, students and consumers.
               Application deadline October 1, 2004 or February 1, 2005.
                                                                                          11


Organizational development funding opportunities.

There are several current federal funding opportunities that, although they do not directly
relate to research activities of interest to the UW Health Systems Lab, may contribute to
the organizational development of DJHCP and its mission objectives. For example,
DJHCP may be eligible for the following programs in the Health Resources and Services
Administration, US Department of Health and Human Services:
         Rural Health Network Development Grant Program, HRSA-05-002
                Supports activities to strengthen organizational capabilities of networks to
                overcome fragmentation and vulnerability of providers in rural areas.
         Rural Health Network Development Planning Grant Program, HRSA-05-003.
                Supports entities that need assistance to plan, organize, and develop a
                health care network for purposes as above.
         Rural Health Care Services Outreach Grant Program, HRSA-05-011
                Supports development of new and innovative health care delivery systems
                in rural communities that lack essential health care services. Requires
                formation of a consortium with at least 2 additional partners that could
                include schools, churches, EMS providers, universities, and private
                practitioners. Services could include hospice care, check-ups for children,
                prenatal care for women in remote areas.

Unfortunately all three programs currently have a very narrow application window, with
deadlines of September 20, September 8, and September 13 respectively. If DJHCP
identifies one of these or another program that appears to be a good fit, however, we will
try to develop an arrangement to provide technical assistance in rapidly developing a
proposal . Alternatively, DJHCP could contact the Health Resources and Services
Administration to inquire about other possible funding programs, or to get assistance in
preparing an application for the next funding cycle.

Governance.

Djhelp.info will require an ongoing governance structure to make editorial decisions and
set policies about website and database content, updates and upgrades, and business and
operational matters. The current technology planning committee has functioned
extremely well, and includes representation of organizational board members of DJHCP,
as well as the Executive Directors of DJHCP and UWJNWC. It would be a very good
idea, however, for the Planning Committee to turn its charge back to the boards of the
two organizations at this point for regeneration of an ongoing oversight committee. This
would accomplish the dual purposes of bringing fresh ideas and vision to the committee
and the project, and to allow for the graceful retirement of any committee members who
wish to turn to other pursuits after their period of service.

The draft MOU (Appendix B) proposes a committee of equal representation for the
DJHCP and UWJNWC boards on the oversight committee, in addition to the two
Executive Directors. The purpose of this balance is to ensure balance, fairness, and
cooperative decision-making. However, the possibility exists in an evenly balanced
                                                                                            12


committee of deadlock and inability to act. If this situation arises, the two boards will
want to reconsider whether some other arrangement would be preferable to allow
effective decision making.
Appendix B
djhelp.info Evaluation Report                                                          HEALTH
                                                                                        SYSTEMS
                                   DJHCP/UW Madison
                              Digital Libraries Project
                                                                                         LAB
                                      Evaluation Report
                                        June 7, 2004

Summary

The University of Wisconsin-Madison Health Systems Laboratory (UWHSL) assisted the Dodge
Jefferson Healthier Communities Partnership (DJHCP) to develop and evaluate djhelp.info, a
prototype web-based health and human services information and referral service. With funding
from a grant from the National Library of Medicine (LM007781-01), DJHCP and UWHSL
jointly developed djhelp.info to provide a locally-controlled Web portal to regional and national
information selected to the identified health information needs of residents of Dodge and
Jefferson counties. The Web portal, djhelp.info, implemented through an arrangement with the
United Way of Dane County (for database development) and NorthLight’s Resource House,
became active in March, 2004. djhelp.info provides access to information about health concerns
and local health resources through two navigation strategies: a keyword search and a guided
search approach. This report summarizes the evaluation of djhelp.info.

UWHSL and DJHCP personnel performed laboratory performance testing and field-testing with
Dodge and Jefferson community members. Performance testing results revealed that the Web
portal functioned as expected, although navigation strategies and screen-based instruction were
insufficient for naïve users to make full use of the resource. Detailed observations made during
the laboratory assessment revealed that the interface was not intuitive. The absence of a site map
or some overall explanation of all available resources left many users confused as to the exact
nature of the resource.

The field evaluation illuminated the general acceptability of the service to the community and
elicited strong expressions of interest in the availability of this service. It also brought to light a
number of areas of need for improvement that correlated with and reinforced the concerns noted
in the laboratory performance testing. Most participants were able to complete searching tasks,
and found djhelp.info retrieved useful results. The degree of acceptability was directly related to
the richness of the database included in the version of the djhelp.info employed in the test, with
higher ratings being associated with the more extensive regional database.

Next-step recommendations for DJCHP to improve the value and function of djhelp.info include:
        5. Develop a sustainable strategy to complete and maintain the database of local
            resources
        6. Manage user expectations by clarifying site functions with on-screen directions
        7. Provide on-screen explanations of the key word search functions
        8. Develop a brief user training module accessible from the main screen

Long-term development should focus on creating a more customized opening screen that
highlights the DJHCP/United Way ownership.
                                                                                       B2
djhelp.info Evaluation Report


   I.      Introduction and overview of djhelp.info


DJHCP, Inc, and the University of Wisconsin-Madison Health Systems Laboratory
collaborated to develop a locally-vetted web portal to digital library resources. Funded
by a grant from the National Library of Medicine (LM 2001) these collaborators created
and evaluated djhelp.info. A Planning Committee, structured as a subcommittee of the
DJHCP Board, oversaw the content selection, design, contracting with the web hosting
service, and all training and evaluation activities. Content selections were based on
surveys of consumers and information and referral professionals conducted by UWHSL
and the Planning Committee.

Web Portal: djhelp.info

 djhelp.info (http://www.djhelp.info/) is a prototype information and referral resource
designed to be used by consumers, health providers and information professionals. The
website is a collaborative effort between the University of Wisconsin-Madison Health
Systems Laboratory, the Dodge Jefferson Healthier Communities Partnership (DJHCP)
and selected community and technical partners including the Dane County United Way.
The project’s scope and purpose articulates with a statewide ―2-1-1‖ effort that seeks to
provide a comprehensive information and referral network across Wisconsin. Users
employ one of three information searching methods to gain access to summaries of local
service providers or particular health concerns. These summaries in turn provide links
that the user can traverse to read more extensive information displayed on the screen. If
the user is located at a computer connected to a printer, information displayed on the
screen can be printed and taken with the user for later review.

Design and Technical Features

djhelp.info is implemented within Resource House, a proprietary web interface and web
hosting service provided by NorthLight, Inc., of Wilmette, Illinois. Resource House
employs an online approach for Information and Referral services. Several regional
organizations have created unique access web pages to Resource House, which is
actually a single site hosting local and regional databases from all around the United
States. Resource House selects the database it searches based on the location input by the
user. Each subscribing client that operates a database on Resource House can re-brand its
segment of the Resource House web page as the home page that appears when its
database is in use, and assign its own preferred URL for the site.

The design strategy behind djhelp.info was to provide both locally relevant service
referrals and trusted state and national sources of general health information by
enhancing the standard Resource House website with links and database elements
pointing to the National Library of Medicine’s MedLine Plus, an Area Health Education
Center (AHEC) digital library, and the State of Wisconsin Consumer Guide to Health
                                                                                         B3
djhelp.info Evaluation Report

Care. Global navigation features such as search help, FAQ, and technical assistance are
available on all screens.

Database Structure and Content

The local database of community resources for Dodge and Jefferson counties was built
by DJHCP with the assistance of the United Way agencies of Dane County and of
Jefferson and North Walworth counties. A relational database was created by the United
Way of Dane County, populated with local information from the Dodge and Jefferson
Counties and transferred to NorthLight to be linked to the Resource House utility.

Accessing Local and National Information on djhelp.info

Citizens and Information and Referral professionals who access djhelp.info first
encounter a greeting page labeled with the DJCHP logo. The Resource House
information and referral search engine is activated in one of two ways: by typing a word
or phrase into a key word search box, by deliberately selecting one of three general
information links, or by initiating a six-step guided search process. Status of the guided
search strategy is displayed through a navigation banner at the top of the page.

The guided search strategy functions as follows: The user begins at step one by
identifying the location or the location of interest (by community name or zip code). This
selection determines what local database the search engine will search, and also the web
site’s configuration and functions. (If the user selects a location outside of Dodge and
Jefferson counties, the user is directed to the local or regional database assigned to that
address.) At step two, the user faces a choice between several options. They can
continue with the built-in Resource House search engine for searching the local
community resource database, by entering a search term in the prominently displayed
keyword search box, or by using the topical pull-down menus provided. They will then
progress through a series of prompts asking them to indicate, and then focus, their topic
of interest.

djhelp.info employs the proprietary search engine used by Resource House for keyword
searching. Resource House keyword searching on djhelp.info conducts searches by
matching the term typed in by a user to fields in the data base that label each data base
entry with relevant topics and subtopics in the taxonomy codes used to classify the
resource records in the database. The key word search does not search the text of the
records in the local database. Instead, keywords are linked to. Keywords thus function as
a shortcut into specific areas of the topical directory. Topics appear in the directory only
if they are linked to a record in the database for a service provided by an agency with the
location selected by the user as part of its service area. Thus, a keyword search will only
successfully connect to one or more records if a related service is available in the user’s
area.
                                                                                          B4
djhelp.info Evaluation Report

Selecting one of the three general health information links, NLM’s MedLinePlus, an Area
Health Education Center (AHEC) digital library, or the Wisconsin Consumer Guide to
Health Care redirects the user out of the six-step search process to the identified resource.
Currently djhelp.info does not have the capability to ―hand off‖ search terms to the
included general health information databases (MedLine Plus, AHEC, or Wisconsin
Consumer Guide). The user who follows a link to one of these resources for health
information is obliged to initiate a new search using the search engines offered by each of
those sites.



   II.       Evaluation

Two sets of evaluation activities were conducted: performance testing and field testing.
Performance tests were conducted by members of the UWHSL research team, and
UWHSL and the Planning Committee collaborated on a field test involving expected
users of the digital health library, including professional health care and health
information providers and lay members of the community. Three field tests were
performed on the Dodge Jefferson Healthier Community Partnership’s
(DJHCP) digital library webs portal, djhelp.info, between March 16th
and April 14th, 2004. The goal of testing was to identify performance, usability, and
general design problems. The first and second evaluation sessions were completed at the
University of Wisconsin-Madison’s Health Systems Lab.

Performance Test

Three users conducted standard navigation test of the djhelp.info under laboratory
conditions. All navigation links were checked. Test of all searching strategies were
conducted.

Field Test

Field evaluation was conducted with volunteer users from the community between March
21 and April 27, 2004. Test participants used the web site to conduct two different types
of searches representing information needs that had been identified in the community.
The first search scenario asked participants to locate a local health or human service
resource; the second scenario asked them to locate a piece of general health-related
information (drug, disease or wellness issue, or location of a specific health-care
provider) of interest to themselves. After each search, the volunteers answered a series of
questions evaluating the process and outcome of their search. Finally, they were asked to
respond to a series of general questions on the overall usability and usefulness of the
djhelp.info site.

Two different groups of users tested djhelp.info under two different modes. Group 1 used
the system with only the still-to-be completed DJHCP-provided database; Group 2 used a
                                                                                      B5
djhelp.info Evaluation Report

northern Midwest regional database (MinnesotaHelp) that had been fully operational for
over a year.

We compared the evaluations of users of the local and regional databases in order to
assess the performance of the website and its functions from assessment of the richness
and completeness of the underlying database. We also compared evaluations by
professional information providers (information professionals and health care providers)
with evaluations by personal information seekers (general community members) in order
to assess usability and utility of the website to each of these groups, which can have
different skill levels and information needs.


Test User Demographics:

Volunteer test-user demographics are summarized in Table 1. Evaluation participants
were predominantly working-age White/European females. Distribution of test versions
was skewed primarily due to timing and logistics of test scheduling and distribution: the
majority of users of the first evaluation version (using the local Dodge-Jefferson
database) resided in Dodge County; participants using the second version, based on the
full MinnesotaHelp database, were predominantly Jefferson County residents. Evaluators
were highly educated, and half or more indicated that they primarily provided health
information for clients or patients. Nearly 2/3 of all test participants considered
themselves highly experienced computer users, and none were inexperienced or novices.
The levels of education, health information and service delivery, and computer
experience are higher than what would be expected for the general population of the two
counties, but more typical among likely users and first adopters of the digital health
information resource.
                                                                                        B6
djhelp.info Evaluation Report



Table 1: Description Of The Samples

   Category                     Group 1 (local database)      Group 2 (MinnesotaHelp)
   Number of participants       22                            15
   Males                         19% ( n = 4)                  6.7% (1)
   Age (percentage):
      26-62                     90.5%                         93.3%

   Ethnicity:
   White/European               95.2%                         100%
   Hispanic/Latino               4.8%
   Dodge Co. residence:         61.9%                          26.7%
   Education:
   High School or some          43%                           60%
     college
   College or Advanced          57%                           40%
     Degree

   Professional Information
   or Service Provider           67%                            46.7%
   Computer Experience:
      High                       67%                             57%



Summary Of Usability Test Results:

The results of user testing are summarized in tabular form below. Responses are
summarized for lay and health information professional users over the Dodge Jefferson
(test group 1) and MinnesotaHelp.info (test group 2) databases. Results displayed are
mean responses of each group on a scale of 1 (Strongly disagree) to 7 (Strongly agree).
Table 2 summarizes user ratings of the search experience on task 1 (searching for a local
service agency using the Resource House search engine). User ratings of their
experience in the task 2 search (searching for general health information) are displayed in
Table 3.

Responses to questions about overall usability of djhelp.info are displayed in Table 4,
broken down by user type (lay or professional), and Table 5, broken down by test group
(using the Dodge Jefferson or MinnesotaHelp database).
                                                                                          B7
    djhelp.info Evaluation Report

    Table 2: Acceptability for task #1: Search for local service agency (Response on 1
    (Strongly disagree) to 7 (Strongly agree) scale):

                                                               Rating (Mean, StdDev)
                                          Group 1 (local database)      Group 2 (Minnesota Help)
                                          Lay             Professional Lay           Professional
1. Found what I was looking for            4.71, 2.563     5.43, 1.399   6.50, .756    6.71, .488
2. Results were understandable             5.43, 2.070     5.79, 1.188   6.13, .641    6.71, .488
3. Results were useful                     4.43, 2.507     4.93, 1.592   6.37, .744    6.83, .408
4. Time to get results appropriate         4.86, 2.410     5.50, 1.401   6.38, .916    6.57, .787


    Table 3: Acceptability for task #2: Search for general health information (Response on 1
    (Not acceptable) to 7 (acceptable) scale):


                                                           Rating (Mean, StdDev)
                                      Group 1 (local database)   Group 2 (MinnesotaHelp)
                                      Lay          Professional Lay          Professional
1. Found what I was looking for       3.14, 2.410 3.85, 2.544 5.57, 2.070      4.71, 2.059
2. Results were understandable        4.71, 2.289 5.20, 2.251 5.71, 2.138      5.50, 1.378
3. Results were useful                3.50, 2.168 4.08, 2.610 5.43, 2.070      4.57, 2.225
4. Time to get results appropriate    3.67, 2.338 4.82, 2.442 4.50, 1.871      4.43, 2.507



    Table 4: Assessment of general usability by user type (Response on 1 (Not acceptable) to
    7 (acceptable) scale):


    Question                                               Rating (Mean, StdDev)
                                             Lay                      Professional
    1. Menus, messages helpful               5.33, 1.633              5.48, .981
    2. Liked overall appearance              5.73, 1.163              5.62, 1.203
    3. Organization of site clear            5.92, 1.03               5.38, 1.071
    4. Has all expected functions and        4.6,1.598                3.81, 1.778
    capabilities
    5. Helps with problem correction         4.67, 2.000              5.00, 1.732
    6. Easy to learn                         5.87, .990               5.62, 1.322
    7. Easy to use                           5.93, 1.100              5.43, 1.248
    8. Satisfactory information quality      5.00, 1.890              4.38, 2.012
    9. Would use again                       5.53, 1.846              5.29, 1.736
    10. Would recommend to others            5.33, 1.915              5.29, 1.678
                                                                                          B8
    djhelp.info Evaluation Report

    Table 5: General usability by underlying database (Response on 1 (Not acceptable) to 7
    (acceptable) scale):


                                                                Rating (Mean, StdDev)
                                          Group 1 (local database) Group 2 (MinnesotaHelp)
1. Menus, messages helpful?               5.4, 1.24                 5.5, 1.36
2. Liked overall appearance?              5.6, 1.20                 5.7, 1.16
3. Organization of site clear?            5.6, 1.10                 5.6, 1.08
4. Has all expected functions and         3.7, 1.82                 4.8, 1.42
capabilities
5. Help with problem correction           4.8, 2.14                 4.9, 1.45
6. Easy to learn                          5.6, 1.29                 5.9, 1.03
7. Easy to use                            5.4, 1.24                 6.0, 1.07
8. Satisfactory information quality       4.0, 2.04                 5.6, 1.4
9. Would use again                        4.9, 2.03                 6.1, .92
10. Would recommend to others             4.7, 1.96                 6.2, .86




    Discussion of survey results

             Local Searching (Table 2). Comparing the assessments of local database users
    and Minnesota database provides an opportunity to measure the importance of the local
    data quality to users. In Task 1, searching for local service referrals, user evaluations
    were more positive when this search was performed on the complete Minnesota database.
    Test users perceived the benefit and potential utility of the local referral service;
    provision of an adequate base of local service providers for Dodge and Jefferson counties
    is critical.

    Professional service and information providers (users who seek information for others)
    rated the local searching function more positively than users who seek information
    primarily for self (and family) use. This suggests that non-professional users may need
    more assistance or instruction in using the resource. Service and information providers
    may also be more interested in or motivated to use this type of resource, since they
    provide this type of information as a regular part of their employment, and because the
    resource was selected and designed predominantly by individuals who are information
    and service providers.

    General Information Searching (Table 3)

    The distinction between evaluations of users who used the local database and those who
    used the complete MinnesotaHelp database persisted and even strengthened in this part of
    the test. Some of the evaluations were negative, with mean ratings below the neutral
    score of 4.
                                                                                        B9
djhelp.info Evaluation Report


The purpose of this scenario was to induce the user to search one of the national or
statewide information links; the intention of the djhelp.info site design was not to have
the local databases used for general information searching. However, the importance of
the local database to satisfaction with results in this task indicates that users did not
perceive or follow this intended mode of use. Free-text comments also indicated that
many users did not understand the process of using a link to take their information search
to an outside site, but tried to use the keyword search box in step two. Although this
would not work particularly well on either the Minnesota or Dodge-Jefferson databases
because the keywords are not set up to provide detailed health-term searching, it would
achieve some successes on the more complete database, but would have almost no chance
of any hits in the very limited Dodge-Jefferson data. Frustration with the performance of
djhelp.info in the local search may also have persisted, to negatively color perceptions of
performance in the general information search.

Evaluations of the general information search were less positive than those for the local
search, for category of user and regardless of which database version was involved.
Again, this probably reflects the fact that many users did not understand the intended use
design, but tried to perform the general search using the search box and guided search
that worked on the local databases.

Information and service providers again rated their search experience more positively
than did general information seekers, but their evaluations did not improve as much when
using the more complete database.


Overall Usability

Across users, djhelp.info was rated as acceptable and usable, with the single exception
that the information and service providers on average did not agree that the service ―has
all the functions and capabilities I expected.‖ Free-text comments and verbal comments
to test conductors indicated that many of these professional users had high expectations
of the site, or felt strongly that there was a need for such a local resource, and were
therefore disappointed with the functional limitations of the prototype. In particular
negative comments focused repeatedly on the failure of the keyword search feature to
perform as expected, and on the weakness of the local community database. The
professional and information providers were generally knowledgeable about many
specific resources that actually exist in Dodge and Jefferson counties, and thus in a
position to directly perceive resources that should have appeared but didn’t.

Responses with a mean rating between 4.0 and 5.0 can be regarded as weak positive
responses, where djhelp.info was perceived as acceptable but not impressive. To the
extent that this resource needs to make a positive impression on users in order to achieve
wide community acceptance, support and adoption, these can also be seen as indicators of
areas where there is room for improvement. These areas of weakness include delivery of
all expected functions and capabilities and help with problem and error correction (as
                                                                                         B10
djhelp.info Evaluation Report

perceived by general users) and quality of information provided (as rated by professional
providers).

General usability by underlying database

Some additional conclusions may be inferred by comparing the overall usability ratings
of users of the Dodge-Jefferson and the more complete MinnesotaHelp databases (table
below). Overall ratings were notably more positive with the more complete database.
Mean ratings did not vary widely between the two tests for site appearance, organization,
or the helpfulness of menus, messages, and documentation. The Minnesota database test
received somewhat higher valuation scores for ease of learning and ease of use.

Users of the Minnesota database ranked the site significantly higher (more than 1 point
difference in mean ranking) for having all expected functions and capabilities, for having
satisfactory information quality, and for likelihood that they would use it again or
recommend it to others. This appears to be a strong indicator reinforcing the message
that early enhancement and completion of the local community resource will go far in
enhancing the appeal and usefulness of djhelp.info in the community.

However, even with the complete MinnesotaHelp database underlying the Website, user
rankings were only weakly positive for completeness of expected functions and
capabilities, and for help with correction of problems and errors. Thus, enhancement of
site functions, or at least better explanation to the users of what functions are available,
and enhancement of error messages or enhancement of training and on-screen
instructions to anticipate and reduce errors, are areas where site managers could direct
improvement efforts.

Summary of key findings and observations of Performance and Field testing

   1. Overview

Searches could be conducted in one of three ways – by clicking on a link, entering a word
into a search strategy, or in a guided process. Participants commonly opted to begin the
search by typing a word into the search box, rather than using the links or the guided
process. Because the databases were structured to match user-provided search terms only
to index terms assigned to each entry and not to the entire text description of the database
entry, this typed-word search approach failed many of the times.

Participants appeared to confuse, or not understand, that djhelp.info included several
databases, and the search process automatically restricted the search to only one of the
databases. Thus, if an individual launched a search for general health information through
the search process linked to the local data base, they received few if any results because
the local database included information only about specific service providers.

 Searches conducted to locate local services received higher ratings than did searchers
conducted to locate general health information. Caution must be taken in interpreting
                                                                                        B11
djhelp.info Evaluation Report

these results due to the small sample size, (which precluded statistical testing) the low
number of tasks (which may negatively bias appraisal).

   2. Interface

Detailed observations made during the laboratory assessment revealed that the interface
was not intuitive to most users. The opening screen provides options to conduct searches,
review the privacy policy, sign-in and select between two search strategies. In addition,
the presence of corporate logos (including DJCHP) indicated sponsorship but did not
provide sufficient information about the nature of resources that one could access
through this resource. The absence of a site map or some overall explanation of all
available resources left many users confused as to the exact nature of the resource.

Two additional problems were noted by test observers during the field evaluation.

          If the user selected their location by name rather than zip code, the distance-
           computation algorithm for displaying results did not function, and resources
           would display in alphabetical order. Since there are a number of national
           resources (toll-free information lines, service organizations) in the database,
           and a relatively small number of local resources in Dodge and Jefferson
           communities, the local resources would frequently not appear on the first
           page, leading the inexperienced user to conclude that none were found. This
           problem may be reduced when the Dodge-Jefferson database is fully
           populated.
          If a user initially selects a location outside of Wisconsin, then later returns to
           the first step to enter a Dodge-Jefferson location, the home page does not
           return to djhelp.info, but instead reverts to the LaCrosse site. This was
           discussed with the NorthLight/ Resource House programmers. They indicated
           that this was a problem in their programming, which assumed the existence of
           only one Wisconsin site, as was the case prior to djhelp.info coming on line.
           NorthLight stated that they were working on programming to resolve this
           fault.

   3. Search and search support

   Several search supports exist, each providing increasing refinement of the search
   strategy. The intent is likely to insure a focused search and precise results, but the
   effect appeared to be confusion of novice searchers.

   The search engine does not perform in the manner that users expect.

   Additionally, there is a need to better distinguish between the tasks of searching for
   information and searching for resources.
                                                                                         B12
djhelp.info Evaluation Report

   III.      Recommendations

          1) Completing the local database is the single most critical improvement for
             community acceptability and utility.

          2) Clarify site functions with up-front direction regarding search choices. The
             choices could be clarified by customizing the Step 2 page (―Choose a topic‖)
             with the screen divided into two choices: ―Find General Health And Wellness
             Information‖ including the MedLine Plus and Healthy Wisconsin People
             links, and ―Find Local Services,‖ to include the Wisconsin Consumer Guide
             to Health Care site under ―Find a Doctor,‖ as well as the Resource House
             search box and topical menu.

          3) Resolve perceived problems with keyword search function:
             Short term: Place a brief note on how keyword searching works on this site,
             directly above the keyword search box, or a link ―Click here for instructions
             on how to use this feature,‖ or a java pop up message over the box, ―Use this
             feature for a shortcut to find local service topics.‖
             Medium term:
                  Develop a brief user training module to appear as a link on the site
                     (perhaps next to the search box) and/or to be presented to information
                     moderators such as public librarians to provide public training and
                     assistance.
                  Develop a role for an individual trained in AIRS Info-line taxonomy
                     and/or NorthLight’s ―People’s Taxonomy‖ coding to monitor reports
                     of keyword failure and enhance keyword linking to topical areas as
                     deficiencies are noted, or reported through the Provider Annex.
                  Discuss with NorthLight whether they will customize the page to
                     eliminate the word ―keyword‖, label the box simply as ―Search,‖ with
                     directions ―Enter a search term to find related local topics.
                  Customize the ―step two‖ page to display the keyword search less
                     prominently and associate it more with the topical pull-down search.


             Long term:
                 Consider and discuss with NorthLight the possibility of addition a
                    utility to hand off search terms from the local search engine to search
                    engines on other databases (along the lines of North Carolina ―Go
                    Local‖).
                 Once database deficiencies are minimized, consider whether adding a
                    true text-based search engine searching specific data fields such as
                    agency mission or groups served would provide a cost-effective
                    enhanced benefit. Request an exported copy of the Dodge-Jefferson
                    database as configured for Resource House to assist in this analysis.
                 Join actively in the online user discussion forum hosted at
                    http://www.northlightsoft.com/discuss.htm to raise questions about the
                                                                                    B13
djhelp.info Evaluation Report

                  search engine and other aspects of web site functioning. Use the
                  discussion forum as well as references provided by NorthLight in their
                  original proposal to contact other users, discuss mutual interests and
                  concerns, and move toward development of a users group to negotiate
                  with NorthLight.

       4) Consider additional customization of the website to enhance DJHCP/ United
          Way ownership by
              re-branding to replace the Resource House logo
              modifying the six-step boxes to a less ponderous graphic of the
                process

       5) Follow up with NorthLight to ensure that the database selection utility works
          properly and reliably in referring users with Dodge and Jefferson search
          locations to the Dodge-Jefferson database and home page.
Appendix C: Memorandum of Understanding to Develop and Sustain djhelp.info                  1
                                                                                           C1
(DRAFT)
Memorandum of Understanding – Joint Venture

Overview

This memorandum of understanding will identify key points of agreement between the United
Way of Jefferson and North Walworth Counties (UWJNWC) and Dodge Jefferson Healthier
Community Partnerships (DJHCP) in relation to the development of a comprehensive database,
Website, and Information and Referral (2-1-1) telephone service inclusive of health and human
service agencies and programs that support the population of Dodge, Jefferson and North
Walworth Counties.

Key Points of Agreement

    DJHCP and the UWJNWC will partner to develop a web based database of health and
     human service providers that serves the populations of the greater Dodge, Jefferson and
     North Walworth counties, and a health and human services information Website and 2-
     1-1 telephone information and referral service to be supported by the database.
    DJHCP and the UWJNWC agree that the developed database will comply with federal,
     state and/or local standards that have been created to support a 2-1-1 call center.
    DJHCP and the UWJNWC agree that the development of a database precedes the
     development of a 2-1-1 call center for Dodge, Jefferson and North Walworth Counties.
    DJHCP and the UWJNWC agree to develop policies and procedures necessary to
     develop, support and maintain a viable health and human services database that is
     interactive and comprehensive, together with the associated website, djhelp.info, and
     the Dodge-Jefferson 2-1-1 call service. These will include but are not necessarily limited
     to editorial management and policies of the djhelp.info website, content management
     and policies of the database, future decisions regarding continuation of present hosting
     arrangements and format of the database and website with NorthLight Inc. (dba
     “Resource House”) or selection of alternate arrangements, and staffing and funding
     arrangements for maintenance of the database and website. The above policies and
     decisions will be made by majority vote of a website management committee
     representing DJHCP and UWJNWC. Membership of the managing committee shall
     include the executive directors of DJHCP and UWJNWC, and up to three board
     members, or designated representatives approved by the boards of each organization.
     The executive director of each organization (with approval of his or her board) may
     withdraw or replace his or her designated managing committee members at will. The
     managing committee will elect a chairperson for a term of one year. If a managing
     committee member resigns or is unable to serve, the executive director of his or her
     organization may request up to one month to replace the member before any future
     meetings or votes, unless the chairman decides there is an urgent need for a decision or
     action within less than a month.
    DJHCP and the UWJNWC agree that the developed database will serve not only as a
     tool of education but also an assessment tool, identifying areas of need, inquiry and
     usefulness to our service area’s populations.
    DJHCP and the UWJNWC agree that the UWJNWC and its First Call for Help and
     Information program will provide staffing to administer the database’s maintenance and
     reporting mechanisms. Both parties agree that such administration will be subject to
     policies and standards set by the managing board (described below). The staff person or
     persons providing the maintenance and reporting activities will need to be trained in
     AIRS InfoLine taxonomy and/or Resource House People’s Taxonomy. If the Executive
Appendix C: Memorandum of Understanding to Develop and Sustain djhelp.info                     2
                                                                                              C2
(DRAFT)
       Directors of either DJHCP or the UWJNWC do not agree that the existing staffing
       arrangement is working satisfactorily, they may require renegotiation of this clause to
       develop an alternative staffing arrangement.
      DJHCP and the UWJNWC agree to jointly select, design, market and monitor the
       activity of the database. All reports, requests, marketing, design and hosting decisions
       and related email and other communications relating to the database and website will be
       shared by and communicated to both DJHCP and UWJNWC, so long as this agreement
       remains in effect.
      DJHCP and the UWJNWC agree to predominately co-brand the database, promoting
       the database as a joint venture developed by DJHCP and the UWJNWC. DJHCP and
       UWJNWC will jointly own the database. Either partner may, for its own purposes,
       request and obtain a copy of the then-current database. The requesting partner will be
       responsible for any downloading or conversion fees payable to the hosting company for
       delivery of the database copy, unless both partners agree to jointly receive the data for
       some joint purpose. Neither party may sell, rent, lease or otherwise distribute the
       database without written approval of the other, other than for the purpose outlined
       above of providing information and services to residents of the Dodge-Jefferson-North
       Walworth service area. Permissible distribution includes provision of one electronic or
       paper copy of the database per year to any agency that has contributed directories,
       resource lists or databases toward the development of the jointly-owned database.
      DJHCP and the UWJNWC agree that other entities may partner with DJHCP and the
       UWJNWC in support of the database, website, and/or 2-1-1 telephone service;
       however, DJHCP and the UWJNWC will remain as predominant co-brands of the
       developed database. Inclusion of any additional primary partners to this agreement
       requires consent of both DJHCP and the UWJNWC.
      DJHCP and the UWJNWC agree that funding by the UWJNWC will be on a year by
       year basis with details to be outlined in subsequent memoranda of understanding.
       DJHCP and the UWJNWC agree that UWJNWC will provide full funding for the
       continued licensing, development, hosting and operation of the database and website
       for the balance of the current year (2004) as outlined in the attached budget, less any
       funds received from other sources for this purpose, including grants and donations.
      DJHCP and the UWJNWC agree that in the interest of sustainability of the database,
       website, and 2-1-1 telephone service, it is desirable to maintain as broad a base of
       funding support as possible within the community. This may be accomplished via grants
       from government and/or foundations, and donations from individuals, organizations, or
       corporations, either directly to DJHCP, or through directed donations to the
       UWJNWC targeted to support the website, database and/or 2-1-1 service. Donors or
       funders may be recognized as “partners” on the website or in other publications of
       DJHCP or UWJNWC; however, this category is distinct from the category of primary
       partners, identified as the organizing, founding managers and creators of the services, for
       example as identified on the top of the djhelp.info website as “brought to you by ….”
      DJHCP and the UWJNWC agree that this memorandum of understanding may be
       amended by mutual agreement. If either party is unable or unwilling to comply with any
       terms of this agreement, and the parties are unable to mutually agree on a modification
       of the agreement, the agreement may be declared dissolved by the unsatisfied party.
      In the event this agreement is dissolved, the following final terms will apply. Each party
       has the right to request and obtain at their own expense, a copy of the then-current
       database. From that point forward, each party will be responsible for ongoing
       maintenance of its copy of the database. UWJNWC will have the right of first refusal of
       ownership and full responsibility for the 2-1-1 telephone service. DJCHP will have the
Appendix C: Memorandum of Understanding to Develop and Sustain djhelp.info                 3
                                                                                          C3
(DRAFT)
     right of first refusal of ownership and full responsibility for the djhelp.info website,
     including contractual relationship with NorthLight, Inc. for maintenance of the database.
     If UWJNWC does not assume responsibility for continuation of the
     2-1-1 telephone service, or if DJHCP does not continue the
     maintenance of the djhelp.info website or the database hosting
     contract with NorthLight, any of these rights and responsibilities may
     be assumed by the opposite party. If this agreement is dissolved,
     DJHCP and UWJNWC each agree to maintain the terms of the
     agreement until released by the other party, up to a maximum of
     three months. For example, UWJNWC would continue to provide
     previously agreed-upon funding support and database
     maintenance activities, and DJHCP would continue to provide
     voting rights, input into database decisions, and copies of all
     database related communications and reports, until no longer
     needed for up to three months.
    “Approval” or “permission” of either primary partner for actions, changes to this
     agreement, etc. are deemed to be granted with the approval or agreement of each
     partner’s Executive Director. However, such approval or agreement may be rescinded
     by either partner’s board of directors within one month (30 days) of the Executive
     Director’s action.
Appendix D: Summary of Budget for djhelp.info                                             1
                                                                                         D1


Summary/explanation of budget for djhelp.info website and database:

Budget items that will be met by in-kind donation (services and equipment provided by
United Way of Jefferson and North Walworth Counties, labor of Planning/Steering
Committee members) are included as both expenses and income in order to more
accurately reflect the true cost and value of the services provided.

Options:
DJHCP/ UWJNWC can purchase the Resource House Website license for an additional 5
months (balance of 2004) for $3,000, or purchase a 12 month license for $5,000.

Data Manager purchase (including Administer, which is necessary to make Data Manager
functional) and Reporting, which may be optional but important, is $3,000.

DJHCP/ UWJNWC can purchase 1 day of training on operating Data Manager and the
Resource House database, MSDE server, etc. for $1,000.

DJHCP/UWJNWC must purchase a software maintenance/license/upgrade package with
the Windows software bundle (Data Manager etc.) The yearly cost is 20% of the
software bundle price. If NorthLight agrees, DJHCP can purchase the software
maintenance package for 5 months to match the web hosting license, or for a 1 year
period. The cheapest up-front option would be to purchase the software and web licenses
both for 5 months, then renew for a year in January 2005. However, this requires
additional funding outlay in 5 months. Purchasing a one-year web license represents a
significant additional up-front expenditure; purchasing a one year software license is not
a particularly big jump (requires an additional $350 up front).

DJHCP/UWJNWC can elect to have the data base administrator AIRS certified in Info-
Line taxonomy or not (at present). This is not a requirement for operating the on-line
database, but may result in better database/website performance and better search results.
Once 2-1-1 is up and running, the 2-1-1 operation needs to have an AIRS-certified
operator or database administrator. If certification is elected, there is a significant cost
discount for AIRS members.

DJHCP can elect to join AIRS or not. If the database is housed and operated out of
UWJNWC, the United Way’s AIRS membership will provide member discounts for the
AIRS Info-Line Taxonomy license fee and for certification of the database administrator
(if desired). However, DJHCP may see a benefit in also joining AIRS in order to have a
vote and voice in AIRS policies and activities, and for additional access to AIRS
information.

Services:
Updating, configuring, customizing djhelp.info website ($150 per hour to Resource
House)
Appendix D: Summary of Budget for djhelp.info                                          2
                                                                                      D2


Personnel:
1. Database/server operator and administrator is assumed to work 150 hours in updating
and completing the database, then two hours per month ongoing:
    Call local resources and providers, collect data or inspire them to add data through
                 Provider Annex
    Conduct fund raising and volunteer recruitment during database recruitment?
    Respond to emails, keyword problems, enhance keyword indexing and taxonomy
    Carry out Oversight Committee directions on inclusion of specific providers to
       database? Or, Make inclusion decisions and report to Oversight Committee for
       review.
    Assign/approve provider access to Provider Annex
    Monitor response to provider update notices; conduct phone calls or write letters
       to elicit updates from providers without email, or providers who don’t respond.
    Maintain server operations, troubleshoot database and software errors
    Maintain contacts and communication with NorthLight regarding technical
       problems, customization etc requested by Oversight Committee? Or, could this
       task be carried out by designated representative of the Oversight Committee, or
       by the Executive Director?


2. Oversight committee (for website and database) are assumed to be volunteers or to
   have their time donated by their regular employer. They will make decisions on
   website configuration, enhancement, customization, and content changes. Forward to
   NorthLight via database/server administrator, or via Executive Director or a
   designated representative of the committee?
        Set inclusion policy for database.
        Set policies and practices for database updating (set up automatic contacting
          and update reminders, set frequency for update reminders)
        Correspond with other Resource House users, and or Resource House user
          group?
        Make inclusion decisions on provider applications? Or forward policies to
          administrator, let administrator make decisions, and review decisions/actions
          for adherence to policies.
        Assist in telephone calling to elicit enrollment in database via Provider Annex,
          or collect info for manual data entry
        Conduct fund raising and volunteer recruitment during database recruitment?

3. Executive Director is assumed to spend 4 hours per month on business management:
       Meet with Oversight Committee
       Carry out business communications with NorthLight (payments, contract
          renewals; requests for Webpage and database configuration and
          enhancement?)
       Contacts and correspondence with Resource House users?
       Technical communications with NorthLight (if not done by
          operator/administrator)
Appendix D: Summary of Budget for djhelp.info                                              3
                                                                                          D3


This effort is not assumed to be donated by DJHCP in the present budget. If the DJHCP
board does agree to donate these hours, that would reduce the 5-month funding
requirement by $1,000 and the full-year requirement by $2,400.

Equipment: $2,000 (or leased/rented, or at Jefferson/North Walworth United Way
MSDE server to run SQL local database--See NorthLight Hardware Needs document.

The Bottom Line:
Additional funding required for the first 5 months of operation (from startup through
calendar year 2004) is $9,065 with one day of Resource House training, and AIRS
certification for the database administrator/operator. Without training or certification, the
additional funding needed for the first 5 months is $8,000. With training but no
certification, $9,000 is required; with certification but no training, $8,065.

To start and operate for the first 5 months, but purchase a full one-year software
maintenance/license, will cost an additional $350.00 up front, added to any of the above
needs.

Full operations for the first one-year period, including all licensing, one-time startup costs
and recurrent yearly costs, requires an additional $12,815 with Resource House training
and AIRS operator certification. The minimum one-year funding without training and
certification would be $11,750.

Subsequent years will require funding input of $8,100 per year, plus an additional $30 in
alternate years for AIRS operator recertification, if the operator has been certified.

If DJHCP can donate 4 hours of Executive Director time per month, the estimated 5
month startup funding requirement would be reduced by $1,000, and any full year startup
or operating requirement would be reduced by $2,400.

Note: One additional possible startup expense that has not been included in this budget is
the payment to Dane County United Way for database development, because of questions
about the amount of work performed under contract by DCUW. This amount (if any)
when known or negotiated should be added to each of the first year startup options,
whether for 5 months, mixed, or 1 year funding.
                 Appendix E: Budget Calculation Spreadsheet for djhelp.info                                                                                                                                                          E1



   Proposed 3 year budget for djhelp.info website and database 2004             Developed by/ contact: David Grindrod email: grindrod@wisc.edu (Revision date: August 27, 2004)
                                                                                One-time,
Budget Category:         labor hours                  wage or fee               startup costs Recurring costs
                                                                                                                                                                                      * assumes DJHCP opts and NorthLight agrees to prorate
                                                                                               2004 (5 mo.*)     2004 (mixed yr**) 2004(full yr***)       2005       2006               software maintenance fees for balance of year 1
Resource House                                                                                                                                                                        ** assumes DJHCP pays for 5 months extension of Web license
Website license                                                                                       3,000.00            3,000.00         5,000.00    5,000.00   5,000.00               but pays full 1-year software fees
Software Purchase:                                                                                                                                                                    ***assumes DJHCP pays for full 1 year web license and
                                                                                                                                                                                         software fees.
Data Manager
Administer
Reports                                                                             3,000.00
Software
Maintenance/Update
Fee                                                                                                     250.00              600.00           600.00     600.00     600.00
Database
Administrator Training                                                              1,000.00
Database Server                                                                     2,000.00
Database and Server
Maintenance                                      12                    $75.00                           375.00              375.00           900.00     900.00     900.00
Database Completion                             150                    $10.00       1,500.00
Business
Administration--
Executive Director                               48                    $50.00                         1,000.00            1,000.00         2,400.00    2,400.00   2,400.00
Database
administration                                   24                    $10.00                           100.00              100.00           240.00      240.00     240.00
Website administration                           96                    $25.00                         1,000.00            1,000.00         2,400.00    2,400.00   2,400.00
Website enhancement/
customization
(Planning& Design)                               16                    $25.00         400.00
Website enhancement/
customization
(Programming)                                     4                   $150.00         600.00
AIRS Info-Line license                                                                                  200.00              200.00           200.00     150.00     150.00 AIRS 3 yr
(AIRS member
discount)                                                                                               -50.00              -50.00            -50.00     -50.00     -50.00
AIRS Info-Line
Certification                                                                                           200.00              200.00           200.00                165.00
(AIRS member
discount)                                                                                              -135.00             -135.00           -135.00               -135.00
AIRS membership
(budget <75,000)                                                                                        125.00              125.00           125.00     125.00     125.00
AIRS membership
(budget >75,000)                                                                                        250.00              250.00           250.00     250.00     250.00
AIRS costs               (Small MEMBER)                                                                 340.00              340.00           340.00     225.00     255.00   820.00
                         (Large MEMBER)                                                                 465.00              465.00           465.00     350.00     380.00 1,195.00
                         (NON-MEMBER)                                                                   400.00              400.00           400.00     150.00     315.00   865.00
                Appendix E: Budget Calculation Spreadsheet for djhelp.info                                                                                                   E2




                                                                                    One-time,     Total Fixed +
                                                                                    startup costs Recurring costs
                                                                                                  2004 (5 mo.*) 2004 (mixed yr**) 2004(full yr***)       2005        2006
                                                          Table of first year startup costs and
                                                          in-kind contributions (note-these lines
                                                          are incorporated in first-year annual
Total Costs                                               costs o the right..
(annual recurring)                                                                                      6,065.00          6,415.00       11,880.00   11,765.00   11,795.00
fixed and recurring:        With Training+Certification                                  8,500.00      14,565.00         14,915.00       20,380.00   11,765.00   11,795.00
                            +Training - Certification                                    8,500.00      14,500.00         14,850.00       20,315.00   11,765.00   11,765.00
                            -Training +Certification                                     7,500.00      13,565.00         13,915.00       19,380.00   11,765.00   11,795.00
                            -Training - Certification                                    7,500.00      13,500.00         13,850.00       19,315.00   11,765.00   11,765.00

INCOME:
source:               item:                       type:
           UWJNWC Server                          in-kind (shared use)                  -2,000.00       -2,000.00        -2,000.00       -2,000.00
                  " " server/database maintenance in-kind (shared use)                                    -375.00          -375.00         -900.00     -900.00     -900.00
                  " " database completion         in-kind (labor)                       -1,500.00       -1,500.00        -1,500.00       -1,500.00
                  " " database admininstration    in-kind (labor)                                         -100.00          -100.00         -240.00     -240.00     -240.00
                  " " AIRS membership             in-kind (shared benefit)                                -125.00          -125.00         -125.00     -125.00     -125.00
         DJHCP Plan
      Committee (from
        member orgs) Website administration       in-kind (labor)                                       -1,000.00        -1,000.00       -2,400.00   -2,400.00   -2,400.00
                             Website enhancement
                       "   " planning                     in-kind (labor)                 -400.00        -400.00           -400.00         -400.00

Total In-Kind Income                                                                    -3,900.00       -5,500.00        -5,500.00       -7,565.00   -3,665.00   -3,665.00

Net funding needed          With Training+Certification                                  4,600.00       9,065.00          9,415.00       12,815.00    8,100.00    8,130.00
                            +Training - Certification                                    4,600.00       9,000.00          9,350.00       12,750.00    8,100.00    8,100.00
                            -Training +Certification                                     3,600.00       8,065.00          8,415.00       11,815.00    8,100.00    8,130.00
                            -Training - Certification                                    3,600.00       8,000.00          8,350.00       11,750.00    8,100.00    8,100.00
9,065.00
Appendix F                      NIH PAR-04-117                                      1
                   Understanding and Promoting Health Literacy
       UNDERSTANDING AND PROMOTING HEALTH LITERACY (R03)




General Information

      Document Type:                        Grants Notice
      Funding Opportunity Number:           PAR-04-117
      Posted Date:                          Jun 28, 2004
      Original Due Date for Applications:   Multiple Receipt Dates - See Link to Full
                                            Announcement for details.
      Current Due Date for Applications:    Multiple Receipt Dates - See Link to Full
                                            Announcement for details.
      Archive Date:                         Nov 13, 2006
      Funding Instrument Type:              Grant
      Category of Funding Activity:         Education
                                            Environment
                                            Health
                                            Income Security and Social Services
      Expected Number of Awards:            Not Available.
      Estimated Total Program Funding:      Not Available.
      Award Ceiling:                        None
      Award Floor:                          None
      CFDA Number:                          93.113 -- Biological Response to
                                            Environmental Health Hazards
                                            93.121 -- Oral Diseases and Disorders
                                            Research
                                            93.242 -- Mental Health Research Grants
                                            93.279 -- Drug Abuse Research
                                            Programs
                                            93.286 -- Discovery and Applied
                                            Research
                                            93.393 -- Cancer Cause and Prevention
                                            Research
                                            93.847 -- Diabetes, Endocrinology and
                                            Metabolism Research
                                            93.865 -- National Institute of Child
                                            Health and Human Development
                                            Extramural Research
                                            93.866 -- Aging Research
                                            93.879 -- Medical Library Assistance
      Cost Sharing or Matching Requirement: No
Appendix F                       NIH PAR-04-117                                         2
                    Understanding and Promoting Health Literacy
                             Eligible Applicants
      State governments
      County governments
      City or township governments
      Special district governments
      Independent school districts
      Public and State controlled institutions of higher education
      Native American tribal governments (Federally recognized)
      Public housing authorities/Indian housing authorities
      Native American tribal organizations (other than Federally recognized tribal
      governments)
      Nonprofits having a 501(c)(3) status with the IRS, other than institutions of higher
      education
      Private institutions of higher education
      For profit organizations other than small businesses
      Small businesses
                                     Agency Name
                                     Description
      The participating Institutes, Centers and Offices of the National Institutes
      of Health (NIH) and the Agency for Healthcare Research and Quality (AHRQ)
      invite investigators to submit R03 research grant applications on health
      literacy. The goal of this Program Announcement is to increase scientific
      understanding of the nature of health literacy and its relationship to
      healthy behaviors, illness prevention and treatment, chronic disease
      management, health disparities, risk assessment of environmental factors, and
      health outcomes including mental and oral health. Increased scientific
      knowledge of interventions that can strengthen health literacy and improve
      the positive health impacts of communications between healthcare and public
      health professionals (including dentists, healthcare delivery organizations,
      and public health entities), and consumer or patient audiences that vary in
      health literacy, is needed. Such knowledge will help enable healthcare and
      public health systems serve individuals and populations more effectively and
      employ strategies that reduce health disparities in the population.

      Healthy People 2010 defines health literacy as the degree to which
      individuals have the capacity to obtain, process and understand basic health
      information and services needed to make appropriate health decisions (U.S.
      Department of Health and Human Services, 2000). Many factors affect
      individuals ability to comprehend, and in turn use or act on, health
      information and communication. Proficiency in reading, writing, listening,
      interpreting, oral communication, and visual analysis is necessary as the
      modern health system typically relies on a variety of interpersonal, textual,
      and electronic media to present health information. Individuals and families
      both must be able to: communicate with health professionals; understand the
      health information in mass communication; understand how to use health-
      related print, audiovisual, graphical and electronic materials; understand
      basic health concepts (e.g., many health problems can be prevented or
Appendix F                       NIH PAR-04-117                                       3
                    Understanding and Promoting Health Literacy
      minimized) and vocabulary (e.g., about the body, diseases, medical
      treatments, etc.); and connect this health-related knowledge to health
      decision-making and action-taking. Access to and understanding of health
      information and services is a reciprocal process among health professionals,
      communication professionals and patients. For instance, these professionals
      must use science-based strategies and tactics, develop resources and
      materials, and understand communication interactions between providers and
      patients.

      Research on health literacy should assist NIH in its mission of communicating
      scientifically-based health information to the public and to the health care
      providers and related professionals who serve the public. The application of
      scientific knowledge from health literacy research may also strengthen the
      health information knowledge and communication skills of the public, and
      further one of the national goals of Healthy People 2010, to improve health
      literacy by the decades end.
                         Link to Full Announcement:
      http://grants.nih.gov/grants/guide/pa-files/PAR-04-117.html

      If you have any problems linking to this funding announcement, please contact
      the NIH OER Webmaster NIH OER Webmaster
Appendix G              NIEHS/NIOSH RFA-ES-04-007                                  1
               Environmental Justice: Partnerships for Communication




    ENVIRONMENTAL JUSTICE: PARTNERSHIPS FOR COMMUNICATION




General Information

      Document Type:                         Grants Notice
      Funding Opportunity Number:            RFA-ES-04-007
      Posted Date:                           Jun 14, 2004
      Original Due Date for Applications:    Nov 17, 2004
      Current Due Date for Applications:     Nov 17, 2004
      Archive Date:
      Funding Instrument Type:              Grant
      Category of Funding Activity:         Environment
                                            Health
      Expected Number of Awards:            Not Available.
      Estimated Total Program Funding:      Not Available.
      Award Ceiling:                        $225,000.00
      Award Floor:                          none
      CFDA Number:                          93.113 -- Biological Response to
                                            Environmental Health Hazards
                                            93.114 -- Applied Toxicological
                                            Research and Testing
                                            93.115 -- Biometry and Risk
                                            Estimation_Health Risks from
                                            Environmental Exposures
                                            93.956 -- Agricultural Health and Safety
                                            Programs
      Cost Sharing or Matching Requirement: No
                            Eligible Applicants
      State governments
      County governments
      City or township governments
Appendix G              NIEHS/NIOSH RFA-ES-04-007                                       2
               Environmental Justice: Partnerships for Communication
      Special district governments
      Independent school districts
      Public and State controlled institutions of higher education
      Native American tribal governments (Federally recognized)
      Public housing authorities/Indian housing authorities
      Native American tribal organizations (other than Federally recognized tribal
      governments)
      Nonprofits having a 501(c)(3) status with the IRS, other than institutions of higher
      education
      Private institutions of higher education
      For profit organizations other than small businesses
      Small businesses
                                     Agency Name
                                     Description
      The purpose of this program is to strengthen the National Institute of
      Environmental Health Sciences' (NIEHS) and the National Institute for
      Occupational Safety and Health's (NIOSH) support of research aimed at
      achieving
      environmental justice for socioeconomically disadvantaged and medically
      underserved populations in the United States. One goal of the NIEHS and
      NIOSH
      is to understand the influence of economic and social factors on the health
      status of individuals exposed to environmental toxicants and occupational
      hazards and impact public health. The intent is to promote health research,
      education and intervention programs that address improved ways to serve low
      income, immigrant, and minorities who may be disproportionately exposed to
      environmental and occupational stressors. This component of the research
      program in environmental justice is designed to encourage community outreach,
      training, research, education and evaluation efforts that will become the
      catalyst for reducing exposure to or reducing the health impact from
      environmental and occupational stressors in underserved populations. The main
      objective of this RFA is to establish methods for linking members of a
      community, who are directly affected by adverse environmental or occupational
      conditions, with researchers and health care providers and to create
      partnerships that can address environmental and occupational health problems
      and develop appropriate research and policy strategies to impact public health.

      With the intention of impacting public health and policy the National
      Institute of Environmental Health Sciences established the Translational
      Research Program. This effort supports research and education examining the
      effects and risks to human health from exposure to physical and social
      environmental agents. The Environmental Justice agenda, which falls within the
      Translational Research Program, is one such endeavor that seeks to minimize and
      prevent adverse health effects from environmental exposures through public
      outreach and education coupled with innovative research to develop solutions
      and have significant impact on public health and policy. For further
Appendix G              NIEHS/NIOSH RFA-ES-04-007                                       3
               Environmental Justice: Partnerships for Communication
      information on the Translational Research Program, see
      http://www.niehs.nih.gov/translat/envjust/envjust.htm.

      This current effort on environmental justice will ensure that:

      o The community is aware of basic environmental/occupational health concepts,
      issues, and resources.

      o The community has a role in identifying and defining problems and risks
      related to environmental and occupational exposures and stressors.

      o The community is included in the dialogue and is integral in shaping research
      and policy approaches to the problem.

      o The community actively participates with researchers and health care
      providers in developing responses and setting priorities for education and
      intervention strategies.

      One aim of this program is to facilitate the process of developing the trust
      needed for establishing effective partnerships among individuals who are
      adversely impacted by environmental and occupational hazard(s) in a
      socioeconomically disadvantaged community, researchers in
      environmental/occupational health, and health care providers. The
      collaborative team should then be able to initiate a research program that
      incorporates all parties and seek to reduce exposure to or health impact from
      environmental and occupational stressors that may manifest in the workplace or
      home environment.

      It is important to note that, because of the wide range of environmental and/or
      occupational health problems to be addressed and the diversity of affected
      communities, applications must be multidisciplinary in nature. At least one
      member of each of the following three personnel groups must have an active and
      meaningful role in both the development of the application and conduct of the
      proposed project:

      o A research scientist in environmental health sciences (including but not
      limited to those at NIEHS Environmental Health Sciences Centers or NIEHS
      Centers for Children's Environmental Health and Disease Prevention Research or
      NIOSH Educational and Research Centers or Agricultural Research Centers).

      o A primary health care provider directly involved in a community affected by
      environmental or occupational hazard(s). This may include a public health
      nurse and other such professionals. This individual must have a record of
      providing health care to the participating community. It is not necessary that
      the person be affiliated with a county or state public health department.
Appendix G              NIEHS/NIOSH RFA-ES-04-007                                       4
               Environmental Justice: Partnerships for Communication
      o A community-based organization (CBO) in an area having an underserved
      population that is adversely affected by an environmental or occupational
      hazard. This CBO must work directly and regularly with the affected community.


      These personnel must be listed on Page 2 of the PHS 398 application, and a
      biographical sketch must be provided for each. The role of each member of
      these three personnel groups in developing the application and carrying out the
      project must be clearly identified and fully described. There should be an
      equitable distribution of responsibilities as well as of requested financial
      resources among the three personnel groups. This goal is often accomplished
      through the inclusion of one or more subcontract arrangements, which may be
      helpful in defining all parties' programmatic and budgetary roles.
      Applications lacking the required personnel will not be considered.

      The NIEHS and NIOSH have a significant commitment to the support of
      programs
      designed to increase the number of underrepresented minority and female
      scientists participating in biomedical, environmental, and behavioral research.
      Therefore, applications from minority individuals and women are encouraged.
      Since the projects are situated in the community, well-established community-
      based or faith-based organizations are encouraged to consider their capacity to
      serve as the primary applicant organization. Due to the complex
      administrative, programmatic, and financial responsibilities associated with
      this role, such groups should contact Program Staff listed under WHERE TO
      SEND
      INQUIRIES for guidance.
                          Link to Full Announcement
      http://grants.nih.gov/grants/guide/rfa-files/RFA-ES-04-007.html

      If you have any problems linking to this funding announcement, please contact
      the NIH OER Webmaster: FBOWebmaster@OD.NIH.GOV
Appendix H                       NLM PA 03-178                                   1
                   Informatics for Disaster Management Grants



             Informatics for Disaster Management Grants

   Introduction
   PAR Number: PA 03-178
   Release Date: 09-30-03
   Expiration Date: 07-21-06, unless reissued


   Scope and Priorities

   Purpose
   The National Library of Medicine, the National Institute of Mental Health, and
   the National Institute of Biomedical Imaging and Bioengineering wish to
   support informatics research that addresses biomedical information
   management problems relevant to management of disasters. Disasters can
   be caused by nature or by man, through accident or by malice. Terrorism,
   particularly bioterrorism, is now an important focus of federal activity, but
   terrorism is only one of a number of threats to public safety classified as
   disasters. Disaster management is heavily dependent on efficient flow of
   information. How best to utilize information technology in a disaster situation
   poses a number of problems for which relevant informatics research is
   necessary.

   Research Objectives
   The Federal Emergency Management Agency (FEMA) classifies disasters as
   Natural (e.g. earthquakes, hurricanes, floods, wildlife fires etc.) or
   Technological (e.g. terrorism, nuclear power plant emergencies, hazardous
   materials etc.). Other classification schemes exist, but whatever the cause,
   certain features are desirable for management of almost all disasters:

         Prevention
         Advance warning
         Early detection
         Analysis of the problem, and assessment of scope
         Notification of the public and appropriate authorities
         Mobilization of a response
         Containment of damage
         Relief and medical care for those affected

   Because efficient realization of these elements is heavily dependent on
   adequate and timely flow of information, appropriate use of modern
   information technology can be of enormous help to those responsible for
   disaster management. How best to use the information systems in place, and
Appendix H                       NLM PA 03-178                                  2
                   Informatics for Disaster Management Grants

   how to improve their contribution to disaster management are the concerns of
   this PA. Some informatics issues relevant for disaster management include,
   among others:

         Standards to permit collection of data from diverse sources
         Coordination of disparate information systems across
          jurisdictional boundaries, and among system owners
         Design and deployment of appropriate databases for
          assessment of impact pre- and post-disaster
         Current, accurate information for clinicians and public
         Decision support systems for diagnosis and treatment
         Pattern recognition, data mining, and knowledge extraction
          algorithms
         Preserving privacy while protecting the public
         Pre- and post-disaster tracking systems for response
          personnel, disaster victims, and other resources
         Timely mobilization (including prior or standing IRB approval)

   For the most part these are the same issues that have interested
   informaticians during the past decade with considerable success for clinical
   information systems. However, specific solutions are needed within the
   context of disaster management. Universal installation of electronic medical
   record systems and physician order entry would doubtless provide multiple
   benefits; however, such systems are not likely to be widely in place in the
   near future. Solutions are needed that can utilize or incrementally improve the
   health care information systems of today. This PA is intended to encourage
   applications that seek such solutions.

   Within this area, NIMH is particularly interested in encouraging researchers to
   develop and test data collection technologies and procedures that will speed
   and coordinate the process of acquiring and utilizing new information about
   mental health consequences of disasters. The development and testing of
   approaches for computerized patient triage and tracking systems,
   coordination of databases and survivor registries, data storage and
   confidentiality procedures, and environmental stress/damage assessments
   are examples of the types of projects that can help to gather, manage, and
   disseminate mental health relevant information in a disaster situation. Such
   information may: improve long-term survival, help prepare for subsequent
   incidents, aid in assessing the physical and emotional needs of a population,
   impact on planning and provision of mental health services for victims and
   other disaster-affected persons, and increase understanding of the human
   response to trauma more broadly.

   NIBIB supports hypothesis-, design-, technology- or problem-driven research
   relating to the discovery, design, development, translation, application, and
Appendix H                       NLM PA 03-178                                    3
                   Informatics for Disaster Management Grants

   assessment of new knowledge in biomedical imaging and bioengineering. In
   the context of disaster management NIBIB is interested in (i) technologies
   related to data reduction, data mining, and knowledge extraction; (ii) the
   enhancement of the utility of existing databases by development of tools for
   meta-databases and integrative services; and (iii) methods for timely,
   appropriate communication of pertinent knowledge to cognizant bodies.


   Eligibility

   Eligible Institutions
   You may submit (an) application(s) if your institution has any of the following
   characteristics:

         For-profit or non-profit organizations.
         Public or private institutions, such as universities, colleges,
          hospitals, and laboratories.
         Units of State and local governments.
         Eligible agencies of the Federal government.
         Domestic or foreign.
         Faith-based organizations

   Individuals Eligible to Become Principal Investigators
   Any individual with the skills, knowledge, and resources necessary to carry
   out the proposed research is invited to work with an eligible institution to
   develop an application for support. Individuals from underrepresented racial
   and ethnic groups as well as individuals with disabilities are always
   encouraged to apply for NIH programs.


   Mechanism
   This PA will use the NIH R21 award mechanism. As an applicant, you will be
   solely responsible for planning, directing, and executing the proposed project.

   This grant program uses "just-in-time" concepts. It also uses the modular and
   non-modular budgeting formats (see
    http://grants.nih.gov/grants/funding/modular/modular.htm )

   Specifically, if you are submitting an application with direct costs in each year
   of $250,000 or less, use the modular format. Otherwise, use the standard
   PHS 398 instructions for detailed budgets.


   Review Criteria
   The goals of NIH-supported research are to advance our understanding of
Appendix H                       NLM PA 03-178                                    4
                   Informatics for Disaster Management Grants

   biological systems, improve the control of disease, and enhance health. In the
   written comments, reviewers will be asked to discuss the following aspects of
   your application in order to judge the likelihood that the proposed research will
   have a substantial impact on the pursuit of these goals:

         Significance
         Approach
         Innovation
         Investigator
         Environment

   The scientific review group will address and consider each of these criteria in
   assigning your application's overall score, weighting them as appropriate for
   each application. Your application does not need to be strong in all categories
   to be judged likely to have major scientific impact and thus deserve a high
   priority score. For example, you may propose to carry out important work that
   by its nature is not innovative but is essential to move a field forward.

   (1) Significance: Does your study address an important problem? If the aims
   of your application are achieved, how do they advance scientific knowledge?
   What will be the effect of these studies on the concepts or methods that drive
   this field?

   (2) Approach: Are the conceptual framework, design, methods, and analyses
   adequately developed, well integrated, and appropriate to the aims of the
   project? Do you acknowledge potential problem areas and consider
   alternative tactics?

   (3) Innovation: Does your project employ novel concepts, approaches or
   methods? Are the aims original and innovative? Does your project challenge
   existing paradigms or develop new methodologies or technologies?

   (4) Investigator: Are you appropriately trained and well suited to carry out
   this work? Is the work proposed appropriate to your experience level as the
   principal investigator and to that of other researchers (if any)?

   (5) Environment: Does the scientific environment in which your work will be
   done contribute to the probability of success? Do the proposed experiments
   take advantage of unique features of the scientific environment or employ
   useful collaborative arrangements? Is there evidence of institutional support?

   Protections: The adequacy of the proposed protection for humans, animals,
   or the environment, to the extent they may be adversely affected by the
   project proposed in the application.

   Inclusion: The adequacy of plans to include subjects from both genders, all
Appendix H                       NLM PA 03-178                                  5
                   Informatics for Disaster Management Grants

   racial and ethnic groups (and subgroups), and children as appropriate for the
   scientific goals of the research. Plans for the recruitment and retention of
   subjects will also be evaluated.

   Data Sharing: The adequacy of the proposed plan to share data.

   Budget: The reasonableness of the proposed budget and the requested
   period of support in relation to the proposed research.

   Additional Review Criteria: In addition to the above criteria, your application
   will also be reviewed with respect to the following:

         Can the project produce results useful now or in near future?
         Is the work applicable to other sites?
         Is the P.I. appropriately aware of existing disaster
          management plans and resources?
         Is attention paid to the use of standards for information
          management?
         Are user acceptance issues considered?
         Will the work be adaptable to tomorrow's information
          technology?
         Is the evaluation plan appropriate?

   Award Criteria
   Applications submitted in response to the Small Grant Program will compete
   for available funds with all other recommended applications. The following will
   be considered in making funding decisions:

         Scientific merit of the proposed project as determined by peer
          review
         Availability of funds
         Relevance to program priorities



   Application Deadlines
   February 1, June 1, and October 1. (application receipt, review, and award
   cycles)


   Application and Process
   Applications must be prepared using the PHS 398 research grant application
   instructions and forms (rev. 5/2001). The PHS 398 is available at
   http://grants.nih.gov/grants/funding/phs398/phs398.html For further
   assistance contact:
Appendix H                         NLM PA 03-178                                 6
                     Informatics for Disaster Management Grants

   GrantsInfo@nih.gov
   Telephone (301) 435-0714

   Specific Instructions for Modular Grant Applications: The NLM Small
   Grant program is covered by the NIH policy requiring all applications
   requesting up to $250,000 per year in direct costs to be submitted in a
   modular grant format. The modular grant format simplifies the preparation of
   the budget in these applications by limiting the level of budgetary detail.
   Applicants request direct costs in $25,000 modules. Section C of the research
   grant application instructions for the PHS 398 (rev. 5/2001) at
    http://grants.nih.gov/grants/funding/phs398/phs398.html includes step-by-
   step guidance for preparing modular grants. Additional information on
   modular grants is available at
    http://grants.nih.gov/grants/funding/modular/modular.htm

   Sending an Application to the NIH: Submit a signed, typewritten original of
   the application, including the checklist, and five signed photocopies in one
   package to:
   Center for Scientific Review
   National Institutes of Health
   6701 Rockledge Drive, Room 1040, MSC 7710
   Bethesda, MD 20892-7710
   Bethesda, MD 20817 (for express/courier service)

   Application Processing: Applications must be received by or mailed on or
   before the receipt dates described at
    http://grants.nih.gov/grants/funding/submissionschedule.htm The CSR will
   not accept any application in response to the Small Grant Program that is
   essentially the same as one currently pending initial review unless the
   applicant withdraws the pending application. The CSR will not accept any
   application that is essentially the same as one already reviewed. This does
   not preclude the submission of a substantial revision of an application already
   reviewed, but such application must include an Introduction addressing the
   previous critique.


   Review Considerations
   Applications will be reviewed for scientific and technical merit by the NLM
   Biomedical Library and Informatics Review Committee (BLIRC)

   As part of the initial merit review, all applications will:

         Receive a written critique
         Undergo a selection process in which only those applications
          deemed to have the highest scientific merit, generally the top
          half of applications under review, will be discussed and
Appendix H                       NLM PA 03-178                                7
                   Informatics for Disaster Management Grants

          assigned a priority score
         Receive a second level review by the appropriate national
          advisory council or board



   Contact Information
   The NLM encourages inquiries concerning this PA and welcomes the
   opportunity answer questions from potential applicants. Before submitting an
   application, interested applicants should contact Dr. Valerie Florance to
   discuss their projects:

   Valerie Florance, PhD
   Extramural Programs
   National Library of Medicine
   Rockledge 1, Suite 301
   6705 Rockledge Drive MSC 7968
   Bethesda, MD 20892
   Telephone: (301) 594-4882
   FAX: (301) 402-2952
   Email: floranv@mail.nlm.nih.gov

   For NIMH and NIBIB contact information, see the NIH Guide program
   announcement at http://grants2.nih.gov/grants/guide/pa-files/PA-03-178.html


   Required Federal Citations
   http://www.nlm.nih.gov/ep/GrantFedCitations.html


   NIH Guide Document
   Informatics for Disaster Management, PA 03-178
   http://grants2.nih.gov/grants/guide/pa-files/PA-03-178.html
Appendix I                      NLM PA 92-093                           1
                           Information System Grants


                        Information System Grants

    Scope and Priorities
    The National Library of Medicine (NLM) offers Information System
    grants to health-related institutions that wish to exploit the
    capabilities of information technology to provide the professionals
    and clients of their organization with access to high-quality health-
    related information. The Information System Grants are offered as
    a means of initiating, improving and fostering the use of computer
    and telecommunications technologies to coordinate and disseminate
    health information.

    Information System Grants emphasize the use of information
    technology to bring usable, useful health-related information to end
    users. Computers and networks provide health professionals,
    researchers, educators, students and consumers with time and
    place independent access to the published biomedical knowledge,
    files, images and clinical and scientific databases they need, and
    with the ability to communicate with patients and colleagues around
    the world. The term 'information system' is defined broadly to
    include equipment, software, knowledge resources and services,
    and human expertise. An Information System grant can be used to
    support a variety of activities, including but not limited to:

            Creating new and/or unique digital information resources
            Integrating digital information that comes from different
             sources to create tailored views
            Testing the usability of interfaces and digital resources
            Customizing information resources or services to meet needs
             of special audiences
            Delivering information resources and services to underserved
             rural and urban populations
            Applying a new information technology to improve access to
             information
            Designing and testing a new information service.

    Defining an information problem, assessing user needs and
    intended uses of health-related information, designing and
    deploying a proposed solution to the problem, training users to use
    the new resource or service, and performing outcome-oriented
    evaluation are fundamental steps in a good information systems
    project that must be addressed in any application for an
    Information Systems Grant.
Appendix I                       NLM PA 92-093                                    2
                            Information System Grants

    NLM strongly encourages applications which: (1) incorporate as an
    essential feature online access to NLM databases and programs
    such as MEDLINE®/PubMed®, MEDLINEPlus®, Entrez, TOXNET®,
    ClinicalTrials.gov, and PubMedCentral; and (2) support some aspect
    of NLM's objectives as articulated in the 2000-2005 Long Range
    Plan.
    Information System grants are not merely grants for hardware or
    telecommunications systems. They should bring health-related
    information to end-users. Applicants should describe their approach
    to these activities, address mechanisms of increasing use of the
    proposed system, and discuss their plans for supporting the system
    after grant funding ends. Applications not incorporating national
    network access will not normally be funded. Connectivity to the
    Internet is fundamental to successful Information System Grant
    applications. The following types of projects are considered outside
    the scope of this grant program: online library catalogs; single-
    purpose clinical systems (e.g. stand-alone laboratory system or
    Picture Archiving system (PACS)); digitization of print materials;
    projects that duplicate NLM products and databases. Applicants
    should contact the program official listed below if they are uncertain
    about the eligibility of their proposed project or activity.
    An Information System grant project must lead to an operational
    service activity. It may encompass a whole system, test a prototype
    and initiate a fuller implementation, establish connectivity of
    existing system components, or explore the feasibility of applying a
    new technology on a small scale.


    Eligibility
    U.S. public and private, non-profit institutions engaged in health
    administration, education, research, promotion and/or clinical care are eligible
    to apply. "Health-related" includes medicine, dentistry, nursing, pharmacy,
    veterinary medicine, and other activities related to the promotion of health and
    the prevention and treatment of disease. Hospitals, clinics, schools and
    community health centers, libraries and public health facilities, units of state
    and local governments among others, are examples of organizations that are
    encouraged to apply. Groups (or cooperatives) of health-related institutions
    are also eligible to apply. However, a single, lead institution must coordinate
    the application and apply on behalf of the group. Participants in multiple
    institution or consortium arrangements may be any organizational entity
    (including, for example, Federal institutions, proprietary hospitals, public
    libraries, etc.) which will significantly contribute to the service improvement
    objectives of the whole project.
Appendix I                        NLM PA 92-093                                      3
                             Information System Grants

    NLM encourages applications come from institutions that serve rural, inner
    city, or other under-served areas. Racial/ethnic minority individuals, women,
    and those with disabilities are particularly welcome as Principal Investigators.


    Mechanism
    Information System Grants use the G08 resource grant mechanism. An
    Information System grant provides up to three years of support and is not
    renewable. The average award is approximately $150,000 per year.
    Applicants who wish to request more than $150,000 per year should contact
    the program official listed below before submitting an application.

    Information System Grants support direct costs for personnel, equipment,
    training, travel, consultants and other costs related to the implementation of
    the proposed project. Those institutions that lack appropriate staff to design
    and implement an information system project or that need technical advice,
    appropriate consultation may be budgeted.

    For complex projects, Information System grant proposals may include
    requests for funds to support short-term planning. Planning support is
    intended to defray costs such as meeting and travel expenses, legal
    (contract) fees, staff release time for surveys of technological resources, and
    consultant fees for assessment of systems compatibility.

    Facilities and administrative (F&A) costs, sometimes called overhead or
    indirect costs, are not provided. Responsibility for the planning, direction, and
    execution of the proposed project will be solely that of the applicant.


    Review Criteria
    Applications are reviewed for completeness by the NIH Center for Scientific
    Review (CSR) and for responsiveness by NLM. Incomplete applications will
    be returned to applicant without further consideration.

    The CSR will not accept any application that is essentially the same as one
    already submitted. This does not preclude the submission of substantial
    revisions of applications already reviewed, but such applications must include
    an introduction addressing the previous critique.

    Applications that are complete and responsive to this program announcement
    will be evaluated for merit in accordance with the review criteria stated below.
    The Biomedical Library and Informatics Review Committee reviews accepted
    applications for merit and makes recommendations to the NLM Board of
    Regents, which provides a second-level of review. Applications recommended
    for support compete with other applications for available grant funds. The
    entire process, from receipt of the application to award decision, takes 9-12
Appendix I                        NLM PA 92-093                                    4
                             Information System Grants

    months.

    In assessing the merit of an application for an Information System grant,
    reviewers will pay close attention to the following elements:

            The need for the project in terms of expressed information
             needs of users or participating organizations
            Who will benefit from the project, the nature of the
             anticipated benefits, and how those benefits will be assessed
            Feasible objectives, milestones, and timeline for their
             achievement
            An appropriate plan for user training
            A plan to assess the outcomes of the project
            Strength of the rationale for the chosen technology or
             approach
            How results of the project will be disseminated to peers
            Plans for continued financial support of project after grant
             funding ends
            Plan for involving librarians and/or library-based resources in
             the project

    The qualifications and experience of key personnel, and the adequacy of the
    participating organizations' resources and environment to support project are
    also factors considered during review, as is the relationship to NLM's priorities
    and long term goals.


    Application Deadlines
    February 1, June 1, and October 1. (application receipt, review, and award
    cycles)


    Application & Process
    Applicants must use the PHS Form 398 grant application (revised May 2001).
    Applicants who use the 'fillable' PDF form can print, but not save, their
    completed proposals. Those who wish to save a copy should download the
    RTF version of the forms and open them with a preferred word processing
    program.

    Information System Grant applications use the non-modular budget format
    rather than the new modular budget format.

    The PHS 398 application form is also used for research grants, and the
    wording in some sections reflects that use. Applicants should not feel daunted
    by this. Information System grants are not research projects, and applications
Appendix I                         NLM PA 92-093                              5
                              Information System Grants

    will be evaluated in that spirit.

    See also the special instructions for completing NLM Information System
    Grant applications.


    Contact Information
    Dr. Valerie Florance
    Division of Extramural Programs
    National Library of Medicine
    Rockledge One, Suite 301
    6705 Rockledge Drive
    Bethesda, MD 20892
    Telephone: (301) 594-4882
    Fax: (301) 402-2952
    E-mail: floranv@mail.nlm.nih.gov


    Required Federal Citations
    http://www.nlm.nih.gov/ep/GrantFedCitations.html


    NIH Guide Document
    NIH Guide for Revised NLM Resource Grant Program (Information System
    Grant) PA 92-093
    http://grants2.nih.gov/grants/guide/pa-files/PA-92-093.html
       Helper Links for Information System Grants
            o Review Guide for Information System Grant
            o Special Instructions
            o Recent Information System Awards
            o Required Federal Citations
            o Reports Required
            o Resource Grants Comparison




       Grant Program Overviews
           o Research Support
           o Resource Support
           o Career Support
           o Training Support
           o Small Business R & D Support
Appendix I                NLM PA 92-093          6
                     Information System Grants

Last updated: 08 June 2004
First published: 24 November 2003

								
To top