High LOE of implementing a standalone
I1 credentialing service
I2 Development scope for Year 1 is too high
I3 QA testing has fallen far behind development
The lightweight approach to identity
verification, authentication and digital
I4 signatures was deemed unsatisfactory.
Our original proposal of leveraging the NES code base to
implement the credentialing service was rejected by the
ESST. The mandate to implement a standalone
credentialing service significantly increases the LOE. Rashmi Srinivasa
As the use cases and user stories for various parts of the
application get fleshed out, it has become clear that all of
the scope originally proposed for Year 1 will be impossible
to accomplish with the current team size. Rashmi Srinivasa
After the latest QA team member transition, QA has made
little to no progress on testing of the delivered
functionality. As of June 28, 5 iterations (24, 25, 26, 27
and 28) are untested by the QA team. Rashmi Srinivasa
The originally-proposed lightweight approach to identity
verification, authentication and digital signatures was
deemed unsatisfactory, and the LOE for multi-factor
authentication is much higher. Rashmi Srinivasa
Actions y s
Since there is no known user for the
credentialing service other than
FIREBIRD, the credential managemen
service implementation has been pushed
to Year 2. This has been approved by the
COTR and John Speakman. High Closed
We have proposed an alternative scope
and schedule which has been accepted by
the COTR, John Speakman and Dr. Sheila
Prindiville. High Closed
The new QA tester started in July 2011,
and we will work with him to ensure that
QA catches up with the last 5 untested
iterations, and starts on an automated
regression testing suite. High Closed
For the moment, we are proceeding with
the single-factor approach in the interest
of getting the rest of the application built.
The schedule will need to be revisited
once the scope for multi-factor
authentication becomes clearer. High Open
After clarifying requirements and understanding the system, it has
become clear that a full-featured FIREBIRD 3.0 application cannot be
delivered by this summer. Some of the issues that caused the revision
of schedule are:
1. We could not leverage any significant portions of the existing code
base from the old FIREBIRD. This was because of the old tech stack it
used, as well as the need to redesign and structure our application
front-end and the database/object model.
2. After requirements clarification, some features turned out to have
a higher LOE than originally estimated. E.g., investigator invitations
is not simply a matter of sending an email inviting an investigator to
register, but is more complex due to the interactions between the
statuses of an in progress registration packet and how the user is
interacting with that packet. The system can't merely send out an
email to an Investigator, it must manage what the Investigator says
they will do, handle what to do when an Investigator doesn't exist and
how they might be linked with this after coming into the system,
handle what happens when a user will not do things in our system at
3. BDA / Systems requirements for application installation changed,
causing us to spend a large chunk of time to rework the installation
set up due to changed requirements from the Systems team.
4. Integrating with the NES took more time than anticipated, in part
because of the need to understand the intricacies of the information
that needs to be passed to and received from the NES. This involved
not just reading the available documentation but also interacting with
the NES team to fully understand the complexity of the integration.
The documentation is out there, but it takes a reasonable amount of
research to find all the right information and the understand the
QA testing has now caught up with the latest iteration.
ID Risk Description
Credentialing work started earlier for COPPA may
bias the expectations for the FIREBIRD credentialing
1 COPPA credentialing bias service.
It is unclear who has the authority to approve
No clear authority to approve our requirements and design for digital signatures,
digital signatures (Part 11 including Part 11 compliance, which may delay
5 Compliance) design. approval and increase the risk of rejection.
Due to negative experiences with previous
incarnations of FIREBIRD, DCP may not fully
6 DCP reluctance in adoption embrace the project.
Poorly defined scope for digital The price proposal assumed a lightweight identify
verification requirement. If the requirement is more
signature services (including identify
verification) means that the LOE complex, the implementation may be more costly
8 may have to be revised later. than the contract can sustain.
The scope of the credentialing service is poorly
Poorly defined scope for defined at the moment. When it does become well
credentialing service means that LOE defined, the LOE may turn out to be higher than
11 may have to be revised later. originally expected.
As the use cases for various parts of the application
get fleshed out, it is becoming unlikely that all of the
Development scope for Year 1 may scope originally proposed for Year 1 will be possible
13 be too high to accomplish with the current team size.
Since FIREBIRD supports FDA-regulated transactions
and maintains FDA-regulated documentation, it is
beholden to 21 CFR Part 11 Compliance. Compliance
is typically demonstrated through independent
validation against Part 11 requirements, which is not
in the remit of the Firebird team and is not typically
the remit of the development team but rather an
15 No testing of Part 11 compliance independent QA body.
Leveraging the NES may limit performance and
3 NES Limitations functionality
If we're able to help OEWG implement some
Not aligning to CTEP OEWG goals mandates, we may be able to enlist their support;
may push FIREBIRD to the bottom of conversely, if we can't help, FIREBIRD may be
4 their priorities. pushed to the bottom of their priorities.
Currently, curation of persons and organizations
depends on the CTRO/CTRP contract and their
deployment timelines. If curation isn't available
when FIREBIRD is ready to deploy, CTEP and DCP
7 Dependence on CTRO curation won't use it.
CTEP-NES integration and presence Any delay in CTEP integration and use of NES for
of investigators and organizations in person/org management may impact CTEP's ability
9 NES may be delayed. to use Firebird.
Old incarnations of FIREBIRD developed a reputation
Poor legacy reputation may impact for being excessively complicated and difficult to use,
10 adotion which may impact adoption going forward.
Investigators and others users may not be available
to support the gathering and refinement of
requirements. Limited input from end users may
negatively impact the quality and usability of the end
12 Unavailability of users product.
The DCP has expressed a desire for all of the NCI to
DCP's desire to move to one NCI- move to one way of doing registrations. If they
wide process for registration may switch to doing annual registrations only, then our
make per-protocol registration per-protocol registration implementation will become
14 unnecessary wasted effort.
Several user stories that were not previously
recorded on the backlog have now been identified
after clarifications with the DCP. This means that the
backlog is quite bigger than it was at the time the
16 Previously-unrecorded requirements project schedule was determined.
There are duplicate organizations (same name but
different addresses) and misspelt organization names
in NES. This makes it difficult for users of a consumer
Some Person-Organization data in application like FIREBIRD to determine which person
17 NES is incorrect. or organization to select.
Increase to scope if CTEP IAM
18 integration is required Increase to scope if CTEP IAM integration is required
Investigator dissatisfaction and
increase to scope if Dorian to NCI Investigator dissatisfaction and increase to scope if
19 LDAP migration is required Dorian to NCI LDAP migration is required
Owner Impact Mitigation Plan Priority
Nobody Medium Define vision and get it reviewed by stakeholders. Low
Identify the stakeholder(s) with authority to approve
Nobody High requirements and design. Low
Demonstrate early on that FIREBIRD 3.0 is addressing
some the key issues that contributed to DCP's earlier
negative experience. Continue to demonstrate in each
subsequent demo that we continue to address their
David Loose High concerns. High
Clearly document our understanding of the vision for
this service based on the historical FIREBIRD
requirement discussions and current industry
practices. Identify the stakeholder(s) with authority to
approve requirements and design (Braulio Cabral,
George Komatsoulis and Bruce Woodcock - CIO).
Review our proposed approach with them, highlighting
David Loose High the impact of increasing the complexity. Low
Nobody High Get our vision and design reviewed by the ESST. High
Break down the high-level adoption priorities into
tasks, estimate the LOEs and compare to the team
Rashmi velocity to see if scope needs to be revisited.
Srinivasa High Reprioritize tasks instead of full adoption priorities. High
Hemant Undale Medium QA team responsible for Part 11 compliance testing. Low
Monitor the performance and functionality
improvements of NES as we approach our release.
Raise visibility of issues through demos and briefs.
Consider saving up updates and sending a batch
Eric Tavela High request to NES. High
Discuss with CTEP PIO Chief on possible ways for
Firebird to aide OEWG response. Due to divergent
timelines, we may only be able to assist with the
broader long-term goals of reduced study inititation
timelines, but the CTEP investigator registration
process is not currently a major delay to study
initiation due to a number of established release points
(i.e. ways to push through study activation without
David Loose Low waiting on PI registration) Low
We are looking to SAIC-F to manage this risk as this
Hemant Undale High has to do with the CTRO/CTRP contracts. Medium
We are looking to SAIC-F to manage this risk as this
Hemant Undale High has to do with the CTRP contracts. High
Engage with DCP in the first year to gain their
sponsorship; Hold regular stakeholder meetings to
solicit feedback on requirements and design, and
demo newly implemented functionality. Expand to
CTEP in the following year. Engage with outside/other
stakeholders throughout to keep them apprised of
David Loose Medium progress and demo new developments. Low
The team has begun regularly engaging with DCP, as
per the adoption plan for engaging stakeholders/users.
We have executive sponsorship at DCP in Dr. Leslie
Ford; but we also need to keep the operations people
happy. To that end, David and Tanya we plan to do
monthly updates at a minimum to keep the core
operations group engaged: Judy Smith and Margaret
David Loose High Schetrum. High
The likelihood of such a switch to annual registration is
low, especially in the short term. We are keeping DCP
engaged so that we are aware of any such
developments. Our team follows an Agile approach, so
changes in requirements can be handled relatively
David Loose Medium smoothly. Low
Rashmi Streamline process, prioritize backlog tightly,
Srinivasa Medium communicate risk. Medium
Display more information than merely the name so
Hemant Undale Medium that the user can select appropriately. Medium
Srinivasa High High
Jose Galvez Medium Medium
Braulio Cabral, ESST, and other stakeholders have reviewed
and accepted the vision for this service described in the
Closed Credentialing Service Vision Document.
Braulio Cabral and George Komatsoulis have been identified as
Closed the authorities.
We have set up regular monthly meetings with the DCP, and
are keeping them engaged. They have provided positive
The lightweight approach was deemed unsatisfactory, and the
scope of identity verification and authentication will be higher if
multi-factor authentication is desired. This increases LOEs, and
Closed this risk has turned into issue I4.
The ESST has reviewed and mandated a standalone
credentialing service that does not sit in the NES codebase.
Closed This increases LOEs, and this risk has turned into Issue I1.
Closed See Issue I2.
Closed 21 CFR Part 11 compliance testing is not in scope.
We can help the overall issue of study initiation, but CTEP has
already had to implement OEWG responses, and we're not
Open ready yet
We believe that adding Firebird entity curation (e.g. Persons &
Orgs) as a CTRO curation task will not result in a greater total
sum of work. Almost any entity requiring curation in Firebird
would ultimately need to be curated for CTRP purposes since all
CTEP/DCP protocols will be registered in the CTRP database,
including all PIs. Further, the vast majority of PIs (15000) are
already in CTEP's system and have been made available to the
CTRO/CTRP. DCP may result in the addition of a few hundred
additional PI person records over the entire lifecycle of Firebird,
but will largely rely on the already existing universe of Persons
in CTRP/CTEP-ESYS. Creating another curation body within the
NCI will require greater coordination both technically and
Open operationally between the CTRO and Firebird curation bodies.
CTEP-NES integration is expected to be done as part of the
Open CTRP contract, but this effort is currently stalled.
We have had several meetings with the DCP so far, and have
set up monthly meetings to keep them engaged. We will
engage CTEP more closely once we have implemented annual
We have had several meetings with the DCP so far, and have
set up monthly meetings to keep them engaged. DCP is
definitely interested in participating in UAT of the sponsor side,
and discussed finding a DCP PI to perform UAT of the
Open Investigator side.
The team has streamlined its process and has been tackling
more items in every iteration. We hope that we can still the
Open meet the schedule since our velocity is now greater.