Oftware Engineering Facilitates the Software Project Management Applications by zhy15740

VIEWS: 18 PAGES: 98

More Info
									                                        SPRUCE
                                      Kickoff Meeting
                                           04/22/2008




Rick Buskens (rbuskens@atl.lmco.com)                                     Douglas C. Schmidt
Patrick Lardieri (plardier@atl.lmco.com)                                 Jules White
Srini Srinivasan (ssriniva@atl.lmco.com)                                 Sean Eade
Lockheed Martin Advanced Technology Labs                                 Vanderbilt University

Allison Park (park_allison@bah.com)        Jonathan Preston              Spiros Mancoridis
Gene Blackburn                             Russell Kegley                Ali Shokoufandeh
Booz Allen Hamilton                        Rick Stone                    Jay Kothari
                                           Lockheed Martin Aeronautics   Dmitriy Bespalov
Blue = kickoff presenter                                                 Drexel University
                                                                                             1
Outline
   Overview
    –   Motivation
    –   Vision
    –   SPRUCE
    –   Background
    –   Key Challenges
    –   Program Overview

   Program Details
    –   Build
    –   Populate
    –   Educate
    –   Solicit
    –   Moderate
    –   Program Timeline

   Risks & Risk Mitigation

   Open Discussion
                              2
Outline
   Overview
    –   Motivation
    –   Vision
    –   SPRUCE
    –   Background
    –   Key Challenges
    –   Program Overview

   Program Details
    –   Build
    –   Populate
    –   Educate
    –   Solicit
    –   Moderate
    –   Program Timeline

   Risks & Risk Mitigation

   Open Discussion
                              3
  Motivation
     Ad hoc collaboration among SISPI community members 
      “valley of disappointment” for DoD programs
        – Unable to find SISPI technologies that meet needs and/or fail to adopt
          promising SISPI technologies
        – Software producibility problems encountered repeatedly across programs
        – Additional problem: “landing path” not typically DoD programs
                                                                                                                DoD acquisition
                                                                                                                  programs
                                                          New software research ideas

                                                                 Advanced
                                                                Engineering
                                                                   Tools

                         sponsors
 Software-Intensive                  Systems / Software                                 Small Scale,
Systems Producibility                 Engineering Tool                                   Abstract
  Initiative (SISPI)                 Research Programs                                  Prototypes


                                                                                                 yields
                                                                                                               Software
                                                                                                          Engineering Tools
                                                                                                          & Methods that Fail
                                                                                                             to Transition
DoD Acq. Programs   Tech. Deficiencies   Research Community
• Requirements                               (sponsors &                              “valley of
• CONOPS            Tech. Capabilities      researchers)                           disappointment”
                                                                                                                      4
Imagine A Future, Where…
   As a Program Engineer toiling away, putting out day-to-day fires
    – …you can quickly & easily discover & reach out to a broad community of
      people who can relate to your technical problems
    – …you can quickly & easily learn about technologies that may help solve
      your problem
    – …you can readily engage a community of experts to solve your problem
    – As icing on the cake, DoD program managers are watching this activity,
      to solicit your input in their next dream project

   As a SISPI researcher building exciting technologies & tools
    – …you had a repository of real-world problems to work with
    – …you can quickly & easily engage an active practitioner community
    – …you had a means to effectively demonstrate how your technologies &
      tools could solve real-world problems

   As a DoD Program Manager
    – …you had access to a continuously evolving virtual community of people,
      problems & technologies from both constituents above
    – …and that these constituents can contribute critical insights, data &
      artifacts to get your program justified, started & transitioned!
                                                                         5
Vision
   Methods & technologies that create a common meeting house
    for program engineers & technology researchers to discover
    joint software producibility interests & form collaborations
    – With testbed resources & tools specialized to particular operational
      domains (e.g., avionics)




   Benefit: challenge problem-driven, at-scale experiments can be
    defined & conducted to prove/disprove suitability of particular
    software producibility technologies
    – …which leads to a more systematic process for transitioning software
      producibility technologies                                             6
SPRUCE
   Systems & Software PRodUcibility Collaboration &
    Experimentation Environment (S2PRUCE2  SPRUCE)
   An open, collaborative research & development environment to
    demonstrate, evaluate & document the ability of novel tools,
    methods, techniques & run-time technologies to yield affordable
    & more predictable production of software intensive systems
    – Brings together researchers, developers and domain experts from
      different communities to de-fragment the knowledge necessary to
      achieve SISPI research, development and technology transition
   Capabilities
    –   Define challenge problems                              High quality
    –   Propose candidate solutions to challenge problems      artifacts are
    –   Define & conduct related experiments                   key!!
    –   Collaborate on all of the above
   Intended Impact
    – Increasingly successful transition across the software producibility
      spectrum & across operational domains
    – Well-connected, self-sustaining community of participants
                                                                               7
SPRUCE Community


          DoD research sponsors      DoD contractors




DoD program sponsors                        Academic researchers




              SISPI domain experts
                                     Tool vendors
                                                              8
Program Background
   AFRL sponsored the Software & Systems Test Track (SSTT)
    – SSTT Phase I (2006-2007): defined a CONOPS & System Architecture
    – SSTT Phase II (2008-2011): build & operate SPRUCE to realize the
      SSTT Phase II CONOPS




                                                                     9
Key Challenges
1. Bringing together communities of interest (COIs) across
   organizations & geographic locations
2. Overcoming competitive barriers to participation
3. Leveraging the latent capabilities in existing DoD transition
   organizations
4. Delivering & communicating value to stakeholders




      Challenge is not just to build & operate SPRUCE, but to
   attract a growing community of participants so that SPRUCE
   can have significant impact on DoD acquisition programs &
                   ultimately be self-sustaining


                                                                   10
SPRUCE Team
       Team Member                               Primary Role
Lockheed Martin Advanced      Systems integrator, outreach, community
Technology Labs               engagement
Booz Allen Hamilton           SPRUCE operations portal

Drexel University             SPRUCE affinity metrics

                              SPRUCE experiment infrastructure; host SPRUCE;
Vanderbilt University
                              validate SPRUCE CONOPS; testing/benchmarking

Lockheed Martin Aeronautics   Initial avionics-related challenge problems

                              Oversee broad direction of SPRUCE development &
Oversight Committee
                              evolution
                              Contributes content in early program stages,
                              identifies & establishes major challenge problem
Technical Advisory Board
                              areas, brings expertise, evangelizes SPRUCE within
                              respective organizations



               And most importantly…YOU!
                                                                              11
SPRUCE Program – At A Glance
   36-month program: April 2008 – March 2011

               Standup SPRUCE; validate, refine & realize CONOPS
 Build         Standup testbed infrastructure



                       Populate SPRUCE with challenge problems, solutions, experiments
Populate               Build initial SPRUCE user population


                                          Broaden SPRUCE awareness among
Educate                                    participant communities
                                             User conferences, marketing, …


                                          Increase SPRUCE adoption
 Solicit                                     Challenge problem events, affinity metrics,
                                              evangelism, …
                     SPRUCE Beta
                                                                          Sustain value
Moderate                                                                     Collect/post “value”
                                                                              metrics, maintain
                                                                              content freshness, …

                        Year 1                     Year                           Year 3         12
Outline
   Overview
    –   Motivation
    –   Vision
    –   SPRUCE
    –   Background
    –   Key Challenges
    –   Program Overview

   Program Details
    –   Build
    –   Populate
    –   Educate
    –   Solicit
    –   Moderate
    –   Program Timeline

   Risks & Risk Mitigation

   Open Discussion
                              13
SPRUCE Program – At A Glance
   36-month program: April 2008 – March 2011

               Standup SPRUCE; validate, refine & realize CONOPS
 Build         Standup testbed infrastructure



                       Populate SPRUCE with challenge problems, solutions, experiments
Populate               Build initial SPRUCE user population


                                          Broaden SPRUCE awareness among
Educate                                    participant communities
                                             User conferences, marketing, …


                                          Increase SPRUCE adoption
 Solicit                                     Challenge problem events, affinity metrics,
                                              evangelism, …


                                                                          Sustain value
Moderate                                                                     Collect/post “value”
                                                                              metrics, maintain
                                                                              content freshness, …

                        Year 1                     Year                           Year 3         14
SPRUCE Task: Build
   Objectives
    1. Standup SPRUCE; realize, refine & validate SPRUCE CONOPS
    2. Standup testbed experimental infrastructure

   Task elements
    –   B1. SPRUCE Functional Architecture
    –   B2. SPRUCE Users
    –   B3. Use Case Classification
    –   B4. SPRUCE Implementation Platform Overview
    –   B5. SPRUCE Implementation Architecture
    –   B6. SPRUCE Portal Taxonomy
    –   B7. Use Case Demonstrations
    –   B8. Build Plan
    –   B9. Deployment Schedule
    –   B10. Intellectual Property, DoD Classification & ITAR Issues




                                                                       15
    B1. SPRUCE Functional Architecture
     Hardware & software that enables                          Critical functional components that
     users to effectively use SPRUCE for                       allow challenge problems, candidate
     collaboration and for                                     solutions & experiments to be created,
     experimentation to test out                               edited, searched, executed, published
     hypotheses, reproduce challenge                           & collaborations to be performed
     problems, & test & evaluate
     candidate solutions to challenge                                                           Content Tagging      Communities
     problems                                                SPRUCE                              Management           of Interest
                                                            Experiment                                Content Tag    Management
                                                                                                      Repository        Communities
                                                           Infrastructure                                               of Interest Data
                            administers                                                                                 Repository

                                                                   manages
                    SPRUCE                                                     Candidate           Challenge         Tracking and
                 Infrastructure            Account        Experimentation      Solution            Problem             Metrics
                  Management              Management       Management         Management          Management         Management
                       Technology                                                  Candidate          Challenge         Tracking           Components
                                             Account           Experiment
                       Artifact                                                    Solution           Problem           Data
                       Repository
                                             Repository        Repository
                                                                                   Repository         Repository        Repository
                                                                                                                                           used for
                                                                                                                                           valuing
                       Collaboration                           Experimental                                             User-Provided      SPRUCE
                       Tools                                   Results                                                  Metrics
                       Repository                              Repository                                               Repository
                                                                                                                                           content &
                                                                                                                                           contributors,
                                                                                                                                           and for
                                                                                                     Collaboration
                                                                                                                                           building &
                                                                              Collaboration          History                               linking
                                                                              Management             Repository                            communities
                           administers                                                                                                     together

Administers both the
SPRUCE Operations
Infrastructure & the                                                 SPRUCE Operations
SPRUCE Experiment                                                       Infrastructure
Infrastructure &
manages user
accounts

                                                   Planned Program
                                                                                                                                             16
B2. SPRUCE Users

   SPRUCE community is comprised of numerous different classes
    of physical users, including
    –   DoD program sponsor
    –   DoD program manager
    –   DoD research sponsor
    –   DoD contractor (engineer or manager)
    –   Academic (SISPI) researcher
    –   Tool vendor / industry partner (for technology maturation)
    –   SISPI domain expert
    –   Administrator

   For convenience, formally distinguish the various roles played
    by members of the above physical user classes by introducing a
    set of logical actors




                                                                     17
B2. “Logical” SPRUCE Actors
                                                          A SPRUCE User that provides
                                                          software producibility challenge
                                                          problems. Typically also
                                           Challenge      interested in accepting (via
                         SPRUCE User        Problem       transition) solutions to the
                                            Provider      challenge problems.

 A registered user of SPRUCE. Will
 have a SPRUCE account that enables
 access to SPRUCE resources. May                          A SPRUCE User that provides
 represent a single physical user or a                    candidate solutions to posted
                                           Candidate      challenge problems.
 group of users.                            Solution
                                            Provider



                                                          A SPRUCE User that creates
                                                          and/or runs experiments and/or
                                          Experimenter    analyzes experimental results.

                 An individual
                 responsible for
  SPRUCE         administering the
Administrator    SPRUCE infrastructure.
                                                          SPRUCE Users that provides
                                                          expertise in one or more domains
                                                          of SPRUCE interaction.
                                          Collaborators
                                                                                             18
B3. Use Case Classification
   Over 100 use
    cases split
    into several
    categories

   SPRUCE
    program will
    implement ~80
    use cases




                              19
 B4. SPRUCE Implementation Platform Overview
    SPRUCE platform =
     Customized collaboration portal +
     Vanderbilt University‟s ISISlab (emulab)                  SPRUCE
                                                            Experimentation
                      SPRUCE                                   Testbed
                   Experimentation
                       Portal




                Define/conduct                Experiments




  SPRUCE                   Validate Results
Collaboration
    Portal
                                                                              20
B4. Initial SPRUCE Experimentation Testbed
                    Emulab infrastructure (www.emulab.net)
                     – Allocate testbed resources for research &
                       experimentation
                     – Configure allocated processors to run any
                       supported operating system
                     – Configure allocated network resources to
                       different network topologies & behavior
                    Current ISISlab infrastructure
                     – 4 IBM BladeCenters & 7 Cisco switches
                     – Per BladeCenter: 14 blades, 4 Gbps network
                       I/O modules; 1 mgmt module
                     – Per blade: two 2.8GHz Xeon CPUs, 1GB
                       RAM, 4GB HD, 4Gbps network interfaces
                    Multi-core equipment & software
                     – HP Proliant DL140 G3 dual quad-core Intel
                       Xeon E5320, 4GB RAM, 72GB HD
                     – Dell PowerEdge dual quad-core Xeon, 32GB
                       RAM, 2TB HD
                     – Intel Performance Primitives 5.1
                     – GFE‟d by AFRL
                                                             21
B5. SPRUCE Implementation Architecture
                                           data storage & retrieval


                             synchronous                             data
                               requests                           storage &
 Web       Web                                                     retrieval   Databases &
                                               Services
Browser   Portal         asynchronous
                                                                               File Systems
                          notifications




Private        generates                              configuration
Storage       relationship                             commands




                         configuration
                          commands
                                             Network
 Web      Testbed          & status         Accessible
Browser    Portal                          Experimental
                                             Testbeds

Remote
 Shell       commands & status

                                                                                       22
B5. Prototype SPRUCE Portal Interface
                                                                                        Collaborate on
         Identify &
                           Propose             Design &           Benchmark,              Challenge
          Evaluate
                          Candidate             Conduct         Validate, & Adopt         Problems,
         Challenge
                          Solutions           Experiments           Solutions            Solutions, &
         Problems     1               2                     3                   4        Experiments 5




     5


                                                                                    5


                                5
 1

 3                                                                                                1
 2                                        1                                                       3

                                                                                                  2 4

                                                                                                  5



                                                                                                      23
B6. SPRUCE Portal Taxonomy




                                                            2

   The SPRUCE Portal structure will flexibly evolve as new challenge problems &
    communities are formulated through collaborative interaction               24
      B7. Use Case Demo 1: Identify &
                                                                                     High Level Operational Scenarios
      Evaluate Challenge Problems
                                                                                     1 Identify & Evaluate Challenge Problems
Seamless Workflow, Preference-Based Awareness
                                                                                     2 Propose Candidate Solutions
SPRUCE will provide seamless challenge problem
workflow by enabling DoD Program Offices &                                           3 Design & Conduct Experiments
Systems Integrators to submit challenge problems                                     4 Benchmark, Validate, & Adopt Solutions
which will trigger automated notifications to
interested participants based on their preferences.                                  5 Collaborate on Challenge Problems,
                                                                                        Solutions, & Experiments


Workflow Thread                  Login to SPRUCE            Create Challenge Problem               Edit Challenge Problem

                                                                Problem Provider
                                                                 Posts/Submits                         Problem Provider
      Problem Provider            Problem Provider
                                                               Challenge Problem                       Modifies Challenge
     Identifies Problem          Logs on to SPRUCE
                                                                to the Challenge                            Problem
                                                               Problem Repository



                                                                                 Parallel Activities

             Register to be                                  SPRUCE Technical
                                                                                                   Problem Provider
             Notified of New                                   Advisory Board
                                 Interested Parties                                                 Decides to Add
             Challenge Problem                                Evaluates/Rates
                                    are Notified                                                   Additional Artifacts
                                                            Challenge Problem, &
                                                                                                     or Description
                                                             Provides Feedback

                                  Parallel Activities       Endorse Challenge Problem



                                                                                                                            Video Clip
       Interested Parties         Interested Parties
       Design & Conduct                                     Interested Parties
                                 Propose Candidate                                                                          Screen Shot
          Experiments                                          Collaborate
                                       Solutions
                            2                           3                        5                                          Email Notification
                                                                                                                                     25
        B7. Use Case Demo 2:                                                 High Level Operational Scenarios
        Propose Candidate Solutions
                                                                            1 Identify & Evaluate Challenge Problems
Seamless Workflow, Preference-Based Awareness
                                                                            2 Propose Candidate Solutions
SPRUCE will provide seamless candidate solution
workflow by enabling various Solution Providers                             3 Design & Conduct Experiments
to submit candidate solutions which will trigger                            4 Benchmark, Validate, & Adopt Solutions
automated notifications to the associated
challenge problem provider & other interested                               5 Collaborate on Challenge Problems,
                                                                                  Solutions, & Experiments
participants based on their preferences.

Workflow Thread                                   Propose Candidate Solution to         Edit Candidate Solution to
                           Login to SPRUCE        Challenge Problem                     Challenge Problem
                                                     Solution Provider
   Solution Provider        Solution Provider        Posts/Submits a                       Solution Provider
      Identifies a         Logs on to SPRUCE       Proposed Solution to                   Modifies Challenge
  Candidate Solution                                  the Candidate                            Problem
                                                   Solution Repository



                                                                                           Solution Provider
     Register to be
                                                                                            Decides to Add
     Notified of New
                            Interested Parties                                            Additional Artifacts
     Candidate Solution
                               are Notified                                                 or Description


                            Parallel Activities


                           Challenge Problem
  Interested Parties            Providers
  Design & Conduct                                Interested Parties
                              Benchmark,
     Experiments                                     Collaborate
                            Validate, & Adopt
                       3        Solutions     4                        5                                             Email Notification
                                                                                                                             26
      B7. Use Case Demo 3:                                                      High Level Operational Scenarios
      Design & Conduct Experiments                                             1 Identify & Evaluate Challenge Problems
Secure & Flexible Resource Allocation & Release
                                                                                2 Propose Candidate Solutions
The SPRUCE Experimental Environment will allow
participants to attain dedicated resources for the                             3 Design & Conduct Experiments
duration of their experiments. The Operations                                  4 Benchmark, Validate, & Adopt Solutions
environment will allow notification & sharing of
experimental information, results, & evidence.                                 5 Collaborate on Challenge Problems,
                                                                                       Solutions, & Experiments


Workflow Thread
                          Login to SPRUCE             Create Experiment                 Update Experiment
    Experimenter
                                                        Experimenter
       receives
                            Experimenter Logs          Posts/Submits                         Experimenter
    notification of
                             on to SPRUCE             Experiment to the                        Modifies
  Challenge Problem
                                                         Experiment                           Experiment
    or Candidate
                                                         Repository
  Solution Proposal
                                                                 Parallel Activities

                                                                                           Interested Parties       Interested Parties
                                                                                              are Notified             Collaborate
     Experimenter            Experimenter &
      configures              conducts the                                                                                                5
      experiment               experiment                                              Register to be Notified of
                                                                                       New Experiment
 Upload Tool/Technology     Begin Experiment
                                                                                Experimenter posts
                                                   Experimenter &
                                                                                   experimental
                                                    Collaborators
      Experimenter                                                            information, evidence,
                                                   Analyze Results
        releases                                                               & results to SPRUCE
      experimental                                                                                                   Video Clip
       resources
                                                Analyze Experimental          Post Experimental Results
                                                Results                       Post Tool/Technology                   Email Notification
    End Experiment
                                                                                                                             27
      B7. Use Case Demo 4: Benchmark,                                        High Level Operational Scenarios
      Validate & Adopt Solutions                                             1 Identify & Evaluate Challenge Problems
Highlight Technology Transitions Successes
                                                                             2 Propose Candidate Solutions
DoD Programs can choose to indicate their intent to
adopt candidate solutions which will highlight                               3 Design & Conduct Experiments
SPRUCE successes & make other participants aware                             4 Benchmark, Validate, & Adopt Solutions
of significant software producibility advances.
                                                                             5 Collaborate on Challenge Problems,
                                                                                Solutions, & Experiments


 Workflow Thread
                                                        Challenge Problem
                                 Challenge Problem
    Challenge Problem                                     Provider Posts
                               Provider Reproduces**,                                  All Interested
   Provider is notified of                                 Assessment
                                Validates & Assesses                                 Parties are Notified
   a Proposed Solution                                    Metrics to the
                               the Proposed Solution
                                                         Operations Portal

  Register to be Notified of
  New Candidate Solution




                                Challenge Problem
                                 Provider Indicates           Solution is Adopted
                                Intent to Adopt the           into DoD Program
                                      Solution

                                                                                                             Video Clip

                                                                                                            Screen Shot

**Reproducing the Solution can be done in SPRUCE or in their own environment                                Email Notification
                                                                                                                    28
   B7. Use Case Demo 5: Collaborate                                               High Level Operational Scenarios
   on Challenge Problems, Solutions                                               1 Identify & Evaluate Challenge Problems
   and Experiments                                                                2 Propose Candidate Solutions

Intelligent & Integrated Collaboration                                            3 Design & Conduct Experiments

The SPRUCE collaboration capabilities will                                        4 Benchmark, Validate, & Adopt Solutions
increase communication & interaction between                                      5 Collaborate on Challenge Problems,
DoD, industry, & academic participants.                                               Solutions, & Experiments


Workflow Thread
                                User Browses                                                                        Users decide they
                                                                                   User interacts with
     SPRUCE                   through Challenge           User Identifies                                              would like a
                                                                                 Identified Collaborator
   User Logs into                 Problems,             another User Having                                        Collaboration Space
                                                                                  via chat, discussion
  Operations Portal             Experiments, &           Common Interests                                               for future
                                                                                board, email, skype, etc.
                                   Solutions                                                                         Collaborations
  Login to SPRUCE              Browse/Search                                    Create Collaboration Item



                                                             Designated
   Users Request                   SPRUCE                    Workspace
    Collaboration                Administrator           Moderator Invites
     Workspace                Creates Workspace         other Collaborators
                                                                                                     Collaborators Post
                                                          into Workspace
                                                                                                     Collaboration Items
Request Invitation to Collaborate                     Invite Collaborator                             to the Workspace
                                           Accept Request to Collaborate
                                                                                                   Create Collaboration Item
    Collaborator                                         Moderator grants
  Requests Access                                         Collaborator to
   to Workspace                                         Access Workspace
 (Email is sent to the                                  (Email is sent to the
     Moderator)                                            Collaborator)                                                   Video Clip

                                                                                                                           Email Notification
                                                                                                                                   29
B8. SPRUCE Build Plan
       SPRUCE CONOPS & Design                           SPRUCE Deployment Architecture
    Functional Architecture
    •   Candidate Solution, Challenge Problem           Physical Infrastructure
        Management, Challenge Problem, Experiment,      Consolidated Interface
        Collaboration, Account, & infrastructure        •   Web-based portal interface to provide a
        Management                                          consolidated view of SPRUCE information,
    •   Experimental & Operations Infrastructure            tools, & events, & user activity
    Operational Scenarios                               Taxonomy
    •   Identify & Evaluate Challenge Problems          •   Reflects the functional architecture
    •   Propose Candidate Solutions                         components
    •   Design & Conduct Experiments                    •   Accommodates & encourages ad hoc
    •   Benchmark, Validate, & Adopt Solutions              collaboration
    •   Collaborate on Challenge Problems, Solutions,   Data & Access Control Model
        & Experiments                                   •   Data describing users, challenge problems,
    User Roles                                              candidate solutions
    •   Challenge Problem Provider                      •   e.g. Problem Type, Domain, Characteristics
    •   Candidate Solution Provider                     •   Permissions to access areas of the Portal
    •   Experimenter                                    Workflows
    •   Collaborator                                    •   Realize the processes to facilitate
    •   Administrator                                       Operational Scenarios
                                                        •   Modularized to flexibly accommodate
                                                            alternate & iterative flows


    SPRUCE participant activities & interactions are envisioned to be parallel,
     iterative, & complex. The proposed SPRUCE environments will seek an
     organizing construct to accommodate variations of activity sequences &
     collaborations by providing modular workflows & ad hoc collaboration spaces
                                                                                                   30
B9. SPRUCE Deployment Schedule




                                 31
B10. Design Issues: Intellectual Property
   Challenge Problems, Candidate Solutions, Experimentation
    – SPRUCE policy: provide generalized challenge problems, omit
      proprietary details where possible, remove all software on testbed
      resources at experiment completion
        • Omit proprietary details – protects IP; helps ensure focus not just short-term
        • Inspired by Open Experimental Platforms (OEPs) on DARPA MoBIES, PCES,
          NEST, ARMS programs & by successful emulab service

   Candidate solution providers can associate licensing conditions
    with their solutions
    – e.g., none, GPL, BSD, Apache, Mozilla, custom, …
    – Users of candidate solutions must agree to licensing conditions and/or
      must engage with the solution providers before the technologies can be
      used




                                                                                   32
B10. Design Issues: DoD Classification
   SPRUCE designed to operate at UNCLASSIFIED level
    – Software producibility challenges typically orthogonal to operational
      requirements
    – Challenge problems & candidate solutions will, or can be sanitized to be,
      UNCLASSIFIED, while still preserving core challenges

   Software producibility technologies are nearly exclusively
    UNCLASSIFIED
    – Ideally, wish to ultimately transition these technologies to COTS vendors

   Although the same procedures can be used to “protect” secure
    content, recommend creating a separate instance with stronger
    access restrictions, explicitly allowing SPRUCE to operate at a
    SECRET level
    – Stronger authentication, logging, product restrictions, etc.

   SPRUCE is not currently designed to operate at multiple levels
    of security


                                                                            33
B10. Design Issues: ITAR
   SPRUCE designed to operate with non ITAR-restricted
    technologies
    – Software producibility challenges & solutions typically not ITAR restricted
      unless tightly coupled to ITAR-restricted computing/information
      technology
   SPRUCE has capabilities to tag portal content as “protected”
    – When posting, users can upload to an ITAR restricted repository
    – SPRUCE User profile includes user pedigree (e.g., email ID, citizenship)
        • Some elements of the pedigree need authorization (e.g., citizenship)
    – All SPRUCE Users can browse “unrestricted” content
    – Must sign-in with matching credentials to access restricted areas of portal
   Additional manual process to support ITAR restricted content
    – Verification of U.S. persons identity prior to receiving SPRUCE account
    – Strong labeling & logging of ITAR-restricted technologies
    – Training & explicit user acceptance of responsibility to protect ITAR
      restricted data
   If ITAR required, may need to deploy SPRUCE in closed
    environment
                                                                                 34
SPRUCE Program – At A Glance
   36-month program: April 2008 – March 2011

               Standup SPRUCE; validate, refine & realize CONOPS
 Build         Standup testbed infrastructure



                       Populate SPRUCE with challenge problems, solutions, experiments
Populate               Build initial SPRUCE user population


                                          Broaden SPRUCE awareness among
Educate                                    participant communities
                                             User conferences, marketing, …


                                          Increase SPRUCE adoption
 Solicit                                     Challenge problem events, affinity metrics,
                                              evangelism, …


                                                                          Sustain value
Moderate                                                                     Collect/post “value”
                                                                              metrics, maintain
                                                                              content freshness, …

                        Year 1                     Year                           Year 3         35
SPRUCE Task: Populate
   Objectives
    1. Populate SPRUCE with challenge problems, solutions, experiments
    2. Validate SPRUCE with initial core users & use-cases

   Task elements
    –   P1. Challenge problem: definition
    –   P2. Challenge problems: importance to SPRUCE success
    –   P3. Challenge problem acquisition plan
    –   P4. Challenge problem lifecycle
    –   P5. Seed challenge problems
    –   P6. Challenge problem evaluation criteria
    –   P7. Evaluation of seed challenge problems
    –   P8. Candidate solutions
    –   P9. Experiments




                                                                         36
    P1. Challenge Problem: Definition
   A description of a software producibility deficiency that negatively
    impacts quality, capability, cost or schedule of software intensive
    DoD development programs.
    – e.g., assuring no race conditions in multi-threaded C++ or Java source code

   Organized into broad areas that contain one or more specific
    challenge problem instances
    – e.g., the problem described above could be a specific instance in the broad
      area of assuring non-functional run-time properties
                                                                                                 Types of Challenge Problems
   Challenge problem classification
    – Type of R&D activity needed




                                                  Type of R&D Activity Needed
        • Basic – focuses on fundamental                                                   Manhattan




                                                                                Basic
                                                                                                                Transformational
          scientific or technology gap                                                     Project Like            Research
        • Applied – focuses on incremental                                                  Research
          capability gap or performance
          improvement beyond state of practice
                                                                                            Experiment             Advanced
    – Transition timeframe needed                                                Applied      Ready               Technology
        • Current generation programs –                                                      Solution            Development
          DoD acquisitions over next 1-5 years
        • Next generation programs –                                                       Current Generation      Next Generation
          DoD acquisitions over next 5-10 years                                             Need for Convincing Demonstration
                                                                                                  of Technology Maturity 37
P2. Importance to SPRUCE Success
   Challenge problems are the             Example: ATL QoS website
                                            Dedicated to the challenge of building distributed real-
    “killer content” that transforms         time embedded (DRE) systems from open QoS-enabled
                                             middleware, operating systems & networks
    SPRUCE from a web portal into           Has attracted a participant community whose
                                             collaborations have driven middleware technology
    a virtual community by:                  development
                                               –   SPOs: DDG-1000, Aegis
    1. Providing a conceptual                  –   Contractors: Boeing, Raytheon, Lockheed Martin, etc.
                                               –   Universities: Vanderbilt, Wash U, CMU, Drexel, …
       foundation that attracts the            –   Commercial vendors: OIS, ZeroC, PrismTech, etc.
       primary groups – SPOs,
       contractors, researchers, &         Example: Boeing Open Experimental Platform (OEP)
                                            DARPA PCES program
       commercial vendors – in              Provided dozens of avionics system scenarios &
                                             software artifacts
       successful SISPI research            Drove evaluation of program composition technologies
       collaborations                       Ultimately established empirical evidence that enabled
                                             transition of ESCHER modeling tools onto FCS
    2. Providing real-world context,
       constraints & metrics to evaluate   Example: CMU‟s Fluid Java code analysis tools
                                            Lockheed Martin‟s Software Technology Initiative (STI)
       SISPI research progress               matched CMU‟s mature Fluid Java code analysis
                                             technology to the U.S. Navy‟s Aegis Open Architecture
    3. Providing well-defined transition     program & DDG-1000 programs
                                            Helped to establish business case for SureLogic startup
       & commercial tool opportunities       & submission of congressional plus-up proposal
       for mature SISPI technology with      between LM MS2-Moorestown, PMS 500, & SureLogic
                                             to apply SureLogic code analyers to DDG-1000 R5
       proven results                        software QA

                                                                                                    38
         P3. Challenge Problem Acquisition Plan
                     seed                                  sponsor                sustain
                6

                5                                                     Community Offered
New CP / Year




                                                                         (projected)
                4
                                            SPRUCE
                3                          Encouraged
                2
                            SPRUCE Funded
                1   Seed challenge problems serve as exemplars for
                    defining a broader range of challenge problems



                    Year 1              Year 2               Year 3     Year 4    Year 5+
                                                                                     39
P4. Challenge Problem Lifecycle

                 Creation                                Refinement
Challenge
 Problem
 Provider




                               Evaluation
 SPRUCE
 Advisors




                                   Evaluation                      Extension

 Collaborators


                                                                               time
                            Mandatory Element   Optional Element
                                                                                40
P4. Challenge Problem Lifecycle
                                                 Year 1 Challenge Problem
                 Creation                              Creation Plan
                                                        Refinement
Challenge
 Problem                                        Kickoff: 5 Seed Challenge
 Provider                                       Problems Selected
                                                Month 3: 1st Challenge
                                                Problem Definition Complete
                               Evaluation
                                                Month 6: 2nd & 3rd Challenge
 SPRUCE                                         Problem Definition Complete
 Advisors
                                                Month 9: 4th & 5th Challenge
                                                Problem Definition Complete

                                   Evaluation                       Extension

 Collaborators


                                                                                time
                            Mandatory Element    Optional Element
                                                                                 41
P5. Challenge Problem Area
   Predicting Computer System Capacity Needs
    – A modern computing architecture consists processors, memory, bus
      controllers, bridges, arbiters, DMA controllers and other components.
      The total behavior of the system is the interdependency of the processor
      configuration, the software load & the message load on the system.
      There is a need for a predictive model that incorporates these elements
      into an environment suitable for trade studies & processor system
      specification.
   Seed Challenge Problem Instances
    1. Predicting Performance of Avionics Mission                  Manhattan




                                                       Basic
                                                                                       Transformational
       Systems Using a Logical Task Network &                     Project Like
                                                                   Research
                                                                                          Research


       Computing Plant Description                                 Experiment            Advanced




                                                        Applied
        •   Sponsored by Lockheed Martin Aeronautics                 Ready
                                                                    Solution
                                                                                        Technology
                                                                                        Development

        •   Seeking Experiment Ready Solution for                 Current Generation      Next Generation

            F-35 & F-22 programs
    2. Optimization Techniques for Deployment of
       Avionics Mission Systems onto a Specific                    Manhattan




                                                       Basic
                                                                                       Transformational
                                                                  Project Like            Research
                                                                   Research
       Computing Plant
        •   Sponsored by Lockheed Martin Aeronautics               Experiment            Advanced

                                                        Applied
                                                                     Ready              Technology
        •   Seeking Experiment Ready Solution for                   Solution            Development


            F-35 & F-22 programs                                  Current Generation      Next Generation

                                                                                                            42
P5. Challenge Problem Area
   Optimally Configuring & Deploying Software on Multi-Core
    Processors
    – Ad hoc deployment of legacy software on multi-core processors can
      decrease performance because of unexpected behavior of cache
      architectures. Technology to optimize the performance of legacy
      software on such architectures is required to cope with shift in
      commercial computing architecture to multi-core.

   Seed Challenge Problem Instance
    3. Optimization Techniques for Deployment of
                                                                 Manhattan




                                                     Basic
                                                                                     Transformational
       Binary Software Systems onto Multi-Core                  Project Like
                                                                 Research
                                                                                        Research

       Platforms
        •   Potential Sponsors Raytheon & Lockheed               Experiment            Advanced




                                                      Applied
                                                                   Ready              Technology
                                                                  Solution            Development
            Martin MS2
        •   Seeking Experiment Ready Solutions for              Current Generation      Next Generation




            DDG-1000 Program and Aegis Program




                                                                                                          43
P5. Challenge Problem Area
   Analyzing & Predicting the Evolution of Software Systems
    – Software development productivity is a complex function on personnel,
      team organization, problem complexity, cost, schedule, and other
      factors. Automated techniques to analyze past software development
      projects and generate accurate productivity metrics and multivariate
      estimation techniques to accurately predict software development paths
      are required to cope with increasingly complex software intensive
      systems.

   Seed Challenge Problem Instances
                                                                 Manhattan
    4. Estimation of Software Development




                                                     Basic
                                                                                     Transformational
                                                                Project Like            Research
                                                                 Research
       Efficiency Metrics Across Software Releases
        •   Sponsored by Drexel University                       Experiment            Advanced




                                                      Applied
                                                                   Ready              Technology

        •   Advanced Technology Development for                   Solution            Development


            Software Project Cost Estimation                    Current Generation      Next Generation




    5. Predicting Evolutions Away from Software
       Development Plans                                         Manhattan




                                                     Basic
                                                                                     Transformational
                                                                Project Like
        •   Sponsored by Drexel University                       Research
                                                                                        Research


        •   Advanced Technology Development for                  Experiment            Advanced
            Software Project Planning & Management    Applied
                                                                   Ready              Technology
                                                                  Solution            Development

                                                                Current Generation      Next Generation

                                                                                                          44
 P5. Example Challenge Problem (Simplified)
                               Optimization Techniques for Deployment of Binary
Name
                                Software Systems onto Multi-Core Platforms
                               How do we take legacy binary code and map it effectively
Problem Description             to a multi-core platform?
                                 – Preserve correct functional behavior
(Overview)
                                 – Non-functional behavior must be consistent with requirements
                                 – Ad hoc migration yields worse performance than for single core
Domain of Applicability        Multi-core analysis & optimization
                               The inability to move legacy code to multi-core
Expected Impact                 architectures quickly and efficiently will cripple the DoD.
                                Manual migration approaches are costly and error prone.
Sponsors                       Lockheed Martin MS2 (potential), Raytheon (potential)
                               Test code: TestDegradedPerformance.c
                                   –Two threads + mainline; equal workload per thread
                                   –False sharing
Problem Demonstration          Experimental platform
Configuration & Artifacts          –Intel Xeon single core 1.8GHz & dual quad-core 1.6-1.8GHz
                                   –GNU GCC 4.1.2
                                   –Linux Debian 32-bit 2.6.18 kernel
                                   –Default operating system scheduler used
                               28% slowdown when run on dual quad-core platform
Observed Results
                                versus single core platform
                                                                                              45
P4. Challenge Problem Lifecycle (Revisited)
                                                 Year 1 Challenge Problem
                 Creation                                Refinement
                                                      Evaluation Plan
Challenge
 Problem                                        Kickoff: 5 Seed Challenge
 Provider
                                                Problems Evaluated
                                                Month 3: 1st Challenge
                                                Problem Evaluation Updated
                               Evaluation
                                                Month 6: 2nd & 3rd Challenge
 SPRUCE
 Advisors                                       Problem Evaluation Updated
                                                Month 9: 4th & 5th Challenge
                                                Problem Evaluation Updated

                                   Evaluation   Month 10: External Community
                                                                Extension
                                                Evaluation of ALL Challenge
 Collaborators                                  Problems

                                                                        time
                            Mandatory Element     Optional Element

                                                                            46
P6. Challenge Problem Evaluation Criteria
  Criterion           Primary Consideration                           Other Considerations
                  High return on investment when          Well supported across users
                   measured on one or more of:               – Applies to multiple programs
Has High            – Improved warfighting efficiency        – Applies to multiple agencies
Impact              – Risk mitigation                        – Encountered by multiple contractors
                    – Cost savings                         Benefit is quantifiable
                  Problem can be bounded                  Representative challenge problem can be
                                                            established
Is Well                                                    Sample artifacts (e.g., code, models) are made
Defined                                                     available
                                                           A means to quantify success exists

Seeks             Seeks disruptive, not just              Medium- to long-term horizon
                   evolutionary, changes in approach         – Commercial community will find short-term
Disruptive          – Change in “what” is done, not            solutions
Approach               necessarily “how” it is done
                  Encourages & leverages expertise        Expertise needed to develop a solution spans
Is Multi-          from multiple domains                    technologies
Disciplinary                                               Anticipated solution complexity managed by
                                                            collaboration, integration
                  Potential for general solution          Supporting tools exist or technology ecosystem
                   approach exists                          can be developed
                    – Initial implementation may be        Offers clear benefits to participants in the
Can Be                 realized for a specific              ecosystem – i.e., adds value
Generalized            environment                         Reduced barriers to participation with open &
                                                            vibrant community, quick-start toolkits &
                                                            examples

  Above criteria permit measure of challenge problem “quality”
                                                                                                      47
  P7. Evaluation of Seed Challenge Problems
      Evaluation
        – Rate against each criterion on scale of 1-10 (1=low, 5=med,10=high)
                                                         Evaluation Criteria
                                                                                                Normalized
                                 Has High   Is Well     Seeks        Is Multi-      Can Be
      Challenge Problem           Impact    Defined   Disruption   Disciplinary   Generalized
                                                                                                 Average
                                                                                                  Score
1. Predicting Performance of
Avionics Mission Systems
Using a Logical Task Network &    MED       HIGH        LOW           MED           HIGH          0.62
Computing Plant Description
2. Optimization Techniques for
Deployment of Avionics
Mission Systems onto a            MED       HIGH        LOW           MED           HIGH          0.62
Specific Computing Plant
3. Optimization Techniques for
Deployment of Binary Software
Systems onto Multi-Core           HIGH      HIGH        HIGH         HIGH           MED           0.90
Platforms
4. Estimation of Software
Development Efficiency Metrics    HIGH       MED        HIGH          MED           HIGH          0.80
Across Software Releases

5. Predicting Evolutions Away
from Software Development         HIGH      LOW         HIGH          MED           HIGH          0.72
Plans


                                                                                                   48
P8. Candidate Solution
   Describes a potential solution to a challenge problem
    – May or may not include a realization of the solution (if a tool or
      technology)

   Solution technologies in SPRUCE consist of:
    – Name
    – Description / capability - high-level view of how the solution works
    – Type of solution – e.g., process, tool, technology
    – Domain of applicability – what‟s in scope, out of scope
    – Licensing conditions – terms for licensing the technology, if any
    – Development platform – important if tool/technology is to be extended
      by other SPRUCE users
    – Run-time platform – helps determine requirements for tool usage in
      actual program environments
    – Associated challenge problems – which challenge problem(s) the
      solution is intended to address
    – Associated solution technologies – related solution technologies



                                                                           49
P8. Candidate Solution: Example (Simplified)
                              Temporal Execution Graph Behavior Collection &
Name
                               Visualization Tools
                              Tools for automatically collecting & visualizing
                               “nominal” execution behavior of legacy applications
Description / Capability        – Collects data by inserting event probes around function calls
(Overview)                      – Upon triggering, each probe emits an event that defines the
                                  thread making the call, call site address, function target address,
                                  cycle count & timestamp
                                – Provides benchmark for acceptable system performance
Type of Solution              Software Tool
Domain of Applicability       Multi-core analysis & optimization
Licensing Conditions          Government rights
                            Linux Debian 32-bit 2.6.18 kernel
Development Platform
                            GNU GCC 4.1.2

                            Behavior Collection Tool: Intel x86 platform
Run-time Platform
                            Visualization Tool: Any (Java); uses JFreeChart libraries

Associated Challenge          Optimization Techniques for Deployment of Binary
Problems                       Software Systems onto Multi-Core Platforms
Associated Solution         Perseus multi-core migration tools
Technologies                TMAM time-based memory access map behavior
                             collection & analysis tools
                                                                                                50
P9. Experiments
   Used to evaluate the suitability/benefits of a particular solution
    technology applied to a specific challenge problem
    – Associated with challenge problems OR solution technologies
    – Developed by any SPRUCE user
   Experiments in SPRUCE consist of:
    – Objectives – summary of specific objectives of the experiment
    – Associated challenge problems – which challenge problems the
      experiment is intended to evaluate against
    – Associated solution technologies – which solution technologies are
      suitable for running the experiments
        • Experiments may be used for benchmarking
    – Experimental configuration & artifacts – so that the experiment can be
      reliably reproduced
    – Benchmarks – baseline to measure results against
    – Expected results – expectations of results to be produced
   To demonstrate suitability of a particular solution technology to
    a specific challenge problem
    – Must store results of executed experiments in SPRUCE
    – Must provide details on how to analyze/interpret experimental results
                                                                              51
P9. Experiments: Example (Simplified)

                           Test ability for optimizer to collect threads onto a
 Objective                  single processor & modulate core for main thread
                            thereby conserving power
 Associated Challenge      Optimization Techniques for Deployment of Binary
 Problem(s)                 Software Systems onto Multi-Core Platforms
 Applicable Solution
                           Perseus
 Technologies
                           Test code: FloatThreads.exe (FloatThreads.c)
                             – Four counting threads + mainline
                             – Equal workload per thread
 Experimental                – No false sharing
 Configuration &           Experimental platform
 Artifacts                   – Intel Xeon dual quad-core 1.6-1.8GHz
                             – GNU GCC 4.1.2
                             – Linux Debian 32-bit 2.6.18 kernel (PAPI modified)
                           Optimizer flags set to optimize power consumption
 Benchmarks                Power measurement without optimizer
                         20% power savings
 Expected Results
                         Negligible slowdown



                                                                                   52
SPRUCE Program – At A Glance
   36-month program: April 2008 – March 2011

               Standup SPRUCE; validate, refine & realize CONOPS
 Build         Standup testbed infrastructure



                       Populate SPRUCE with challenge problems, solutions, experiments
Populate               Build initial SPRUCE user population


                                          Broaden SPRUCE awareness among
Educate                                    participant communities
                                             User conferences, marketing, …


                                          Increase SPRUCE adoption
 Solicit                                     Challenge problem events, affinity metrics,
                                              evangelism, …


                                                                          Sustain value
Moderate                                                                     Collect/post “value”
                                                                              metrics, maintain
                                                                              content freshness, …

                        Year 1                     Year                           Year 3         53
Community Building

   Community building is crucial to the long-term success of a self-
    sustaining SPRUCE
    – Grow SPRUCE community – new participants, new challenge problem
      areas & challenge problems
    – Leverage latent capabilities in existing DoD transition organizations

   SPRUCE program Educate & Solicit tasks directly target
    community building (will discuss soon)

   First, will review context of opportunity through a case study,
    then detail our community building approach to fulfill the need
    – Innovative technology & tools required to assist in community building




                                                                           54
A Case Study in DoD Need
   Various initiatives within DoD have recognized the value of
    collaboration during program formation
    – e.g., SBIR offices, industry workshops, DACS, RIAC, …
    – That‟s the whole point of requesting white papers!

   Many do not have resources to thoroughly consider all
    implications & implement a comprehensive solution
    – e.g., industry workshops for program solicitations (e.g., BAA‟s) suffer
      from venue, schedule & audience constraints

   SPRUCE will allow us to reach out to these initiatives
    – The ease & extent of integration will depend on program scale & the
      maturity of SPRUCE itself

   Will illustrate this with a mini-case study of an Army SBIR
    seeking collaboration to define a program
    – …and discuss our proposed approach to address this problem



                                                                                55
Army SBIR Topic Collaboration Case Study (1)

                           http://www.armysbir.com/topic/concept.aspx
                         Can enter new topics or comment on existing
                         topics, that have been populated by SBIR
                         Program Managers. Great idea!!




                                   Let‟s explore collaborating on this
                                   “Exploiting Parallelism” topic,
                                   which is of interest to one of our
                                   team members.




                                                                  56
Army SBIR Topic Collaboration Case Study (2)




                  This portal has been built & re-designed again. First
                  allowed the user to only enter unstructured comment.
                  Now has been modified to accept more metadata.
                  Practice makes perfect!
                  ------------------------------------------------
                  Still far away from a „wikipedia-style‟ community
                  participation & „lively‟ feedback & exchange.



                                                                          57
Lessons from the Mini-Case Study
   There is a perceived value & need within the DoD for
    collaboration around challenge problems
    – So much so that they are willing to spend their own precious resources to
      custom-implement their own collaboration patterns

   Each of these initiatives do not have the luxury to set aside
    sufficient resources to implement a fully-considered, scalable, &
    „pluggable‟ collaboration solution

   SPRUCE offers a potential to fill this void – within BAAs, SBIR
    processes, within partner sites & as a stand-alone portal
    – Problem providers & other users can provide comments, upload artifacts
      & paper references, discuss solution goals, etc.
    – Can collaboratively evolve the topic within the community
    – Result is a more meaningful & community-validated SBIR topic

   Let‟s look at how we propose to realize this value


                                                                           58
SPRUCE Program Approach to Fulfill Need
   We bring unique perspectives of:
    –   DoD (DARPA, AFRL, etc.) Program Managers & Program Officers
    –   Systems integrators
    –   University researchers
    –   Small business with hands-on SBIR participation
   The idea of collaboration around challenge problems needs to
    be baked & validated offline in a stand-alone community
    – SPRUCE program will provide this bootstrapping function
   Once baked, we will approach the program offices & invite them
    to setup multiple challenge problems in the SPRUCE portal
    – Offers the Program Office a better solution for collaboration, but incurs a
      higher resistance from the Program Office & its users
         • Ideally, in-place, embedded, seamless integration – “white-label” approach
         • Possible with SOA, Web Services – our architecture allows this

   How do we convey that SPRUCE is ready & mature for THEM?
    – Need to show that SPRUCE is in use by various third parties
    – …and that we have community building tools & processes
    – …and that SPRUCE can fit into, & augment, their own processes
                                                                                    59
Path to Broader Community
   Structured collaboration is a key initial path to success
    – Simply pointing people to the initiative will not be successful
    – Repositories (e.g., SourceForge, ESCHER) necessary but not sufficient
    – Our approach incorporates processes, environments & partners who
      understand & are willing to seed initial SPRUCE

   Limited community, who we can explicitly invite/solicit:
    1. SPRUCE team, including extended partners (e.g., from LM Software
       Technology Initiative & other AFRL/IF-funded R&D groups)
    2. Raytheon (DDG-1000 collaborator)
    3. Boeing (SSTT Phase I participant; key stakeholder in FCS & various
       avionics programs)
    4. Other systems integrators (e.g., Northrop Grumman, General Dynamics)
    – SPRUCE Technical Advisory Board facilitates these relationships

   Program efforts to broaden community
    – Yearly user conferences, challenge problem events
    – Technical Advisory Board & SPRUCE Oversight Committee
    – Outreach to Program Offices
Structured participation: limited community  broader community
                                                           60
Community Building Technology & Tools
   Need a means to measure “value” of the SPRUCE community
   Our approach: Affinity Metrics
    – An innovative people, problem, & solution matching & ranking
      technology, to be developed for, & integrated into, SPRUCE

   Objective: provide a framework for:
    – Ranking researchers
        • Based on reputation, past performance, past collaborations
    – Ranking solution technologies (tools)
        • Based on number of active users, tool reliability, tool portability, successful
          application of the tool on past projects
    – Ranking the fitness of tools to address challenge problems
        • Based on the degree to which the requirements descriptions of challenge
          problem matches features of available tools

   These metrics are in addition to standard usage metrics
    available with collaboration environments
    – e.g., number of users, number of accesses, …


                                                                                        61
Affinity Metrics: Proposed Metrics
   Our model manages several affinity relations
      Affinity                Affinity is based on                    Example Result
Researcher to        Compatible interests, similarity of prior   If you have worked with
Researcher           challenge problems worked on,               researcher A, you may find
                     similarity of tools used & developed,       researchers B & C‟s work
                     …                                           relevant
Tool to Tool         Platform & domain compatibility, tool       If you use Tool A, Tool B
                     interoperability, …                         may also be of interest
Researcher to Tool   How similar a tool is to other tools        People that have worked
                     created or used by the researcher           on Tool A are also likely to
                                                                 be conversant with Tools B
                                                                 &C
Tool to Challenge    How closely a tool‟s features match         If you are interested in
Problem              the components of a challenge               Problem A, perhaps tools B
                     problem                                     & C may be useful to you




                                                                                        62
Example: Model to Match Researchers & Tools
   Select the optimal set of researchers based on their experiences &
    specialties to solve a challenge problem
   Identify the most suitable teams of researchers that can provide
    maximum coverage of the sub-challenges of a challenge problem




                                                                     63
Example: Model to Match Researchers & Tools
   Select the optimal set of researchers based on their experiences &
    specialties to solve a challenge problem
   Identify the most suitable teams of researchers that can provide
    maximum coverage of the sub-challenges of a challenge problem
   wRi,j captures affinity
    between researchers
    Ri & Rj




                                                                     64
Example: Model to Match Researchers & Tools
   Select the optimal set of researchers based on their experiences &
    specialties to solve a challenge problem
   Identify the most suitable teams of researchers that can provide
    maximum coverage of the sub-challenges of a challenge problem
   wRi,j captures affinity
    between researchers
    Ri & Rj
   wTk,l captures affinity
    between tools Tk & Tl




                                                                     65
Example: Model to Match Researchers & Tools
   Select the optimal set of researchers based on their experiences &
    specialties to solve a challenge problem
   Identify the most suitable teams of researchers that can provide
    maximum coverage of the sub-challenges of a challenge problem
   wRi,j captures affinity
    between researchers
    Ri & Rj
   wTk,l captures affinity
    between tools Tk & Tl
   wRiTk captures affinity
    between researcher Ri
    and tool Tk




                                                                     66
Example: Model to Match Researchers & Tools
   Select the optimal set of researchers based on their experiences &
    specialties to solve a challenge problem
   Identify the most suitable teams of researchers that can provide
    maximum coverage of the sub-challenges of a challenge problem
   wRi,j captures affinity
    between researchers
    Ri & Rj
   wTk,l captures affinity
    between tools Tk & Tl
   wRiTk captures affinity
    between researcher Ri
    and tool Tk
   wTkCp captures which tool
    Tk contributed to solving
    sub-challenge of challenge
    problem Cp


                                                                     67
Example: Model to Match Researchers & Tools
   Select the optimal set of researchers based on their experiences &
    specialties to solve a challenge problem
   Identify the most suitable teams of researchers that can provide
    maximum coverage of the sub-challenges of a challenge problem
   wRi,j captures affinity
    between researchers
    Ri & Rj
   wTk,l captures affinity
    between tools Tk & Tl
   wRiTk captures affinity
    between researcher Ri
    and tool Tk
   wTkCp captures which tool
    Tk contributed to solving
    sub-challenge of challenge
    problem Cp
   wPs,i captures affinity
    between sub-challenges of
    different challenge problems                                     68
SPRUCE Program – At A Glance
   36-month program: April 2008 – March 2011

               Standup SPRUCE; validate, refine & realize CONOPS
 Build         Standup testbed infrastructure



                       Populate SPRUCE with challenge problems, solutions, experiments
Populate               Build initial SPRUCE user population


                                          Broaden SPRUCE awareness among
Educate                                    participant communities
                                             User conferences, marketing, …


                                          Increase SPRUCE adoption
 Solicit                                     Challenge problem events, affinity metrics,
                                              evangelism, …


                                                                          Sustain value
Moderate                                                                     Collect/post “value”
                                                                              metrics, maintain
                                                                              content freshness, …

                        Year 1                     Year                           Year 3         69
SPRUCE Task: Educate
   Objectives
    1. Broaden SPRUCE awareness among participant communities

   Task elements
    – E1. Newsletters
    – E2. Webinars
    – E3. User conferences




                                                                70
SPRUCE Task: Educate

   E1. Newsletters
    – Quarterly, starting 9 months into the program
    – First one announcing the existence & benefits of SPRUCE
    – Subsequent ones highlighting the populated challenge problems,
      solution technologies, etc.
    – Leverage existing communications vehicles where possible
        •   SISPI, DACS, …

   E2. Webinars
    – Yearly, starting 9 months into the program
    – Describe SPRUCE usage & benefits
    – May be delivered by subject matter experts

   E3. User conferences
    – Yearly, 6 months out of phase of newsletters, webinars
    – May include birds-of-a-feather challenge problem area


                                                                       71
SPRUCE Program – At A Glance
   36-month program: April 2008 – March 2011

               Standup SPRUCE; validate, refine & realize CONOPS
 Build         Standup testbed infrastructure



                       Populate SPRUCE with challenge problems, solutions, experiments
Populate               Build initial SPRUCE user population


                                          Broaden SPRUCE awareness among
Educate                                    participant communities
                                             User conferences, marketing, …


                                          Increase SPRUCE adoption
 Solicit                                     Challenge problem events, affinity metrics,
                                              evangelism, …


                                                                          Sustain value
Moderate                                                                     Collect/post “value”
                                                                              metrics, maintain
                                                                              content freshness, …

                        Year 1                     Year                           Year 3         72
SPRUCE Task: Solicit
   Objectives
    1. Increase SPRUCE adoption

   Task elements
    –   S1. Challenge problem events
    –   S2. SPRUCE committees
    –   S3. Affinity metrics (discussed previously)
    –   S4. Community building plan (discussed previously)




                                                             73
S1. Challenge Problem Events

   Purpose
    – Provide an effective forum for bringing together challenge problem
      providers & candidate solution providers to agree on “baseline” problems
      that represent their interests
        •   “Winners” will receive special recognition by SPRUCE community
        •   Intent is to create new Communities of Interest (COI)
        •   Activities leading up to these events offer a collaborative forum for the COI
    – Supports guided experiments

   Plans
    – Yearly, starting late in Year 2
    – Later start provides opportunity to “iron out wrinkles” of defining
      challenge problems, candidate solutions, experiments & collaboration
      among SPRUCE participants




                                                                                       74
S2. SPRUCE Committees

   Oversight Committee
    –   Why: to provide oversight & direction
    –   Who: DoD SISPI Community (OSD, AFRL, ARL, ONR, …)
    –   What: review progress, provide strategic direction, assist DoD outreach
    –   When: every 6 months

   Technical Advisory Board
    – Why: to provide usage feedback & evangelization
    – Who: DoD system integrators - technical program leads, SISPI subject
      matter experts
    – What: contribute usage feedback & content in early program stages,
      identify & establish major challenge problem areas, bring expertise,
      evangelize SPRUCE within their organizations
    – When: every 6 months
    – Seeking participation from SPRUCE Oversight Committee to identify
      board members
         •   Propose that Patrick Lardieri of LM ATL lead this board & serve as liaison for
             diverse organizational concerns

                                                                                       75
SPRUCE Program – At A Glance
   36-month program: April 2008 – March 2011

               Standup SPRUCE; validate, refine & realize CONOPS
 Build         Standup testbed infrastructure



                       Populate SPRUCE with challenge problems, solutions, experiments
Populate               Build initial SPRUCE user population


                                          Broaden SPRUCE awareness among
Educate                                    participant communities
                                             User conferences, marketing, …


                                          Increase SPRUCE adoption
 Solicit                                     Challenge problem events, affinity metrics,
                                              evangelism, …


                                                                          Sustain value
Moderate                                                                     Collect/post “value”
                                                                              metrics, maintain
                                                                              content freshness, …

                        Year 1                     Year                           Year 3         76
SPRUCE Task: Moderate
   Objectives
    1. Sustain value

   Task elements
    – M1. Collect/post/act on “value” metrics
    – M2. Maintain content freshness/relevance

   M1: Collect/post/act on “value” metrics
    – e.g., measure activity index of a challenge problem area, appropriately
      highlight the one with a high activity index, with a goal of:
        •   Generating more traffic/value for that community
        •   Presenting a role model for other communities’ advocates

   M2: Maintain content freshness/relevance
    – e.g., track emerging trends (e.g., software for sensor networks) &
      encourage SISPI challenge problems in those areas
    – Explore content syndication with other sites


                                                                           77
Program Timeline
                                                                               Program Review Meetings

                                                      Steering Committee and Technical Advisory Board Meetings
             Spiral 0: Out of the Box             Migrate from Pilot to Operations                              Infrastructure Scaling




                                                                                                                                                       BUILD
  BUILD




                                        Pilot Operations                                      Operations                                  Operations

                              Spiral 1                          Spiral 2                                        Spiral 3
                              OOTB Customization of             Customized Collaboration and                    Usage Patterns, Benchmarking,
                              Fundamental Capabilities          Workflow Enhancements                           and Trend Intelligence
                 Post/Deploy




                                                                                                                                                       POPULATE
  POPULATE




                 Challenge                                                               Challenge Problems
                 Problem
                 Tools

                 Metric                                                                  General Usage
                 Tools                                                                                         End Beta
                                             Start Beta

             Release/Conduct
                Admin Guide                                                                     Challenge Problems




                                                                                                                                                       EDUCATE
  EDUCATE




                User Guide,
                White paper,
                Webinars
                 User Conference                                                               General Usage



             Publish/Conduct




                                                                                                                                                       SOLICIT
  SOLICIT




                 Newsletters/                                                                                     Challenge Problems
                 Promotional
                 Highlight of the
                 Quarter
                Challenge
                Events                                                                                           General Usage




                                                                                                                                                       MODERATE
  MODERATE




              Refresh/Redesign
                                                                                                     Challenge Problems
                  Content/links

                  Appearance &
                  Theme                                                                                           General Usage


                           Year 1                                                    Year 2                                              Year 3
                                                                                                                                                                  78
Outline
   Overview
    –   Motivation
    –   Vision
    –   SPRUCE
    –   Background
    –   Key Challenges
    –   Program Overview

   Program Details
    –   Build
    –   Populate
    –   Educate
    –   Solicit
    –   Moderate
    –   Program Timeline

   Risks & Risk Mitigation

   Open Discussion
                              79
Risks & Mitigation Strategies
Challenges &       Tech   Market   Mgmt
                                                         Comments                                   Mitigation Plan
    Risk           Risk    Risk    Risk
 Realizing the                            Technology pieces are standardized;          Have established a well defined timeline &
  Functional       Low     N/A     Low    already have well defined requirements;      a monitoring process
 Architecture                             our team has prior experience.
                                          Technology pieces are readily available;     Involve HMI experts in design reviews;
 Realizing the                            but, interface & usability is subjective &   release early, get user feedback &
                   Low     N/A     Med.
  Use Cases                               may incur budget overruns, if left           establish a disposition process
                                          unchecked by process
                                          Market adoption risk is high; past           Release early & experiment often, both on
                                          technology efforts to-date have not been     technology & communication initiative
                                          tried for DoD applications; challenge        front; leverage Steering & Advisory
   Bringing
                                          problem quality & relevance needs is         Committee expertise & inputs; establish
  Together         Med.   High     Med.
                                          critical for the Community of Interest       agile processes to enable rapid changes
 Communities
                                          (COI); our team & the community need         in strategic (e.g., new COI, frameworks) &
                                          to learn new skills & be able to overcome    tactical (new challenge problems,
                                          inherent competitive biases.                 participants) approaches
                                                                                       Use Technical Advisors to solicit
                                          Users must overcome traditional barriers
  Enabling                                                                             „unexpressed‟ inhibitions & motivations;
                   Low    High     Low    & see value; technology elements must
 Participation                                                                         implement technology to break barriers &
                                          be relevant to DoD applications
                                                                                       establish a must-be-in network-effect
                                                                                       Our relationships, partnership
Integrating with
                                          Inevitable technology challenges when        conversations, & rapid standardization of
    Existing       Med.    Low     Low
                                          integrating with diverse data repositories   integration technologies (SOA, Web
   Initiatives
                                                                                       services) will be exploited.
Communicating                             The relevance of proposed metrics to         Proactively & monitor & establish the
                   Med.    N/A     Low
   Value                                  user-base is untested                        relevance of metrics


                                                                                                                         80
SPRUCE Program: Biggest Risks

   Bringing together communities
    – Although technologies exist, DoD ecosystem participant‟s culture may
      not be amenable to social networking
    – Challenge problem quality must be high & must keep evolving to warrant
      repeated participation
    – Community must overcome competitive biases
    – Our team must learn new skills
        • e.g., How to influence third-party organizations & people to voluntarily
          participate; how to discriminate when an approach is working & just needs
          time vs. approach needs strategic changes

   Enabling participation
    – Must overcome traditional barriers
    – Challenge problems must stay relevant




                                                                                  81
Elements of the Risk Mitigation Plan
1. Release early, experiment often
2. Demonstrate the SPRUCE promise with good examples
    – Core users & initial challenge problems are critical
3. Leverage Oversight Committee & Technical Advisory Board
4. Align with researchers, other agencies (e.g., DARPA, NSF)
5. Implement changes rapidly
    – New communities (e.g,. Cyber Physical Systems)
    – New problems, participants
6. Use insights to understand “unexpressed” inhibitions
    – e.g., “Is posting a problem acknowledging our weakness?”
        •   Explore technology/positive reinforcements to mitigate them
        •   “No; LM Aeronautics has highly rated problems – they are well regarded”

7. Aim to establish MUST-BE-IN-THE-NETWORK effect
    – Must reach tipping point
    – e.g., when all your friends are in facebook!
   Any other ideas from YOU?
                                                                                  82
    Concluding Remarks
    Recall: Imagine A Future, Where…
   As a Program Engineer toiling away, putting out day-to-day fires
     – …you can quickly & easily discover & reach out to a broad community of
       people who can relate to your technical problems
     – …you can quickly & easily learn about technologies that may help solve your
       problem
     – …you can readily engage a community of experts to solve your problem
     – as icing on the cake, DoD program managers are watching this activity, to
       solicit your input in their next dream project
   As a SISPI researcher building exciting technologies & tools
     – …you had a repository of real-world problems to work with
     – …you can quickly & easily engage an active practitioner community
     – …you had a means to effectively demonstrate how your technologies & tools
       could solve real-world problems
   As a DoD Program Manager
     – …you had access to a continuously evolving virtual community of people,
       problems & technologies from both constituents above
     – …and that these constituents can contribute critical insights, data & artifacts to
       get your program justified, started & transitioned!

           SPRUCE – the path to realizing this future
                                                                                  83
    Concluding Remarks
    Recall: Vision
   Methods & technologies that create a common meeting house for
    program engineers & technology researchers to discover joint
    software producibility interests & form collaborations
    – With testbed resources & tools specialized to particular operational domains
      (e.g., avionics)




   Benefit: challenge problem-driven, at-scale experiments can be defined
    & conducted to prove/disprove suitability of particular software
    producibility technologies
    – …which yields a more systematic process for transitioning software producibility
      technologies
        SPRUCE – the path to implementing this vision                          84
                     SPRUCE
Revolutionizing the way that software-intensive systems
producibility technologies are identified, developed, and
                       evaluated




                                                      85
Questions?
Comments?




             86
Backups




          87
B9. SPRUCE Deployment Schedule
  Contract Start Kickoff & Reqts.     Virtual Design  Design Package Pre-Deployment
  (3 Apr)        Review (22 & 23 Apr) Review (16 May) (6 Jun)        Readiness Review (30 Jun)
                 Planning (10 Apr)
                                                                     Deliver                                                   Milestone Review
               Infr. Plan                           Install
                                                                     Spiral 0
              *Qualified     Procure             Receive                                                                       Milestone
              Product List   Equipment           Equipment
                                                                                                                               Critical Path Event
  DEFINE




                                         Draft         Final
                        Requirements       Updates                                                                         *   SOW Deliverable
                       *Requirements matrix
                                                                  Draft            Final
                                  Design                          Updates                                            Over 70% of use cases
                               *Design: Taxonomy, Data Model, Use Case Flows, Interface, Access Control Matrix       implemented

                                                                                                                                      Deliver
                                                                  Operations Env. OOTB Customization                Updates
  IMPLEMENT




                                                                                                                                      Spiral 1
                                                         Begin Customization Activities
                                                                                               Validation                         Deploy
                                                                                           Validate Functional Use Cases
                                       Integrate Multi-Core Equip.              Experimental Environment
                                                                            Coordinate access and provisioning
                                                                            of Vanderbilt Emulab resources

                                                                                                         Documentation
  DEPLOY




                                                                                           *User & Administrator Quick Guides Final

                                                                                                                                  Maintain &
                                                                                                                                  Operate

                  Month 1: April 2008              Month 2: May 2008                 Month 3: June 2008               Month 4: July 2008
  Spiral 0: Out of the box capabilities & Spiral 1: Fundamental Capabilities
                                                                                                                                                     88
Challenge Problem Evaluation




 Members of the SPRUCE Technical Advisory Board evaluate challenge problems
 against established quality criteria                                    89
   PUT IN BACKUP IN CASE NEED TO EDIT
   (Scaled Version used in presentation body)
                                                 New software research ideas

                                                            Advanced
                                                           Engineering
                                                              Tools


                                 Systems / Software
                                                                             Technology               Challenge
                                  Engineering Tool
                                                                             Researchers   yields      Problem
                        sponsors Research Programs
 Software-Intensive                                                                                   Driven, At-
                                                                         collaboration                  Scale
Systems Producibility
  Initiative (SISPI)                                                          Program                Experiments
                                     Software &
                                                                             Engineers
                                 Systems Test Track
                                      Program                                                       yields

                                                                Tool
                                                             Evaluation
                                                            Environment                         Systematic transition
                                                                                                     of software
                                                      New test track ideas                          producibility
                                                                                                    technologies




                                                                                                             90
Lockheed Martin
Software Technology Initiative
              Vision

  Establish Lockheed Martin Corporation as a software development &
   systems integration technology innovator in the Intelligence,
   Department of Defense, & DHS communities



                         Mission
           Contribute – Give back to DOD, DHS, & Intelligence communities
           Leadership – LM recognized as a leader in the field
           Competitive advantage – Partner of Choice in the marketplace



                                      Strategy
                             Perform transformational technology research: 5x faster @ 1/5 the cost
                             Address functional & non-functional issues; eliminate “red” programs
                             Contribute technology & knowledge back to LM, our customers & the
                              research community




                                                                                                       91
B2. Actor Relationship to SPRUCE
Functional Architecture Components


                                                                Candidate Solution
                                                                    Provider                          Challenge
                                                                                                   Problem Provider
             SPRUCE                                  provides resources    provides candidate
                                      Experimenter
           Administrator                                   & tools         technology solutions



   SPRUCE                                                           Candidate              Challenge
                      Account              Experimentation
Infrastructure                                                      Solution               Problem
                     Management             Management
 Management                                                        Management             Management


                                                                            provides strategic
                                                                            SISPI challenges

                                                                          enables & facilitates collaboration

                                                                   Collaboration
                                                                   Management
                      Collaborators



                                                                                                           92
Challenge Problem Backup Slides


     Detailed Description of Each of the 5 Challenge
                   Problem Instances




                                                       93
Predicting Performance of Avionics Mission Systems
Using a Logical Task Network & Computing Plant
Description




                                                     94
Optimization Techniques for Deployment of Avionics
Mission Systems onto a Specific Computing Plant




                                                     95
Optimization Techniques for Deployment of
Binary Software Systems onto Multi-Core
Platforms




                                            96
Post Mortem Estimation of Software Development
Efficiency Metrics Across Software Releases




                                                 97
Predicting Fine Grained Software Productivity Metrics




                                                    98

								
To top