Get documents about "Slide 1 Naval Postgraduate School (PowerPoint)
Document Sample
ASW METOC Metrics: VS07 Data Collector
Workshop
Bruce Ford
Clear Science, Inc. (CSI)
bruce@clearscienceinc.com
Tom Murphree
Naval Postgraduate School (NPS)
murphree@nps.edu
Brief for VS07 Training
19 June, 2007
Visualizing the Metrics Collection Process
METOC METOC Operational Operational
Forecasts * Observations Plans Outcomes
METOC Operational
Performance Performance
Metrics Metrics
Metrics of METOC
Impacts on Operational
Performance
* or other products Ford B. et al., VS07 Data Collection, May 07, bruce@clearscienceinc.com
Overall Goals
• Collect that data
• Examine the process
Ford B. et al., VS07 Data Collection, May 07, bruce@clearscienceinc.com
General Data Collection Plan
• Aboard each CV
• Mimic NOAT data collection role by collecting:
• Discrete data
• Verifying in situ data
• Recommendations (tactical and mitigation)
• Recommendation outcomes
• Customer measures of success
• Investigate other potential data sources, methods, customer
measures of success, etc.
• At RBC
• Monitor watch officer actions and log for potential data collection
• Monitor NOAT interactions
• Observe product generation process
• Embedded with deployed MPRA TSC (MOCC)
• Mimic MPRA data collection by collecting:
• Data from each GREEN and PURPLE
• Discrete forecast elements
• Investigate other potential data sources, methods, customer
measures of success, etc.
Ford B. et al., VS07 Data Collection, May 07, bruce@clearscienceinc.com
General Data Collection Plan (continued)
• Embedded with each major staff
• Mimic NOAT data collection role by collecting all:
• Recommendations (tactical and mitigation)
• Recommendation outcomes
• Customer measures of success
• Investigate other potential data sources, methods, customer
measures of success, etc.
Ford B. et al., VS07 Data Collection, May 07, bruce@clearscienceinc.com
Data Collector Responsibilities
• Prior to exercise
• Train on data collection procedures (what you are doing now)
• Help improve data collection forms (spreadsheets)
• Participate via email in discussion/form
preparation/arrangement-making, possible conference call
• Become familiar with proposed data collection system for
NOATs and MPRA metrics nodes (briefs provided)
Ford B. et al., VS07 Data Collection, May 07, bruce@clearscienceinc.com
Data Collector Responsibilities
• During the exercise
• Check in with RBC metrics coordinator when settled on scene
and provide SIPR email address if available
• Record data on specially-prepared paper forms and
spreadsheets
• From METOC products
• From briefs (be present at as many as possible!)
• From interactions with staff
• Process improvement recommendations
• Metrics collection process
• METOC support process
• Record personal observations and suggestions
• Assess how eventual metrics interface and process will
impact future deployed NOATs
• Strive to understand the ASW warfighter and how their
success is measured
Ford B. et al., VS07 Data Collection, May 07, bruce@clearscienceinc.com
Data Collector Responsibilities
• During the exercise (continued)
• Participate in twice-daily metrics collector chats (hosted by
the RBC). Details TBD and will be sent to everyone.
• Submit spreadsheets to RBC for archival as soon as
completed
• Timing will differ by NOAT and NOAD/MOCC
• Stay vigilant regarding the overall goals and methods
• Ask questions!
• Out-of-the-box thinking is encouraged!
• Remain available to answer questions from metrics data
coordinator (RBC)
• Keep in mind how the VS07 metrics effort fits within the
larger ASW metrics project
• Record and comment on processes (METOC and metrics)
• Collect METOC impactseton VS07 DataWarfighter bruce@clearscienceinc.com
Ford B. al.,
ASW Collection, May 07,
Data Collector Responsibilities
• After the exercise
• Remain available to answer questions expeditiously via email
or phone regarding data collection
• For those at SSC, plan on post exercise meeting TBA.
Ford B. et al., VS07 Data Collection, May 07, bruce@clearscienceinc.com
Brainstorming
• Understanding how the VS07 metrics collection fits with the
larger ASW metrics project
• Metrics data collector day
• Where and when to collect the best data
• Positioning ourselves to collect obvious and subtle impacts
• Data collection forms – Go over tab-by-tab
• Improving the forms prior to the exercise
• Improving them on the fly during the exercise
Ford B. et al., VS07 Data Collection, May 07, bruce@clearscienceinc.com
Overview of ASW Metrics Data Collection System
NOAT Metrics Node Customer
MPRA Metrics Node
MEP
Recco Measures
Builder Green
Contact Information of Success
Information Purple
In Situ
Data Freeform
Data Entry
FCST/Anal Quality
Control
Metrics
Server
Exercise
Intentions
Watch
Officer
Log
Planning
Impacts
NOAT
Exercise Survey
Outcomes Flag Objective
Exercise Data
Records Collection
R&A Metrics Node RBC Metrics Node
METOC Data Source Non-METOC Data Source
Ford B. et al., VS07 Data Collection, May 07, bruce@clearscienceinc.com
Questions?
Ford B. et al., VS07 Data Collection, May 07, bruce@clearscienceinc.com
Backup slides
Ford B. et al., VS07 Data Collection, May 07, bruce@clearscienceinc.com
Why Bother With Metrics?
1. Develop and transition to operational use systems for:
a. collecting data from METOC units and their customers
b. quantifying METOC performance and impacts on customer
operations
c. modeling and predicting impacts of METOC support on
war fighting operations
2. Identify methods for improving quality and efficiency of METOC
support.
3. Recommend:
a. focus directions for METOC resources
b. methods to incorporate METOC into OPNAV assessments
Ford B. et al., VS07 Data Collection, May 07, bruce@clearscienceinc.com
Definitions
Metrics: Objective, quantitative, data based measures of products
and services. Examples:
Metrics of product quality
Metrics of effects of products on customers
METOC Metrics: Metrics of METOC organization’s products and
impacts. Two main types:
Performance metrics: metrics of capacity, readiness, quality,
efficiency / return on Investment
Impacts metrics: impacts on warfare customer operations
(planning, execution, post-assessment)
Methods for Generating METOC Metrics
Collect / analyze real world data on METOC and customer ops
Model METOC and customer ops
Ford B. et al., VS07 Data Collection, May 07, bruce@clearscienceinc.com
Hierarchy of Metrics
Higher Level
Metric Symposium (Navy-wide SLD accuracy) CNMOC/Fleet
Focus Space MetricsLarger Spatial and/or
Temporal Scale
(Exercise forecast location)
Directorate
Metrics
NOAC
Metrics
Performance
(Temperature and salinity Impacts
accuracy) Exercise (Number of positively identified
Metrics submarines)
NOAT
Metrics
Ind. Forecast
Smaller Spatial
and/or Temporal Metrics
Scale Lower Level
(Point forecast location) (MOATs SLD accuracy)
Ford B. et al., VS07 Data Collection, May 07, bruce@clearscienceinc.com
VS07 Opportunity
• Offers the opportunity to simulate a larger metrics system by
collecting and analyzing data as an established metrics program
would
• Offers an excellent opportunity to develop, test, and improve
data collection and analysis process from multiple perspectives:
• NOAT
• RBC
• MPRA
• Exercise
• Two key types of data to collect
• Performance data (METOC and customer)
• Impacts data
• Process data
Ford B. et al., VS07 Data Collection, May 07, bruce@clearscienceinc.com
VS07 Data Collection Sites
Stationing of Data Collectors in Priority Order
1. Aboard each CV/ASWC NOAT
2. At RBC
3. Embedded with CTF-74
4. Embedded with deployed MPRA TSC (MOCC)
5. Embedded with 3rd Fleet staff
Ford B. et al., VS07 Data Collection, May 07, bruce@clearscienceinc.com
