Learning Center
Plans & pricing Sign in
Sign Out

Metrics of METOC Support of ASW


									                         Metrics of METOC Support of ASW
                       Summary of Symposium Held at CNMOC
                                 04-05 January 2007

1. General Symposium Information

The METOC Metrics for ASW Symposium was held 04-05 January 2007 at the facilities
of the Commander Naval Meteorology and Oceanography Command (Bldg 1100)
aboard Stennis Space Center, Mississippi. On 04 January, background briefs and
discussion was held regarding the requirements, role and interplay of metrics
information on Anti-Submarine Warfare (ASW) support to Naval forces. On 05 January,
brief and discussions were concluded and symposium findings, recommendations and
action items were decided on. Participants attended the symposium in person and by

The symposium web site is available by clicking here

The symposium agenda is available by clicking here

A list of symposium participants is available by clicking here.

2. Goals

The overarching goals of the symposium were:
   1. Develop a concise description of needed METOC metrics process for ASW,
      a.     Inputs to and outputs from system
      b.     System components and interaction of components
      c.     Interactions of system with ASW customers
      d.     IT infrastructure
      e.     Operators of system
      f.     Uses and users of system
      g.     Benefits and beneficiaries of system
      h.     Challenges in developing system
   2. Outline plan for developing this ASW metrics process, including desired/tentative:
      a.     Performers
      b.     Deliverables
      c.     Timeline
      d.     Sponsors
      e.     Interactions with ASW customers
      f.     Interactions with other METOC metrics and assessment efforts
   3. Establish ASW METOC Metrics working group to help guide and facilitate
      present and future efforts.
   4. Develop and assign action items with target dates

3. Summaries of Background Briefs (in order of presentation)

a.    CAPT Jim Berdeguez (ASW DOO) – ASW Metrics Symposium Kick-off Brief

CAPT Berdeguez summarized his responsibilities, and the structure and relationship of
the ASW directorate to the rest of CNMOC. In addition, he presented his overall goals
for the symposium and eventual metrics process. He stressed that any metrics system
           assist the warfighter in maintaining and increasing an asymmetric
           assist CNMOC in force-shaping and making budgetary decisions
           improve the accuracy and methods employed to create METOC products.

A key point in CAPT Berdeguez’ talk was that an ASW metrics system must be
compatible with the Battlespace on Demand (BonD) three-tiered concept.
           Tier 1 – Environmental Layer (e.g., improving sampling quality and
           Tier 2 – Performance Layer (e,g., improving METOC product quality)
           Tier 3 – Decision Layer (e.g., improving the decisions of commanders
              regarding METOC conditions)

      Link to the Berdeguez brief

b.    CDR Van Gurley (NOAC SSC)– Operational ASW Program Overview

CDR Gurley summarized the mission of the units that provide METOC support for ASW:
to provide an asymmetric advantage for ASW forces through the application of
oceanographic sciences. ASW forces are supported primarily by Naval Oceanographic
ASW Teams and Detachments (NOATs and NOADs) aligned with major ASW staff and
Maritime Patrol and Reconnaissance (MPRA) units, with significant support provided by
the ASW Reach Back Cell (RBC, a joint effort of the NOAC and NAVO).

CDR Gurley presented a list of possible metrics to be collected, pointing out that the
focus should be on quantifying the impact of the METOC community on the ability of
ASW warfighters to perform their missions, rather than internal metrics (e.g., number of
products issued, number of surveys completed).

      Link to the Gurley brief

c.    Ed Gough (CNMOC) – Historical Perspective on ASW Metrics

Mr. Gough discussed the role of science in combating submarine threats in WWI, WWII,
the Cold War, and extending to today’s emerging threats. He noted that ASW has been
conducted for many years, and that data collected during those years by ASW
operators may be relevant to our metrics efforts. Some of metrics of ASW effectiveness
used by ASW operators are screen penetrations, number of submarine attacks, and
number of submarines that survive after launching an attack. He recommended

working with these ASW operator metrics and determining how METOC products affect
these ASW operator metrics.

He noted that getting accurate analyses and forecasts of the ocean are critical and that
metrics need to determine how well we do that. In peacetime, exercises are the best
situations in which to collect data and perform reconstructions and analyses.

Mr. Gough pointed out that around $8bln/year is spent on ASW yet we have no
evidence that our ability to combat submarines has improved. His talk was concluded
by encouraging participants to leverage all existing efforts going on at NAVO,
reconstruction and analysis efforts to help identify discrepancies or gaps in our existing
procedures and capabilities, including gaps in METOC support.

       There were no slides accompanying this brief.

d.    Tom Murphree (NPS) - METOC Metrics: Design, Development, and

Dr. Murphree reviewed the basic definitions, goals and methods of any metrics system,
and noted the importance of metrics in reducing uncertainty about the environment in
order to improve warfighting operations. He defined metrics as objective, quantitative,
data based measures of an organization’s operations, products, and services. He
identified two main methods for generating methods: (1) collecting and analyzing data to
develop real world metrics; and (2) modeling of operations to develop model based

He described three basic types of metrics: (1) METOC performance metrics that
describe the quality of METOC products and are determined by comparing products to
observations; (2) operational performance metrics that describe the performance of the
operational customers for METOC products that are determined by comparing customer
plans to customer outcomes; and (3) operational impacts metrics that describe the
impacts of METOC products on customer operations that are determined by comparing
METOC performance metrics to customer performance metrics. He showed examples
of each of these types of metrics based on real world Naval and Air Force data sets.
He outlined an operational, automated, real-time, sipr-based system for collecting and
analyzing data to produce these types of metrics currently in use for developing METOC
metrics for Naval strike warfare. This system was discussed as a working model on
which to base an ASW metrics data collection, analysis, and reporting system.

Dr. Murphree concluded his talk with his proposal for an ASW Metrics System:
         a. Develop ASW real world data collection, analysis, and reporting system
         b. Develop ability to model ASW operations and METOC support of them
         c. Use real world and model systems to test and experiment (e.g., with
            variations in objectives, weighting of objectives, means to meeting
            objectives, success measures, etc.)
         d. Report results rapidly, automatically, and in optimal formats to both
            METOC personnel and ASW customers

          e. Support facts-based decision making at multiple levels by both METOC
             personnel and ASW customers

       Link to the Murphree metrics brief

       Link to NPS METOC Metrics Reports

e.     CDR Steve Woll – METOC Metrics Lessons Learned

CDR Woll presented his brief via telephone and opened his talk with a discussion of a
potential end-state of a community-wide metrics effort. This end state was illustrated by
the “dashboard” concept in which CNMOC leadership can routinely view an automated,
real-time, stoplight type of graphic to diagnose the state of their organization. This state
would be described in terms of metrics that leadership has determined best monitor the
organization’s key elements (e.g., capacity/readiness, quality, impact, and
efficiency/ROI) over time. This display would allow users to drill down to view additional
metrics and other details. CDR Woll discussed the benefits of such a dashboard for
management of the organization, budget justification, and training of personnel. He
noted that it is important to: (1) automate the metrics process to minimize impacts on
limited resources; (2) ensure that desired data can be feasibly collected with existing
human and budget resources; (3) carefully select the optimal set of metrics --- too many
metrics can cause overwhelm users of metrics but too few can over simplify the analysis
of your organization.

In part two of his presentation, CDR Woll described the SGOT./STW forecast
production and analysis process, including GWEAX and TAF production, and data
collection, analysis, and reporting for the SGOT Norfolk metrics system. He then used
this process as the basis for a proposed ASW metrics system.

       Link to Woll brief, Part 1

       Link to Woll brief, Part 2

f.  Luke Piepkorn and Matt McNamara (Systems Planning and Analysis – SPA) –
METOC Metrics in Operational Modeling

Mr. Piepkorn gave an overview of operational analysis modeling efforts at SPA that are
designed to develop model-based operational impacts metrics for strike warfare. The
main model, the Weather Impact Assessment Tool (WIAT), was described, including
examples of the operational impacts metrics that are generated by the model. Inputs to
the model include results from the real world metrics system described by Dr. Murphree.
This model is based on logical rules that determine the interaction of the model
variables (e.g., the interaction of METOC forecasts on mission planning and execution
decisions). Mr. Piepkorn emphasized that WIAT can be used to conduct experiments to
test the sensitivity of the operational impacts metrics to a wide range of variations,
including variations in the quality of METOC products and CONOPS.

Mr. McNamara then discussed the application of WIAT to the ASW problem. The
metrics decided on and collected within the real world ASW metrics system would
provide inputs to WIAT. ASW warfighting (engagement) scenarios would then be
created. The results of the model could include the impacts of accuracy of each
METOC parameter, impact of METOC product timeliness, the level of METOC support
required to support engagement, etc.

      Link to the Piepkorn and McNamara brief

g.    Jim Dykes (NRL-SSC) - ASW Automated Metrics RTP FY07 – FY09

Mr Dykes presented a talk about a recently funded project for developing automated
metrics of ocean products used to provide METOC support for ASW. A main goal of the
project is to develop tools for subject matter experts (SMEs) in the RBC to use in
assessing the products they use to develop the tactical ocean analyses (TOAs) they
prepare for the NOATs/NOADs. In addition, this project will develop metrics of the
impacts of the TOAs on the support provided by the NOATs/NOADs.

The metrics to be addressed in this project are broken into three areas:
           Ocean model metrics (e.g., accuracy of ocean model outputs)
           Acoustics metrics (e.g., accuracy of acoustic model outputs)
           Product performance and operational impacts metrics (e.g., accuracy of
             TOAs, impacts of TOAs on NOATs/NOADs)

Mr. Dykes described the overall motivation and process for developing these types of
metrics. He also presented a POA&M for the project and noted that this project will be
coordinated with several other related projects.

      Link to the Dykes brief

h.    CDR Mike Angove (OPNAV/N84) - OPNAV Perspective on ASW Metrics

CDR Angove reviewed the budgetary process and the role of his office in that process.
He presented an OPNAV perspective on the need for metrics in that process. He noted
that in terms of funding, it is not merely enough to improve our knowledge of the
environment, but we must know how our improved knowledge of the environment will
improve the mission ability of the warfighter. In order to obtain funding to improve
METOC support, clear connections between the proposed changes in support and
warfighting effectiveness must be shown through the use of metrics derived from data
and operational modeling. That is, a credible quantitative assessment must be
developed of the operational ROI expected from the proposed METOC support change.
Examples were presented to support these concepts (e.g., modeling of the reduction in
HVU losses (in dollars) that results from an investment (in dollars) that reduces
environmental uncertainty).

      Link to the Angove brief

i.    Rich Kren (CNMOC/N8) – CNMOC N8 Perspective on ASW Metrics

Mr. Kren presented an overview of metrics from the CNMOC N8 perspective. He
identified a number of reasons for having a metrics program, including: (1) developing
baselines measures (e.g., of skill, accuracy); (2) document impacts of changes you are
trying to make; (3) determine ability to make changes; (4) improve allocation of
resources; (5) improve and document your understanding of your organization (helps
build a sense of credibility and trust with customers); (6) gain insight into what you’re
doing, where you are; and (7) inform decisions. He suggested that we think carefully
about what we want to measure, and that we make the number of measures small
(there is a cost to developing these metrics).

He also described what CNMOC N8 is doing with respect to metrics. His office reports
to FFC, which is concerned with readiness. So they are in the process of learning how
to develop readiness metrics, with FFC as the customer for these metrics

There were no slides accompanying this brief.

j.    Tom Murphree (NPS) – NPS Smart Climatology: Support for Naval Warfare

Dr. Murphree presented a lunch time brief on the need for the development of smart
climatology to support Naval warfare. He defined smart climatology as state-of-the-art
basic and applied climatology that directly supports DoD operations. The need for
improved ASW support at leads of weeks to months described in the brief by CDR Van
Gurley was cited as an example of the need for smart climatology at intraseasonal time
scales. He described four components of the smart climatology program at NPS: (1)
education; (2) development and application of smart climatology methods; (3)
development of operational products; and (4) transitioning of the products to operational
use. He gave examples of each of these components, with a focus on intraseasonal to
interannual climate variations of the atmosphere and ocean in regions of DoD interest
(e.g., SWA, western North Pacific). Dr. Murphree also outlined the practical value of
smart climatological methods in the context of ASW support, and outlined a
collaborative process for developing and applying smart climatological methods to
METOC support for ASW. He proposed that metrics of the impacts of smart climatology
methods on METOC product performance and ASW customer operations be developed
and used to determine the ROI of resources directed toward smart climatology.

      Link to the Murphree smart climatology brief

      Link to the NPS Smart Climatology site

4. Conclusions

Discussions throughout the symposium and especially on 05 January, led to a number
of general conclusions, including:

       Collection of ASW metrics, though challenging, is a valid requirement and should
        be done as soon as practical.
     Metrics fall into several categories:
            o METOC performance metrics – Metrics of the quality of the METOC
                organization, including its operations, products, and services (e.g.,
                accuracy of METOC forecasts of SLD)
            o Customer performance metrics – Metrics of customer success (e.g.,
                number of screen penetrations by threat submarines)
            o Operational impacts metrics – Metrics of the impacts of METOC
                products on customer operations (correlation of SLD forecast accuracy to
                screen penetrations)
            o Proxy operational impacts metrics - Metrics of product performance
                that are acknowledged by METOC and customer personnel to be closely
                related to customer performance, so that the proxy operational impact
                metrics can be assumed to be good indicators of operational impacts
                (e.g., accuracy of SLD forecasts which is assumed to be closely related to
                customer success)
            o Phenomena metrics – Metrics that relate product performance, customer
                performance, or operational impacts to specific environmental phenomena
                (e.g., SLD forecast accuracy in the presence of high winds and seas;
                screen penetration in the presence of high winds and seas; correlation of
                SLD forecast accuracy and screen penetration in the presence of high
                winds and seas)
            o Other examples of METOC metrics:
                     METOC Quality Metrics – Metrics of the overall quality of a product
                        or service (e.g., TOA grade). These are a type of METOC
                        performance metric.
                     METOC Capacity/Readiness Metrics – Metrics of organization’s
                        operations (e.g., number of SMEs available, number of trained
                        watchstanders). These are a type of METOC performance metric.
                     METOC Efficiency/ROI Metrics – Metrics of how well an
                        organization uses its resources (e.g., number of SMEs required to
                        publish TOA). These are a type of METOC performance metric.
3-D Metrics Space – Figure 1 shows the conceptual space into which the metrics
efforts discussed in the symposium fit. In this figure, the green metrics type axis
describes the degree to which a given metrics effort focuses on METOC performance
(to the left) versus operational impacts (to the right). The blue spatial / temporal axis
describes the spatial and/or temporal scale on which the metrics effort is focused
(smaller (larger) spatial/temporal scales to front (back)). The red organizational axis
describes the organizational level on which the metrics effort is focused (lower (higher)
levels at the bottom (top)). Figure 2 shows examples of points within the metrics space
at which data may be collected and analyzed, and for which metrics may be calculated
(e.g., the metrics shown on the metrics type axis).

           Figure 1. 3-D Metrics Space

Figure 2. Example Points Within 3-D Metrics Space

Initial and End State Focus Regions - The initial efforts of our overall ASW metrics
program will tend to focus on the central region of the 3-D metrics space but shifted
toward the smaller scales and lower level organizations levels (purple oval in Figure 3).
Our goal is to build toward an end state focus on operational impacts occurring over
large spatial/temporal scales, and assessed at high levels of both the METOC and ASW
customer organizations (orange oval in Figure 4). Individual projects within the overall
ASW metrics program may start off with a focus on other regions of the 3-D metrics
space (e.g., the project described by Jim Dykes will be focused to the right and below
the purple oval; OPNAV/N84 projects will focus in the region noted by the orange oval).

                Figure 3. Initial Focus Region in 3-D Metrics Space

              Figure 4. End State Focus Region in 3-D Metrics Space

      The three most likely areas within the METOC ASW directorate for metrics data
       collection and analysis are:
           o Support to Maritime Patrol and Reconnaissance Aircraft (MPRA)
           o ASW Reachback Cell (ARBC)
           o Naval Oceanography ASW Teams (NOATs)
      Three investigative focus committees were formed to answer specific questions
       and concerns regarding each of these areas.
      An executive committee was formed to guide the overall ASW metrics program.
      An additional meeting will be held to discuss the focus team findings and plan the
       next steps of the project.

5. Action Items

      The following persons were assigned to the executive committee:

          o   LEAD: Dr. Tom Murphree (NPS)
          o   CNMOC Technical Director - Mr. Ed Gough
          o   ASW DOO/DDOO - Capt Jim Berdeguez, LT Scott Parker
          o   CNMOC N9 – Steve Lingsch
          o   NMAWC - CDR Ash Evans
          o   N84 – CDR Mike Angove
          o   NRL - Pat Hogan, Josie Fabre, Greg Jacobs
          o   ONR – CDR Doug Marble

       o SPA – Paul Vodola, Matt McNamara, Luke Piepkorn
       o PDC – Merrill Stevens
       o PMW 180 – Marcus Speckhahn

   The following persons were assigned to focus committees:

       o MPRA
            LEAD: Clear Science – Mr. Bruce Ford
            NOAD Kadena OIC - LCDR (Sel) Danny Garcia
            NOAD JAX OIC – LT Eric Macdonald
            CPRG - CDR Sopko
            NRL - Pat Hogan
            APL- UW – Mr. Bob Miyamoto
            FNMOC – LTJG Dave Watson
            PDD South - Doug Lipscombe
            SPA - Paul Vodola, Matt McNamara, Luke Piepkorn

       o RBC
                LEAD – NOAC RBC LT Heather Hornick
                NRL - Jay Shriver, Jim Dykes, Josie Fabre
                NAVO – John Blaha, Frank Bub, Keith Atkinson, Dennis Krynen
                FNMOC – LTJG Dave Watson
                Clear Science – Bruce Ford
                SPA - Paul Vodola, Matt McNamara, Luke Piepkorn

       o NOAT
                LEAD – NOAC SSC LT Tim Campo
                NOAC Yoko - LCDR Joel Feldmeier
                Clear Science - Bruce Ford
                NRL - Jim Dykes, Josie Fabre
                SPA - Paul Vodola, Matt McNamara, Luke Piepkorn

   Each focus committees will make every attempt to accomplish the following
       1. Determine the customers for the METOC products in your ASW
       directorate area (MPRA, ARBC, NOATs). Customers may include other
       METOC personnel, warfighters, and/or others).
       2. Describe the METOC support process for these customers in terms of the
       customer’s mission timeline (e.g., planning, execution, debrief). Diagram
       and/or flowchart this process. Include:
            Unclassified examples of products issued
            Annotations of how METOC support is incorporated
            Are TTPs used for METOC support? If so, how?
       3. Develop an initial list of the metrics that should be developed for the
       METOC support process. See list of metrics at beginning of section 4, above.
       List should include at least:

                METOC performance metrics
                Customer performance metrics
                Operational impacts metrics
                Proxy operational impacts metrics
                  Must be closely correlated with customer performance
                  End user must agree that these proxy metrics are directly related to
                     their ability to complete their mission successfully
                  Examples: Accuracy of SLD, Sea Surface Height, MDR; buoy
                     depth setting recommendations
          4. Describe data collection process, including:
              Data sources (observations, METOC products, mission debriefs)
              Data collection frequency (hourly, 12-hourly, weekly)
              Relationship of data collection to end-user mission timeline
          5. Describe data analysis process, including process for verifying products,
          calculating desired metrics, developing reconstructions, etc.
          6. Describe operational analyses and/or modeling needed, including data
          and methods to be used, validation of methods, types of results to be
          generated, verification of results, etc.
          7. Map recommended metrics to 3-D metrics space (Figure 1)
          8. Relate your METOC metrics process to other relevant METOC and
          customer metrics efforts, including past, on-going, or near-future data sets
          and metrics calculations.
          9. Refer to symposium briefs and contact symposium participants for
          background information, examples of the results you need to prepare for the
          executive committee, coordination, collaboration, etc.
      The executive committee will arrive at funding proposals based on three levels of
           Bare bones
           Adequate to complete project
           Completely funded

6. Draft Plan of Actions and Milestones (POA&M)

          Focus committee tasking released – 17 Jan 07
          Focus team committee completion, with Findings due to LT Parker – COB, 21
           Feb 07.
          ASW metrics meeting and funding proposal – 07-09 Mar 07
          Strategic plan development – ongoing
          List of related projects - TBD
          List of potential resources (thesis students, ONR?) - TBD
          Draft plan for Valiant Shield metrics effort – TBD
          NMAWC R&A visit – 22-26 Jan 07
          Provide input for METOC data collection for DCM - TBD
          SWDG R&A Visit - TBD
          DEVRON 12 R&A Visit - TBD

          CPRG Visit - TBD
          ARL-UT Visit - TBD
          IUSS Visit - TBD
          ONI Visit - TBD
          List SPA, CNA, John Hopkins, and related projects that have already been
           completed that we could learn from - TBD
          Investigate ASW cross functional team findings for potential metrics
           information – TDB

Link to the tasking orders for more information regarding symposium action items and
the POA&M

7. Acknowledgements

The success of the ASW Metrics Symposium was due to the hard work and dedication
of everyone involved. We would like to specifically thank:

      CAPT Jim Berdeguez (ASW DOO) for developing the concept for and hosting the
      LT Scott Parker (ASW DDOO) for administering the symposium
      Tom Murphree for developing the symposium plan, agenda, and presenters list
      Marcus Speckhahn, Josie Fabre, CDR (sel) Brett Martin, and Bruce Ford for
       taking careful notes of the symposium
      Tom Murphree, Bruce Ford, CDR (sel) Brett Martin, CAPT Jim Berdeguez, and
       LT Scott Parker for preparing the symposium summary and tasking orders
      Bruce Ford and Tom Murphree for developing the METOC metrics for ASW
       symposium web site at:


To top