UFS EVALUATION METRICS WORKSHOP: A KEY FOR TRANSITIONING NEW SCIENCE INTO OPERATIONS

Tara Jensen

The Developmental Testbed Center (DTC), in collaboration with the National Oceanic and Atmospheric Administration (NOAA) and the Unified Forecast System’s Verification and Validation Cross-Cutting Team (UFS-V&V), is hosting a three-day workshop to identify key verification and validation metrics for UFS applications. The goal is to identify and prioritize key metrics to apply during the evaluation of UFS research products, guiding their transition from research-to-operations (R2O).
The workshop will be held remotely 22-24 February, 2021. Approximately 275 participants have registered for this event from across the research and operational community. Because all UFS evaluation decisions affect a diverse set of users, workshop organizers are encouraging members from government, academic, and private-sector organizations to participate in the workshop.

In preparation for the workshop, a series of three pre-workshop surveys were distributed to interested parties. The results of the surveys have been used to prepare the discussion points of the breakout groups to streamline the metrics prioritization process. Pre-workshop survey results are published in the DTC UFS Evaluation Metric Workshop webpage. The organizing committee used the outcome of the 2018 DTC Community UFS Test Plan and Metrics Workshop and 2021 pre-workshop surveys to form the foundation of the workshop.

The resulting agenda is kicked off with presentations by Dorothy Koch, Director of NOAA’s Office of Science and Technology Modeling Division, and UFS Steering Committee Co-Chairs Ricky Rood, from the University of Michigan, and Hendrik Tolman, from NOAA. The first day features an opening plenary to curate the issues to be addressed in the breakout groups. The end of days 1 and 2 include breakout groups to allow for final community input into the application forecast challenges along with prioritization across all applications.

The workshop will end on day 3 with a final set of breakout groups to discuss how to apply the prioritized metrics to the full R2O development stages and gates. A wrap-up plenary rounds out a robust agenda. The breakout groups will focus on the nine UFS applications along with key forecast challenges identified in the pre-workshop surveys, as follows.

Model Applications

  • Short Range Weather (SRW)
  • Medium Range Weather (MRW)
  • Sub-Seasonal
  • Seasonal
  • Atmospheric Quality and Composition
  • Coastal
  • Hurricane
  • Marine and Cryosphere
  • Space Weather
  • Land/Hydrology

Additional Key Forecast Challenges

  • Aviation
  • High Impact Weather (beyond hurricanes)
  • Weather Extremes
  • Data Assimilation

The 2021 DTC UFS Metrics Workshop will ensure that the community participates in the process of selecting and prioritizing the evaluation and diagnostics measures to be used for assessing the UFS. This is an essential part of the transition of UFS community innovations to operational forecasting systems at NOAA, and an important step to improve our Nation’s weather forecasting systems to the benefit of society.
———
Tara Jensen is the Developmental Testbed Center (DTC) Verification Team Lead at the National Center for Atmospheric Research (NCAR).

The Workshop Organizing Committee includes Tara Jensen (NCAR and DTC), Jason Levit (NOAA/EMC), Geoff Manikin (NOAA/EMC), Jason Otkin (UWisc CIMSS), Mike Baldwin (Purdue University), Dave Turner (NOAA/GSL), Deepthi Achuthavarier (NOAA/OSTI), Jack Settelmaier (NOAA/SRHQ), Burkely Gallo (NOAA/SPC), Linden Wolf (NOAA/OSTI), Sarah Lu (SUNY-Albany), Cristiana Stan (GMU), Yan Xue (OSTI), Matt Janiga (NRL), and the entire UFS V&V Team.