JEDI Code Sprint at JCSDA for UFO Development to Match GDAS: Implications for UFS Applications

Katherine Shanahan, Dick Dee, and Hui Shao

The Joint Effort for Data assimilation Integration (JEDI) project work is set to be the foundation for performing data assimilation across many different kinds of UFS applications. JEDI is an innovative data assimilation framework that provides shared and portable software with generic components that can be configured for a variety of operational-grade applications in Earth system analysis and prediction. This joint project is carried forward by the Joint Center for Satellite Data Assimilation (JCSDA) with several community partners such as NOAA, NASA, DoD, NCAR, and the UK Met Office.

Like the UFS, the JEDI project has a strong community focus, allowing software engineers and scientists alike to download and run public code, access support forums, and contribute to new developments. The ultimate goal is to leverage collaborative community development to craft a truly state-of-the-art unified data assimilation system. Sharing resources will enable a faster transfer of research to operations so that JCSDA stakeholders and the data assimilation community can utilize JEDI in transitions to next-generation prediction systems.

Recently, efforts have ramped up to accelerate ongoing work within the JCSDA to build and expand operational-grade capabilities for handling observations in JEDI. An important milestone in the transition of any current operational data assimilation system to a JEDI-based system is to demonstrate that JEDI can handle observations as well as (or better than) the current system. Such a milestone was achieved during two successful 2-week code sprints, conducted jointly by JCSDA core staff and NOAA contributors, that served to greatly advance the capabilities of the observation component of the Joint Effort for Data Assimilation Integration (JEDI). This is a critical step in the right direction towards building the next-generation operational systems based in JEDI for use in UFS applications.

The use of observations in JEDI is generally handled by the Unified Forward Operator (UFO) software component. The function of the UFO is to represent, as well as possible, the precise relationship between what is observed and what is modeled, so that the information provided by the observations can be used to improve the model forecasts. Implementation of UFO for a specific observing instrument requires detailed information about the instrument, the measurement technique, measurement uncertainties, etc. It involves a simulation of each individual measurement, with possible bias adjustments, as well as robust quality control steps to prevent the use of observations that are not physical or cannot be adequately represented by the model.

One of the key deliverables in the current JCSDA Annual Operating Plan is a set of UFO configurations that implement the use of observations in the Global Data Assimilation System (GDAS) operated by NOAA’s National Center for Environmental Prediction (NCEP). The specific goal for the JEDI code sprint was to replicate as closely as possible the current capabilities of the Gridpoint Statistical Interpolation (GSI), which is the component of the GDAS responsible for handling and analyzing observations. To achieve this, it was necessary to produce specific UFO configurations for each of the observing instruments used in GDAS and to verify in detail that they produce equivalent results, in terms of the output of forward operators, bias adjustments and quality control steps.

GDAS currently uses millions of observations daily, obtained in real time from more than 20 different satellite sensors and a large variety of surface stations, aircraft, radiosondes, etc. The work accomplished during this code sprint is a prime indicator that JEDI can handle any observing instrument currently used in global forecasting applications, with a performance that matches that of GSI.

graphic of bias-corrected

Figure 1. This figure shows the differences between bias-corrected and quality-controlled brightness temperatures generated by JCSDA’s JEDI UFO and NCEP’s Global Data Assimilation System (GDAS), respectively, for the Spinning Enhanced Visible and InfraRed Imager (SEVIRI) channel 5. The differences are extremely small, which demonstrates that the JEDI UFO can replicate the current operational GDAS forward operator capabilities for this sensor.

scatter plot

Figure 2. This scatterplot depicts the comparison between the final observation errors generated from the JEDI UFO and GDAS. The observation errors are inflated through various quality control checks implemented in both systems. It shows that the JEDI UFO matches GDAS’s QC procedures, which leads to equivalent observation inflations.

The JCSDA core staff and NOAA collaborators have worked closely together throughout the year on the very complex process of checking every single detail, for all satellite platforms, sensors and channels, as well as for all conventional instruments used in GDAS to replicate the forward operators, bias corrections, and quality control steps in JEDI. The 4 weeks of code sprint greatly expedited progress by consolidating and extending previous work, using an agile software development approach with systematic testing protocols.

GDAS is a prime example of an operational application that is successfully transitioning to JEDI, and the JCSDA is ready to leverage and extend this work to many UFS other applications. With additional capabilities already provided by various JCSDA partners, rapid improvements and innovations towards the goal of a world-class unified data assimilation framework are within reach, signaling exciting implications for the collaboration between UFS and JEDI.


Katherine Shanahan is a program coordinator at JCSDA, Dick Dee is a senior research scientist at JCSDA, and Hui Shao is the JCSDA UFO lead.