Focus 2: Use-Inspired Research in Environmental Science

Broad Goals

  1. Use trustworthy AI to provide actionable ES information to diverse users
  2. Enhance scientific and physical understanding of basic ES processes through trustworthy AI.  

These goals focus on five specific use cases to achieve the broader science goals of the Center. The AI2ES Risk Communication (RC) team will interact with the users of selected ES AI models and the development team to investigate and quantify users’ perceptions of, trust in, and uses of the AI models.   

Use Case #1 – Convective Weather 

Motivation: Thunderstorms hazards, e.g., tornadoes and hail, produce billions of dollars in damage and kill hundreds of people each year. Tornado warning skill has not significantly improved in the past decade.

Goals: Develop physically based Explainable AI (XAI) models to improve understanding of severe weather phenomena (tornado and hail formation). Facilitate the development of improved tornado and hail guidance.

  • Develop physically based, robust AI methods that improve convective hazard prediction.
  • Develop and use XAI methods to improve scientific understanding of convective hazards.
  • Develop physically-based, robust AI and XAI methods for convective hazard prediction that are deemed trustworthy and useful by forecasters.

Interpretation of convolutional neural networks for tornado predictionInterpretation of convolutional neural networks for tornado prediction. From McGovern et al 2020  and Lagerquist et al MWR 2020


Year 2
  • Convective weather is prototype use-case for the synergistic cycle with AI & RC
  • Researchers at OU, CMU, and NCAR are collaborating with IBM and Vaisala to develop a prototype short-term hail prediction system which will serve as a prototype for R2O transition for AI2ES
  • NCAR AI and RC researchers collaborated to solicit forecaster feedback and significantly improve performance and visualizations of ML storm mode product for real-time testing
  • Robustness testing for tornado project provided insights into model choices
  • Presented 6 presentations at AMS and NOAA AI conferences and 1 review paper in preparation
Year 1
  • Convective weather is our prototype use-case for the synergistic cycle with AI & risk communication
  • Submitted a paper summarizing our work to data on developing and using AI for convective weather.
  • Hosted two summer REU students (Lydia Spychalla and Jordan Robinson) who began the work of processing >100 TB of data to make the machine learning possible.
  • Presented work on hail and tornado forecasting using U-nets at AMS 2022.Working with one undergraduate student and one graduate student to continue the work on global hail nowcasting and tornado nowcasting
  • Storm centered image database extended to include ~200 severe weather days (including full 3d dual-pol measurements). This will be used for improving our deep learning tornado prediction models and for the development of XAI.


  • Leaders: Snook (OU), Gagne (NCAR), McGovern (OU)

    Members: Lagerquist (CSU/NOAA). Griffin (Disaster Tech). Becker, Demuth, Molina (NCAR). Flora, Kumler, Potvin, Stewart (NOAA). Chase, Diochnos, Homeyer, Nozka (OU). Bostrom (UW).

NY snowfall prediction graphicEstimated 24 hour snow accumulation derived from automated snow depth observations during the December 16-17th snowstorm. Image created by Nick Bassill using NYS Mesonet observations. Current data available here.

Use Case #2 – Winter weather

Motivation: Winter weather is a major hazard in the US, with societal impacts such as travel (road, air, rail) and utilities. An opportunity exists to increase resiliency through exploitation of weather data and AI.

Goal: Develop AI methodologies to exploit underutilized winter weather data (observations and forecasts) and non-meteorological sources (e.g., roadway data) to provide tailored guidance to emergency managers and decision makers.

  • Train graduate students and postdoc in interdisciplinary research and Explainable AI (XAI) methods for winter weather applications.
  • Improve our knowledge and understanding of the nature and predictability of winter weather from synoptic to local scale.
    • Use XAI to identify features most predictive for short-term forecasting of winter events 
  • Develop physics-based AI that improves winter precipitation forecasts including precipitation type and transitions.
    • Develop and use XAI methods using data from New York State Mesonet and Oklahoma Mesonet as well as other meteorological data and non-traditional data to create tailored analyses and predictions to support local and state-level transportation decision-making before and during winter weather events.
    • As above but working with the National Weather Service offices in New York and Oklahoma.
  • Quantify uncertainty of AI predictions for winter weather prediction, identify the relationship between uncertainty and trustworthiness, and communicate the uncertainty to And end-users
  • Develop models for estimating degree of visibility from Mesonet (or other) images. Construct individual models that can be used for a wide variety of image sources. Develop learning algorithms that are robust to variable trust labels.


Year 2
  • Winter Road Weather/Decision Making:
    • Building archive of approximately 2400 DOT camera images starting January 2022, data stored on xCITE servers
    • Preliminary CNN model built using 3 camera sites, achieved accuracy above 90%
    • Initial model results presented to DOT and were well-received; DOT shared feedback and discussed an end-product goal that would improve their current processes. This will facilitate inclusion of Risk Communication efforts on this project.
  • Winter Weather: Precipitation Detection from Cameras:
    • Successful hiring of a new Postdoc with appropriate background, and provision of training opportunities for 2 REU students
    • Worked with Risk Comm team to reliably hand-label over 44,000 NYSM camera images into four classes by four individuals
    • Achieved intercoder reliability on a subset of images, to evaluate for trustworthiness
    • Spatiotemporal matching of in situ measurements from NYSM instrumentation with camera imagery for verification
    • Model development on 7 different deep learning architectures with class-averaged accuracies above 90%
    • Model interpretability methods applied to predictions to understand important features
    • Operationalized model predictions as streams of images are generated for each station to be integrated into the NYSM website
  • Visibility Classification in Mesonet Images:
    • Constructing CNN models for two individual sites (rural and urban)
    • Developed data loading/transformation pipeline to address the large size of the Mesonet data set
    • Preliminary CNN models: class-weighted validation accuracy of 72.3%
  • Winter Precipitation-Type Project:
    • Established bi-weekly research group meeting on P-type which includes researchers from XAI, ES and RC
    • A research plan for the coming year has been created.
    • Trained baseline neural network models on ASOS and mPING data with pressure and height coordinate data to understand sensitivity to vertical coordinate system
    • Created project code repository:
Year 1
  • Camera Work:
    • Precipitation detection from NYSM cameras is underway, led by postdoc Lauriana Gaudet. A tremendous amount of work has gone into understanding the features within the images, and the best labels for training. Collaboration with the RC decision team has facilitated a trustworthy approach to the labeling process through the development of a codebook to reduce/eliminate human bias in labeling, requiring a trustworthiness threshold on the labeled images.
      Roadway precipitation detection is similarly underway and led by 1st-year graduate student Carly Sutter.
  • Forecast Verification:
    • An archive of data from the GFS, NAM, and HRRR models has been developed, with the appropriate filtering based on use needs. Analysis into the differences among the models is also being performed. In particular, it is found that continual updates to the GFS model may result in complications in developing statistics using this model, as the model statistics themselves may be inconsistent and change with updates. The significance of these differences is being investigated. Beyond this task, work is underway analyzing the biases in the various forecast models relative to NYSM surface observations, specifically, 2-m temperature, wind speed, and precipitation. This is being completed seasonally as well as a function of forecast hour. The intent is to develop a suite of ML models that predict the relative bias of a particular variable given a forecast input to guide forecasters in the trustworthiness of a particular model, at a particular location in NYS, time of year, event, etc.
  • NYS Mesonet for P-Type work:
    • The New York State Mesonet (NYSM) is providing AI2ES with unlimited access to its data and products to facilitate several research projects. NYSM data are being used to verify numerical weather prediction model output. Finally, the NYSM is leveraging NOAA grant NA21OAR4590376 to provide AI2ES with winter weather data and products. These data are being used as ground validation to improve the monitoring and prediction of precipitation type and other model-derived fields.

  • Leaders: Thorncroft (Albany), Sulia (Albany), Fagg (OU)

    Members: Bassill, Brotzge, Kurbanovas, Przbylo, Sutter, Torn, Tyle (Albany). Griffin (Disaster Tech). Becker, Demuth, Gagne, Gantos, Schreck, Wirz (NCAR). Harrison, Stewart (NOAA). Chase, Diochnos, McGovern, Rothenberger, Wilson (OU). Bostrom (UW).









Use Case #3 – Tropical Cyclones

Motivation: Tropical cyclones (TCs) have tremendous societal impact in terms of damage and flooding. 

Goal: Improve understanding of TC temporal evolution and rapid intensification to improve understanding and forecasting of TCs.

  • Develop a physics-based AI algorithm that generates synthetic microwave imagery from satellite data
  • Develop physics-based AI algorithms to improve TC prediction including: rainfall and wind-field estimation as well as and intensity changes
  • Use Explainable AI (XAI) to discover relationships between TC structure evolution and rapid intensification onset

tropical cyclone prediction improvement with AI

 Adapted from C. J. Slocum and J. Knaff, Using Geostationary Imagery to Peer through the Clouds Revealing Hurricane Structure, AMS annual meeting, 19th Conference on Artificial Intelligence for Environmental Science, Wed, Jan 15, 2020



Year 2
  • Developed CNN approach to simulate microwave imagery and characterized CNN model behavior:

    • Quantified errors by brightness temperature, storm type, and distance from storm center
  • Added capability to estimate uncertainty along with central prediction

    • Evaluated uncertainty estimates using spread-skill and PIT diagrams
    • Created method of viewing uncertainty using overlaid X’s
  • Extending method to night-time predictions:

    • Started performing feature and model search to improve night-time predictions
Year 1
  • Hired a research scientist (Jan 2021) and a postdoc (July 2021).
  • Collected and pre-processed data. 
  • Developed simple, pixel-based, fully connected neural network algorithm.  Performed hyper-parameter optimization.
  • Developed CNN for image-to-image translation. Optimization in progress.
  • Currently evaluating algorithms, including comparison to existing random forest algorithm. The CSU team improved their AI algorithm to generate simulated Microwave imagery from geostationary satellite imagery and presented their progress at the AMS annual meeting in Jan 2022 (Haynes et al., 2022).

Leaders: Ebert-Uphoff, Musgrave (CSU), Thorncroft (Albany), Gagne (NCAR)

Members: Haynes, McGraw (CSU). Griffin (Disaster Tech). Demuth (NCAR). Kumler, Stewart (NOAA). Hall (NVIDIA). Bostrom (UW).

Use Case #4 – Subseasonal to Seasonal (S2S) Prediction of Extreme Weather

Motivation: S2S prediction will improve resiliency as the climate changes.

Goal: Predict extreme weather two weeks to two months ahead (see Congressional Weather Research and Forecasting Innovation Act of 2017).

  • Develop physics-based AI to improve prediction of extreme weather at S2S scales
  • Develop and use Explainable AI (XAI) and interpretable methods to identify sources of predictability (and their physical mechanisms) on S2S timescales.
  • Develop and use XAI methods to leverage imperfect dynamical model simulations, along with observations, to make more accurate S2S predictions of the real world at the S2S timescales (i.e. through transfer learning)

Sub-seasonal to seasonal scale gapsAdapted from the Subseasonal Prediction Project, Earth Institute, Columbia University.


Year 2
  • Published two papers to JAMES on “abstention networks” for skillful forecasts of opportunity
  • Paper under review on Interpretable AI using a “this looks like that” prototype network with applications to climate prediction (Barnes et al., under review)
  • GRA has successfully coded an analog S2S model to act as our baseline for the ProtoLNet (interpretable prototype network)
  • GRA has successfully gotten the ProtoLNet working with S2S data from the CESM2 pre-industrial control simulation to predict temperatures on S2S timescales.
Year 1
  • Started obtaining hindcast and climate model data for developing, testing and implementing transfer-learning framework.
  • Started research on applying transfer learning between climate models and ERA5 observations to improve S2S predictions of precipitation and temperature
  • Submitted two papers to JAMES (on arxiv as well) on “abstention networks” for skillful forecasts of opportunity
  • Wrote blog post sharing new abstention network research
  • “This Looks Like That There” paper submitted for review in AIES. This paper lays the groundwork for the machine-learning-enabled analogue S2S forecasting approach that we are developing.
  • Multiple invited talks on applying abstention networks and neural network uncertainty quantification to climate/S2S applications.

Leaders: Barnes (CSU), Ebert-Uphoff (CSU), Tissot (TAMUCC)

Members: Hickey (Google). Williams (IBM). Demuth, Molina (NCAR). He (NCSU). Stewart (NOAA). Vicens Miquel (TAMU-CC). Bostrom (UW).

Use Case #5 – Coastal Oceanography

Motivation: Coastal phenomena impact humanity on a regular basis.  Improving our prediction of coastal events will save lives and property.

Goal: Develop physics-based AI methodologies to provide more accurate predictions of Ocean eddy shedding, Harmful algal blooms, and Compound flooding.

  • Create shared multivariate high-resolution data set for the Gulf of Mexico and U.S. east coast shelf seas that will support the specific use-cases and AI
  • Design a common deep learning platform allowing for the spatio-temporal characterization of air-sea-land interactions while accommodating different spatial resolutions
  • Implement, test, and validate (with RC) the AI predictions with stakeholders
  • Develop physics-based AI to improve the prediction and understanding of:
    • Cold-stunning events of sea turtles in the Laguna Madre
    • Timing and intensity of marine fog
    • Coastal beach flooding (cross-cutting with TC) 
    • Compound flooding from storm-surge and rainfall (cross-cutting with TC)
    • Loop Current Eddy shedding in the Gulf of Mexico
    • General trend, timing, and locations of harmful algal blooms

Graph of green turtle count.

Estimated green turtle abundance over time, compared with water temperature. Courtesy of Dr. Amy McGovern, University of Oklahoma – Norman.


Loop Current

Loop Current pinched-off eddy in the Gulf of Mexico. over 4,000 oil and gas rigs in the northern Gulf waters are at risk of damage from these eddies. From the CNAPS model of the Ocean Observing and Modeling Group, North Carolina State University.



Year 2
  • FogNet – Coastal Fog Predictions:
    • Paper “Importance of 3D convolution and physics on a deep learning coastal fog model” applying XAI to FogNet (Kamangir et al.) accepted in  Environmental Modeling and Software (
    • FogNet results including XAI are used as part of a collaboration with risk communication. 
    • The collaboration with risk communication has led to shifting FogNet to probabilistic outputs.
    • As part of a collaboration with IBM a VAE alternative to FogNet is being explored (would be easier to implement operationally).
  • CNAPS:
    • Have formed an AI team at NCSU that consists of two postdocs (Anna Lowe, Naz Chaichitehrani) and two graduate students (Laura McGee and Michael Gray) to work on AI2ES ocean research problems.
    • A 28-years (1993-2020) data assimilative ocean circulation reanalysis has been produced. This dataset is being used for AI model training and testing.
    • Have generated AI/ML models and initial results & analyses of 3.1 (Loop Current eddy) and 3.2 (HAB)
    • 4 meeting presentations at international and national conferences
  • Coastal Inundation Predictions:
    • Created a dataset of over 3,000 drone based coastal images from 12 locations in Florida and Texas with labeled shoreline wet/dry line including elevations and Deep Learning predictions.
    • Undergraduate students developed water level predictions visualizations for stakeholders and scientists.
    • A set of cameras including a stereo camera are being installed on a local pier to provide point cloud time series
    • A peer reviewed conference paper (Vicens-Miquel et al.) “’Deep Learning Automatic Detection of the Wet/Dry Shoreline at Fish Pass, Texas” was accepted for IGARSS 2022 proceedings about to be followed by journal paper submission.
  • Sea Turtle Conservation Models:
    • The AI operational prediction model (shallow neural net) was extended to 120 hours lead time facilitating the decision process ahead of cold stunnings.
    • The team provided predictions and guidance for the determination and adjustment of the start and stop of navigation interruptions during a February 2022 cold stunning event.
    • Continued preparation of the stakeholder engagement including tentative agreement from stakeholders such as the Texas Marine Cold-water Response Collaborative and Texas Parks and Wildlife to participate.
Year 1
  • The paper describing FogNet (Kamangir et al.) was accepted “FogNet: A Multiscale 3D CNN with Double-Branch Dense Block and Attention Mechanism for Fog Prediction” in  Machine Learning with Applications 
  • The performance of FogNet was compared to operational models, HREF, SREF showing substantial performance improvement.
  • Two other journal submissions in review
  • Working with IBM on potential R2O VAE implementation of coastal fog predictions
  • Tested Channel-Wise PartitionSHAP on 13-channel EuroSAT dataset (FogNet)
  • Created experiment where adding an additional channel (from RGB to RGB & NIR) expected to allow CNN to learn strategy. Channel-Wise PartitionSHAP results strongly suggest that model exploited the additional channel (FogNet)
  • Development on visualization tool progressing (FogNet)
  • Used on analysis of FogNet paper currently under submission (FogNet)
  • Labeled over 3,000 coastal images (Coastal Inundation Predictions)
  • Processing of images from Florida and Texas (Coastal Inundation Predictions)
  • Developed methods to label orthomosaics with ESRI software (Coastal Inundation Predictions)
  • Undergraduate students developing water level predictions visualizations for stakeholders and scientists (Coastal Inundation Predictions)
  • Completed first drone flight to acquire beach imagery including wet/dry line. Ready to acquire more imagery in different conditions (Coastal Inundation Predictions)
  • Provided guidance including recommended start and stop of navigation interruptions in the Laguna Madre during the largest sea turtle cold stunning event in recorded US history (February 2021).
  • Developed visualisations of IBM ensemble predictions and NWS predictions to help with cold stunning prediction guidance (Sea Turtle Conservation Models)
  • Held meeting with state and federal agency stakeholders to review predictions. Invited presentation at industry navigation association annual meeting (Sea Turtle Conservation Models)
  • Long-term ocean observations including both remote sensing and in situ ocean data have been collected. A 26-years (1993-2018) data assimilative ocean circulation reanalysis is being produced. These observations and numerical model output will be used for AI model training and testing starting in fall 2021
  • Self-organizing map (SOM) method was applied, which can effectively cluster LC patterns (Meso-scale Ocean Eddy Prediction)
  • A preliminary  ML (ANN) model for the LC variations and eddy shedding process has been developed (Meso-scale Ocean Eddy Prediction)
  • Further validation and refinements of this LC ML model is ongoing (Meso-scale Ocean Eddy Prediction)
  • One NCSU PhD student (Laura McGee) has been working on this ML technique and remote sensing data analysis as a part of her Ph.D. dissertation (Cloud-free satellite marine data reconstructions)
  • Based on MODIS observations, we have successfully reconstructed a 18-year (2003-2020) cloud-free, daily time series for SST, CHL, POC, and PIC for the US Atlantic coast and the GoMEX waters (Cloud-free satellite marine data reconstructions)
  • The product is being used for feature detection and extreme event prediction (Cloud-free satellite marine data reconstructions)

Leaders: He (NCSU), Tissot (TAMUCC), Williams (IBM), McGovern (OU)

Members: Griffin (Disaster Tech), Demuth, Gagne (NCAR). Gray, Lowe, McGee, Warrillow, Wu, Zambon (NCSU). Collins (NOAA). B. Colburn, K. Colburn, Duff, Durham, Estrada, Huang, Kamangir, King, Krell, Medrano, Nguyen, Starek, Vicens Miquel, White (TAMUCC). Demuth, Gagne (NCAR). Bostrom (UW).