Focus 1: Foundational Research in Trustworthy Artificial Intelligence / Machine Learning
Broad Goals
1. Develop explainable AI (XAI) methods aligned with environmental science domain perspectives and priorities
- Develop XAI and interpretable methods for ES data (including regression-based predictions, data with high spatiotemporal autocorrelations, and fielded data)
- Develop XAI and interpretable methods that integrate physics into the explanations
- Develop XAI methods to explain AI model failures
- Develop XAI methods that facilitate knowledge and hypothesis discovery
- Develop XAI approaches to effectively communicate (measured through RC research) estimated uncertainty to the end user and tailor these to the needs of the end-user
Real-time storm morphology prediction with self-supervised Convolutional Neural Networks (CNNs). Built real-time visualization identifying supercells and squall line segments for forecasters from CNN trained on proxy task.NCAR Realtime Forecasts
Application: Convolutional Neural Network analysis of severe hail storms. Saliency maps identify different types of storms that produce severe hail and enable generation of storm type distributions.Interpretable Deep Learning for Spatial Analysis of Severe Hailstorms (Gagne et al. 2019)
Progress:
Year 4
- Published paper (Rader and Barnes, 2023) in Geophysical Research Letters oninterpretable neural networks for climate prediction across timescales (includingS2S) using optimal analogs with a machine-learned spatially-weighted mask.
- Published in a NeurIPS 2023 workshop (Stock and Anderson, 2023) andpresented in an Oral Spotlight on a machine learning method to model human eyegaze to localize salient regions in images for classification tasks.
- Published a paper (Krell et al., 2023) in Environmental Data Science comparing
different levels of aggregation of XAI outputs for a 3D CNN model, Fognet, in
addition to presenting this work at AMS and AGU conferences. - Developed and published a tutorial in Artificial Intelligence for the Earth Systems
on XAI methods for Atmospheric Sciences in collaboration with OU, CIWRO,
and NOAA/NSSL (Flora et al., 2024).
Year 3
- Published paper on Interpretable AI using a “this looks like that” prototype
network with applications to climate prediction (Barnes et al., 2022). - NCAR visitor Da Fan preparing paper on “Physically Explainable Deep Learning for Convective
Initiation Nowcasting Using GOES-16 Satellite Observations” with different SHAP backgrounds
highlighting different cloud features. - NCAR presentations at ECMWF Workshop and AMS AI Conference on explainable uncertainties for storms and p-type.
- Developed a novel artificial neural network architecture
that allows the partitioning of the pixels of a given image to different classes,
thus enabling regions of explanations that are non-overlapping and not
necessarily contiguous with applications for detecting winter-weather road conditions. Work in submission (Rothenberger et al., 2023).
Year 2
- Developed a 2nd benchmark to evaluate XAI methods for ES applications, this time for CNNs (paper submitted, figure above).
- Two papers published on abstention networks for regression and classification tasks (Barnes and Barnes (2021a,b).
- Paper under review on Interpretable AI using a “this looks like that” prototype network with applications to climate prediction (Barnes et al., under review)
- Poster presented at ICLR workshop on interpretation of novel cascade network structure for modeling climate change (Anderson and Stock, 2022)
- Developed tutorial on uncertainty quantification (UQ) in AI methods for ES applications (TAIES summer school 2022).
Year 1
- Developed benchmark to evaluate XAI methods for ES applications (paper under review, figure above).
- Paper: Analyzing Severe Storms in a Future Climate with XAI
- Created PyMint XAI Python package
- NCAR AI group received ASOS and MPING precip type data from Kim Elmore, matching data with RAP soundings.
- NCAR AI Research thrusts: time consistency and uncertainty in ML methods, uncertainty in XAI outputs by combining XAI and UQ ML methods. We have assembled three benchmark datasets and plan to evaluate the sensitivity of XAI results to data and model perturbations.
Leaders: McGovern (OU), Gagne (NCAR), Ebert-Uphoff (CSU), Barnes (CSU)
Members: Bassill, Kurbanovas (Albany). Anderson, Bansal, Gordillo, Haynes, Lee, McGraw, Musgrave, Stock (CSU). Lagerquist (CSU/NOAA). Becker, Demuth, Gantos, Molina, Schreck (NCAR). Potvin, Stewart (NOAA). Hall (NVIDIA). Diochnos, Fagg, Homeyer (OU).Kamangir, King, Krell, Tissot, Vicens Miquel (TAMUCC). Bostrom (UW).
2. Develop physics-based AI techniques for environmental science domains
Physical constraints and physics-based AI/ML give us:
- Robust feature creation
- Physical, semantic, and redundancy constraints on generated features
- Physics-constrained loss functions
- Conditional hybrids of physical model and AI system predictions
- Novel architectures for XAI
Overall Goal: Improved AI predictions and understanding of physically-based ES phenomena
- Ensure AI methods always produce produce physically plausible and consistent solutions
- Develop physics-guided approaches to autonomous feature discovery
- Develop hybrid models that incorporate physics-based AI
- Develop geometric AI models that can learn features across different spatial and temporal resolutions
Progress:
Year 4
- Published paper (White et al., 2024) in Artificial Intelligence for the Earth
Systems that uses domain knowledge for AI-based super resolution of satellite
imagery. - Published and presented paper (Stock et al., 2024) at ICLR 2024 in collaboration
with NVIDIA on diffusion models to generate realistic convectively coupled
tropical disturbances across daily to multi-week timescales. - Published paper (Chaichitehrani et al., 2024) in Artificial Intelligence for the
Earth Systems on ocean reanalysis and AI learning for ocean wave prediction. - Presented work at AMS (Moen et al., 2024) on exploring mathematical methods
for texture recognition in meteorological imagery.
Year 3
- Presentation at AMS Annual Meeting on rotationally-invariant deep learning.
- Submitted a paper that compares different levels of aggregating XAI outputs for
3D CNN geoscience models with correlated, high-dimensional raster inputs
(Krell et al., Environmental Data Science, 2023). - Tutorial paper published (Ver Hoef et al. 2023) on using Topological data analysis (TDA) to extract physical properties from meteorological imagery.
- Tutorial paper published (Lagerquist et al., 2023) on creating and evaluating uncertainty estimates with neural networks for environmental-science applications.
- Started a working group including NVIDIA, Google, and multiple universities on investigating how to improve sharpness (minimize blurriness) in meteorological imagery. First sharpness paper in preparation.
Year 2
- Explored the use of topological data analysis (TDA) to extract physical properties from meteorological imagery (paper submitted to AIES journal, May 2022)
- Explored use of meteorologically meaningful loss functions for neural networks:
- Prior year: Developed practical guide for meteorologists, posted on arXiv in June 2021.
- This year: Explored use of Fourier and wavelet transforms in loss functions (paper accepted in journal AIES subject to major revision, May 2022)
- Evaluating the effect of changing the input vertical coordinate system on estimating precipitation type
Year 1
- Developed a neural network algorithm with physics-based components for estimating radiative transfer using AI (figure above), published July 2021
(https://journals.ametsoc.org/view/journals/atot/aop/JTECH-D-21-0007.1/JTECH-D-21-0007.1.xml). - Developed a tutorial for implementing meteorologically meaningful loss functions for neural networks
(https://arxiv.org/abs/2106.09757). - Evaluated temporal consistency of ML predictions and explanations
- The CSU team conducted experiments with spatially enhanced loss functions for neural networks that utilize neighborhood filters or spectral filters (using Fourier or Wavelet transforms). Lagerquist and Ebert-Uphoff are currently in the process of writing up the results for a journal paper.
- Gagne and Ebert-Uphoff are coordinating a visit by a CSU Mathematics PhD student to investigate the use of geometric deep learning on AI2ES use cases.
Leaders: McGovern (OU), Gagne (NCAR), Tissot (TAMUCC), Ebert-Uphoff (CSU)
Members: Anderson (CSU), Barnes (CSU), Becker (NCAR), Collins (NOAA), Gantos (NCAR), Gordillo (CSU), Griffin (Disaster Tech), Hall (NVIDIA), Homeyer (OU), Kamangir (TAMUCC), Kimpara (NCAR), Kurbanovas (Albany), Lagerquist (CSU/NOAA), Lee (CSU), McGraw (CSU), Musgrave (CSU), Potvin (NOAA), Schreck (NCAR), Snook (OU), Stewart (NOAA), Stock (CSU), Sulia (Albany), Ver Hoef (CSU)
3. Develop robust AI prediction techniques, and empirically and theoretically validate their performance with adversarial data (e.g., missing data or intentionally wrong data).
Some ES datasets are limited by data collection challenges and the rareness of extreme events. AI2ES will address robust AI (theoretically and empirically) with the following approaches:
- Class imbalance: characterize theoretical sample size needed to meet important verification metrics
- Transfer learning: train ML on common phenomena and transfer to rare events; train within simulations and transfer to observations
- Self-supervised learning: identify proxy supervised learning tasks that most aid in prediction when reliable labels are expensive or unavailable
- Adversarial Data/Classifier Robustness: provide theoretical guarantees for robustness of ES ML models to intentionally or accidentally adversarial data
Goals:
- Develop robust semi-supervised and unsupervised learning algorithms for situations where reliable labels are not available
- Develop theoretical and practical bounds on the robustness of the AI methods given class imbalance, a lack of reliable labels, and for adversarial situations (e.g. data may be missing or corrupted based on weather conditions)
- Ensure AI methods are robust to noise (e.g., missing data or (un)intentionally wrong data, different regimes)
Progress:
Year 4
- Published paper (Barnes et al., 2023) on a method for adding uncertainty to a
neural network regression task. This methodology is being tested in operations Summer 2024 at the National Hurricane Center. - Presentation at ISAIM 2024 (Rothenberger and Diochnos, 2024) on a novel
method of meta-learning which uses multiple representations of a data set to improve few-shot classification accuracy on image classification
tasks. - NSF NCAR MILES group developed a new package, MILES GUESS for robust machine
learning uncertainty quantification using evidential neural networks and ensemble methods (Schreck et al. 2023).
Year 3
- Paper accepted on a simple method for adding uncertainty to neural
network tasks with applications to tropical cyclone forecasting (Barnes et al.
2023) - NYS Mesonet symposium and AMS presentation on semi-supervised learning
using self-training with winter-weather application on predicting road surface
conditions, based on REU student’s summer work (Ferrera et al. 2023). - Paper on semi-supervised machine learning algorithm to analyze storm mode
(Sobash et al. 2023). - Presentations at ECMWF workshop, AMS Annual Meeting, and World
Meteorological Congress on estimating aleatoric and epistemic uncertainty with
ML and how it can be used in different applications. - Iterative analysis of ML p-type predictions vs crowd-sourced and simulated labels revealed
physical inconsistencies and adversarial labels in mPING p-type observations. Presented work at WINTRE-MIX workshop.
Year 2
- Algorithms: Effect of regularization methods as defense mechanisms in situations where we have randomly corrupted datasets (datasets of reduced size). Improved semi-supervised storm mode algorithm to be competitive with fully-supervised approaches.
- Data: Experimental evaluation using a (high-dimensional) wind-prediction dataset by McGovern et al (2021).
- Publications: Published results at AAAI conference (student abstract) – Flansburg and Diochnos (2022).
- Publications: Barnes and Barnes (2021a,b) published in JAMES
- White Papers: posted paper on arxiv summarizing how to add uncertainty to neural network regression tasks that account for heteroscedastic, asymmetric uncertainties
Year 1
- Developed and refined methods for neural networks to say “I Don’t Know” and learn the more confident samples better, termed controlled abstention networks (CAN)
- Submitted two papers on CAN (Barnes and Barnes 2021a,b) to JAMES for earth science applications
- Subdivision algorithm for computing a lower bound on the rate of the minority class
- Reduction of a learning problem with high recall and high precision, to a learning problem with improved risk bound, within Probably Approximately Correct (PAC) learning
- Paper in SIAM International Conference on Data Mining, SDM 2021
- “Wind Prediction under Random Data Corruption (Student Abstract)” paper accepted in the Student Abstract track of the AAAI Conference on Artificial Intelligence (AAAI), 2022. This is the outcome of the REU work of Conner Flansburg with Dimitris Diochnos for the summer of 2021.
Leaders: McGovern (OU), Gagne (NCAR), Diochnos (OU)
Members: Anderson (CSU), Barnes (CSU), Bassill (Albany), Becker (NCAR), Cabrera (OU), Ebert-Uphoff (CSU), Fagg (OU), Hall (NVIDIA), Kadiyala (OU), Kimpara (NCAR), Kurbanovas (Albany), Lagerquist (CSU/NOAA), Le (OU), McGraw (CSU), Molina (NCAR), Potvin (NOAA), Rothenberger (OU), Sasser (OU), Schreck (NCAR), Stewart (NOAA), Sulia (UA), Sutter (UA), Thorncroft (UAlbany).
4. Develop the field of AI ethics for ES applications.
An important cross-institute research focus is examining and raising awareness of ethical issues in AI for the environmental sciences. Our goals for this new effort include developing new research in AI ethics for ES applications and publishing and presenting about these efforts to spread the awareness across the AI and GEO communities.
Goals:
- Highlight the need for responsible and ethical AI for weather and climate.
- Develop a framework for identifying and characterizing biases in AI in the environmental sciences
- Develop and demonstrate methods to mitigate biases in AI for weather and climate
- Develop a framework for responsible AI development for weather and climate applications
Progress:
Year 4
- Published paper (McGovern et al., 2024) in Bulletin of the American on identifying and categorizing bias in AI/ML for earth science applications.
- The team gave multiple invited talks on ethics for AI for weather and climate, and was
represented on a strategic planning committee for AI at the University of Washington.
Year 3
- Participated in the AGU workshop and report on the development of ethical and
responsible AI/ML for the Earth Sciences. - Gave multiple presentations on responsible and ethical AI/ML for Earth and
environmental sciences (AMS 2023, ECMWF MOOC, UK Met Office). - Preparing a paper on bias in AI/ML for Earth and environmental sciences.
- Participated in the AI institute ethics conference at Georgia Tech in 2023. Six
members were sent and were selected to give a 20 min presentation on our work. - Our first paper (published in Y2) remains the paper with the highest attention score in the
Journal of Environmental Data Science (a Cambridge Core journal).
Leaders: McGovern (OU), Bostrom (UW), Gagne (NCAR)
Members: Chase (CSU), Ebert-Uphoff (CSU), McGraw (CSU), Musgrave (CSU), Perez (OU), Schumacher (NCAR)(through year 4)