“When [weather forecasters] cannot easily understand the workings of a probabilistic product or evaluate its accuracy, this reduces their trust in information and their willingness to use it.” 

From Recommendations for Developing Useful and Usable Convection-Allowing Model Ensemble Information for NWS Forecasters (Demuth et al. 2020) 

Our risk communication team work on furthering our understandings of trust and trustworthiness in the context of AI and Earth system science. Our team’s goals include:

1. Increase knowledge and understanding of how transparency, explanation, reproducibility, and representation of uncertainty influence trust in AI for environmental science (ES) for influential user groups.
2. Develop models to estimate how attitudes and perception of AI trustworthiness influence risk perception and use of AI for ES
3. Develop principled methods of using this knowledge and modeling to inform development of trustworthy AI approaches and content, and the provision of AI-based information to user groups for improved environmental decision making.

To improve the reproducibility and transparency of supervised machine learning (ML) training data, our team has integrated quantitative content analysis (QCA) from social science into the hand-labeling for machine learning applications process. We applied and refined this approach through two winter weather projects: the New York State Department of Transportation road surface conditions model and the New York Mesonet nighttime precipitation detection model. This work addresses challenges in ensuring consistent labels across diverse datasets while facilitating interdisciplinary collaboration. Through this work, our team established a scalable, interdisciplinary framework for improving the quality and reproducibility of ML training datasets, enhancing their value for Earth system science applications.

Related Projects: Winter Road Weather/Decision Making, Precipitation Detection from Cameras

This project focuses on co-developing a machine learning (ML) tool for winter precipitation-type (p-type) nowcasting and prediction that is both trustworthy and usable by weather forecasters, transportation officials, and other partners. The team employs a mixed-methods approach—combining interviews and naturalistic observations—to understand users’ forecasting processes, informational needs, and decision-making related to p-type, as well as their perceptions of the trustworthiness and interpretability of the ML model.

Significant achievements include the development of interactive tools and model outputs, such as ensemble and timing predictions, for evaluation with National Weather Service (NWS) forecasters. Our team has also created a structured interview protocol to assess forecasters’ interactions with the model, including how they interpret model uncertainty and performance over time and space. This work will help refine the model’s trustworthiness, transparency, and utility in operational forecasting.

Related Project: Winter Weather P-Type Analysis and Forecasts

Our team is co-developing a think-aloud interview guide to identify factors influencing trust in AI/ML Loop Current Eddy (LC/LCE) prediction products among expert users, specifically metocean specialists from oil and gas companies operating in the Gulf of Mexico. Our work builds on insights from prior interviews with severe weather and coastal fog forecasters to assess which trust factors generalize across expert domains.

Related Project: Physics-Inspired AI for Offshore Oceanography