Day 1 Discussion
On day 1, we learned about the Space Weather prediction problem, became more familiar with the data, trained machine learning models for the problem, and performed some initial evaluations, and started investigating XAI.
Please have one of your team members reply in the comments to to each of these questions after discussing them with the team.
Here are the Space Weather Jupyter notebooks:
- More beginner oriented: https://github.com/ai2es/tai4es-trustathon-2022/blob/main/space/magnet_lstm_tutorial.ipynb
- More advanced user oriented: https://github.com/ai2es/tai4es-trustathon-2022/blob/main/space/magnet_cnn_tutorial.ipynb
The TAI4ES Space GitHub Readme page is here.
Discussion prompts
- The lectures today and introduction to the Trust-a-thon both emphasized the importance of thinking about trust and the end user while developing AI. In this reflection, describe how your group thinks about both the trustworthiness of AI and how the end user fits into the AI development process. Then discuss how this thinking came into play for your first day of work on the Trust-a-thon: How did you integrate your user and/or their needs today? How would the day’s work have gone if you didn’t have an end user or had been assigned a different one?
- Finally, the lectures also covered interdisciplinary collaboration, with this in mind take some time to think about how you worked together as a team: What went well and what could have been better? Did you integrate all team members? How will you keep doing things well the rest of the week and what is your plan for improving the areas that could have gone better?
- Can you describe the physical process between solar wind and ground geomagnetic disturbances? What is the Dst index primarily used for? [Hint: Large changes in solar wind velocity and density combined with the magnetic field oriented southward typically results in significant changes in the geomagnetic field near the Earth’s surface about an hour later.]
- Roughly 85% of the time, near Earth is geomagnetically quiet. How might these infrequent solar wind events make modeling their predicted effects challenging? How might you make an accurate model with very few extreme events/samples?
- How are the input data differently distributed? How are the input data correlated or uncorrelated with each other? How are they correlated with Dst?
- Based on what you’ve learned so far, what do you think your user will care about most? How do you think you can help your user view this model as trustworthy?
Day 1 6/27 Notes for Team 2
We are selecting Persona 1 “Precision Navigator for an International Research Institute” as our end user
24 hours in advance: whether it’s gonna happen
1 hours in advance: intensity of the geomagnetic storm
Q1: The lectures today and introduction to the Trust-a-thon both emphasized the importance of thinking about trust and the end user while developing AI. In this reflection, describe how your group thinks about both the trustworthiness of AI and how the end user fits into the AI development process. Then discuss how this thinking came into play for your first day of work on the Trust-a-thon: How did you integrate your user and/or their needs today? How would the day’s work have gone if you didn’t have an end user or had been assigned a different one?
Answer to Q1:
We have to make a time-sensitive accurate prediction 24 hours and 1 hours in advance
Most AI models don’t explain each component, so really hard to follow what each models are doing
Very important to keep end user in mind in the development process
An interesting aspect would be the end user’s feedback in the trustworthy AI process may not be often reliable if the technology is meant for public usage. So, in this case, we are lucky to have a precise need from the precision navigator as getting end users can be challenging. This would be challenging if we were assigned an end user with less technical expertise.
We need to understand 4 things:
1. Code component where the “whether” decision is made
2. Code component where the “intensity” decision is made
3. Time sensitive decision making component
4. Constraints against predicting too early which can be an unspoken need of the precision navigator
Without an end user, we may not have had a specific time period of decision making
Q2: Finally, the lectures also covered interdisciplinary collaboration, with this in mind take some time to think about how you worked together as a team: What went well and what could have been better? Did you integrate all team members? How will you keep doing things well the rest of the week and what is your plan for improving the areas that could have gone better?
Answer to Q2:
What went well and what could have been better?
Good to get 3 of us together in a Zoom room working together and speaking together.
Our different backgrounds (computer science, water resources engineering; atmospheric science) gave us diverse perspectives of the problem where we complemented each other’s knowledge and expertise while learning about new things.
Our different backgrounds (computer science, water resources engineering; atmospheric science) made us hard to understand the specific problem without background subject expertise. Getting logged into JupyterHub was a technical issue faced by many leading to some loss of time. Better computational resources that we could promptly could have been awesome.
Did you integrate all team members?
3 of us met on Zoom and integrated with our respective backgrounds. 1 team member asynchronously worked on Slack.
How will you keep doing things well the rest of the week and what is your plan for improving the areas that could have gone better?
It would be cool to run the Jupyter notebooks individually and use the 3-6 pm EST time slot for discussions to use the time efficiently.
Tuesday running: Saptarashmi or anybody else joining our Zoom
Wednesday running: Jeongwoo
Thursday running: Kathrin
Q3: Can you describe the physical process between solar wind and ground geomagnetic disturbances? What is the Dst index primarily used for? [Hint: Large changes in solar wind velocity and density combined with the magnetic field oriented southward typically results in significant changes in the geomagnetic field near the Earth’s surface about an hour later.]
Answer to Q. 3
Dst (disturbance-storm-time) is a measure of the severity of the geomagnetic storm in the space weather and is used to drive geomagnetic disturbance models
The efficient transfer of energy from solar wind into the Earth’s magnetic field causes geomagnetic storms due to impacts of the solar wind velocity and density. The resulting variations in the magnetic field increase errors in magnetic navigation which require precise decision on whether and how much will be the impact of such storms. There is a 1 hour delay between the solar winds and ground geomagnetic disturbance being reflected on the earth which justifies the prediction navigator’s need to know the intensity of the geomagnetic storm, 1 hour before impact.
Q4: Roughly 85% of the time, near Earth is geomagnetically quiet. How might these infrequent solar wind events make modeling their predicted effects challenging? How might you make an accurate model with very few extreme events/samples?
Answer to Q4:
This is a situation of low resource machine learning where the dataset is sparse and needs to be efficiently processed to make precise calculations. Low resource machine learning is challenging as we do not have much training data to learn about storms which can bias the model’s decision making capability.
A few solutions for an accurate model with very few extreme events/samples are
1. We equally sample features of situations when geomagnetic storms occur (n samples) and geomagnetic storms do not occur (n samples). This will lead to a fair training dataset at the cost of less number of training samples which can still impact the prediction quality.
2. We can weight the loss function based on the frequency of occurrence of geomagnetic storms to debias the training dataset.
3. We could generate synthetic datasets to balance the training data.
4. We can use sparse matrix representation like complex sparse row transformation on the training data
5. We could try semi-supervised learning to address the issue of incomplete labels when geomagnetic storms do not occur.
6. We can do ensemble learning with different models aggregated together to boost user confidence
7. We can apply episodic learning with reinforcement learning techniques to reiterate the evolved strategies to predict geomagnetic storms till we get high rewards.
Q5: How are the input data differently distributed? How are the input data correlated or uncorrelated with each other? How are they correlated with Dst?
Answer to Q5:
The 3 input datasets share the same means but have different number of samples. The 3 input datasets represent the time series of solar wind measurements at L1 (Lagrangian 1 position). The 3 time periods in the respective datasets are 1998 to 2001 (train_a), 2013 to 2019 (train_b) and 2004 to 2010 (train_c).
Some features are correlated while some are not correlated. e.g. bx_gse and by_gse are strongly correlated but negatively (probably because they belong in separate dimensions). Density and temperature are kind of uncorrelated.
All the features in the dataset as listed below appear to be correlated with dst.
Features positively correlated with dst: bz_gse, theta_gse, bz_gsm, theta_gsm, density
Features negatively correlated with dst: bt, speed, temperature, smoothed_ssn
bx_gse
by_gse
bz_gse
theta_gse
phi_gse
bx_gsm
by_gsm
bz_gsm
theta_gsm
phi_gsm
bt
density
speed
temperature
source
Q6: Based on what you’ve learned so far, what do you think your user will care about most? How do you think you can help your user view this model as trustworthy?
Answer to Q6:
In terms of requirement, our user cares about whether a geomagnetic storm happens, what is the intensity of the storm and the specific timeliness of these decisions.
Feature engineering the model so that correlated features are considered together can be useful to make the user trust the model.
Balancing the dataset and making it fair to address sparse distribution of geomagnetic events will significantly boost the trustworthiness of the model.
Ensemble learning can also inspire user confidence by accounting for different distributions of the training dataset.
Hi Team 2, Great job and detail in your first post – sounds like a productive first day!
I think your discussions about the users are really insightful, specifically this comment: “An interesting aspect would be the end user’s feedback in the trustworthy AI process may not be often reliable if the technology is meant for public usage. So, in this case, we are lucky to have a precise need from the precision navigator as getting end users can be challenging. This would be challenging if we were assigned an end user with less technical expertise.”
If I understand this right, you’re pointing out that we don’t always know what users want and they might not either. This is a HUGE point for trustworthy AI development. It takes a lot of work early on in the development process to find out these needs and then work to meet them. We’re going to talk about this tomorrow and give some more detail about how we at AI2ES have been trying to tackle this issue.
Keep up the great work and looking forward to seeing your next posts!
Answer 1
Applications of a theoretically profounded model which is attractive to the end-user is trustworthy. Considering an AI model as an advanced statistical model, it is rather not a physical model based on a simplified but explainable using theory. Thus, we should make XAI trustworthy! Accommodating needs of end users integrated into the AI model development, objectives of the model should be elaborated. In addition, the use of the AI model should be tractable studying its interpretation and analysis.
Answer 2
All active team members were collaborating well. Working on own notebooks, we discussed theoretical and practical and issues. We also expect to do so later on.
Answer 3
A solar wind shock wave and/or cloud of magnetic field interacts with the Earth’s magnetic field and causes a temporary disturbance of the Earth’s magnetosphere that is known as a geomagnetic storm. The Dst index (Kyoto Dst, disturbance storm time), is used as a measure of the ring current strength around Earth caused by solar particles as follows: Earth’s magnetic field becomes weaker (Dst becomes negative) if the difference between solar electrons and protons gets higher, particularly during solar storms. It is used to drive geomagnetic disturbance models.
Answer 4
Different phases of solar activity, the latitudinal changes of the solar wind structure observed over the solar cycle and, at least, zones of the geomagnetic field make it challenging. Simulating samples using, e.g. Monte-Carlo, in the worst case or reconstructing samples using other indices.
Answer 5
High positive correlations are among bz_gsm and bz_gse, theta_gsm and theta_gse. The positive correlation is highest between speed and temperature. High negative correlations are among bx_gse and bx_gsm, by_gse and by_gsm, by_gse and by_gsm as well as phi_gse and phi_gsm. Negative correlation between speed and Dst is also high, and in absolute value it is the highest one. For Dst estimation, satellite location seems to be not important.
Answer 6
Engaging and working with your intended end-users, possible applications should be presented for cases being other than requested and so on.
Great start and looks like you are off to an awesome start!
I love seeing that you’re thinking about the end users, this is so important for making sure the AI is useful and will actually be used by the people we hoped would use it. I’d love to hear more about how exactly you would engage the end users and what you would do to learn and meet their needs. Would you host a workshop? Conduct interviews? Do a survey? Are there changes to the model you could do to increase trust?
Keep up the great work!