Day 1 Discussion
On day 1, we learned about the Tropical prediction problem, became more familiar with the data, trained machine learning models for the problem, and performed some initial evaluations, and started investigating XAI.
Please have one of your team members reply in the comments to to each of these questions after discussing them with the team.
See the Jupyter notebook for this topic.
Discussion prompts
- The lectures today and introduction to the Trust-a-thon both emphasized the importance of thinking about trust and the end user while developing AI. In this reflection, describe how your group thinks about both the trustworthiness of AI and how the end user fits into the AI development process. Then discuss how this thinking came into play for your first day of work on the Trust-a-thon: How did you integrate your user and/or their needs today? How would the day’s work have gone if you didn’t have an end user or had been assigned a different one?
- Finally, the lectures also covered interdisciplinary collaboration, with this in mind take some time to think about how you worked together as a team: What went well and what could have been better? Did you integrate all team members? How will you keep doing things well the rest of the week and what is your plan for improving the areas that could have gone better?
From team 34:
1. Trustworthiness and user involvement in the AI development process are both important since the user is the one that will end up using the AI and they will need to have confidence in the final model. Today our group looked at a linear regression model and were worried that we wouldn’t be able to deliver some of the user’s needs because of certain shortcomings of the datasets. Specifically, there is only one wind variable for each timestep and both users want high spatial resolutions. Additionally, the model we saw today only predicts wind speeds well within a certain range whereas the users want a model that has high prediction accuracy for any wind speed. We found that the most important thing in this situation is to be honest with the user about advantages and disadvantages of the model which will build trust about the model’s capabilities and the AI development process as a whole. If we were’t assigned the users, we still would have been skeptical about the dataset but our focus would have been more centered on how that would affect the models developed rather than potential impacts to the specific use cases of the users.
2. We found that as a team we weren’t completely interdisciplinary since we all work in atmospheric science however we do have different research areas within that field. Overall, we thought that the communication between our team was good. We had a google meet with all available team members at the end of the day which facilitated good discussion about the model and its impact on the user’s needs. We plan on continuing this form of discussion throughout the week, especially when we explore the problem in greater detail.
Love that you brought up transparency with the user about advantages/disadvantages–no model, AI or otherwise, is ever going to be perfect, with zero drawbacks. One of the most useful things you can do is identify strengths, and weaknesses, and be HONEST about when you don’t know something! This transparency can also help you justify your own choices that you make as a scientist–so often, we have several options for how to accomplish a task, and more than one of those options will be a totally valid choice. One of the best things you can do for yourself and other scientists is to understand why you have made the choices you did and be able to explain that reasoning–even if someone else may have done it differently, they will probably understand how you got to the place you did.
Q. 1: The difference in the needs between the two end-users are identified, which informs the model’s requirements and objective, in addition to scoping for the type of data science problem the model is designed for (i.e., predicting binary outcomes versus multi-classifications). For instance, if the forecast end-user wants to roughly know the storm intensity, it’s sufficient to use small amounts of PCA components to make the prediction. But for the transportation end-users, who use the insights of the model to manage emergencies, larger PCA components are needed to better construct the structure for predicting locations of the possible convective systems (causing high winds and heavy rainfall). Strong wind could be caused by a very small-scale storm like tornado or even a mesoscale system like thunderstorm and derecho. If it’s a small-scale phenomenon — time scale is also small, the model we play is not appropriate to address this issue since the time resolution of satellite images is not sufficient. For mesoscale systems, radar images are better than the satellite images for developing the model. Since it is an issue related to ground transportation, we should have more data to help train the model.
These strong wind events (no matter tropical cyclones or strong winds over land) would not be handled well in the linear regression. Thus a classification problem solving framework (based on available attribute data in the dataset) before training might be better to predict those extreme events. Therefore, the varied needs of end-users not only define the definition of the problem, but the requirements of the input data’s spatial and temporal dimensionalities.
End-users, i.e. stakeholder engagement, is essential to define the model’s strategy and data schema. Furthermore, end-users can also contribute to the robustness and validity of modeling by calibrating the model’s outputs with expert judgment, and by informing modelers on specific insights required (metrics and KPIs). This means that the end-users can validate the actionability of predictive models. Finally, end-users are essential in the early stage of project planning, as they bring clarity to the scope, investment, and investment to method/model development.
Q.2: We realize that it is the Interdisciplinary nature of our team that really brings color to collaborative learning. Integration of ideas is one of the key aspects of this. We are definitely making progress towards it as we are parsing all team member’s inputs.
Understanding what each member hopes to learn and get out of the experience would be helpful so that we know how to help one another and keep making progress. There’s emphasis on working through the Jupyter Notebook and learning about the modeling methods featured, as well as discussing the blog post questions and coming up with synthesized answers. The main expectation is simply to engage in intellectual curiosity with the lectures and program’s content, and learn from one another by asking questions via common forums or meetings.
1) You are spot on with your points about different user needs impacting your model design choices. This is why it’s critical to work with your end users from the start, rather than just presenting them with a pile of results at the end (which I think is how many of us are used to working–I know that’s true for me!)
1. We chose to work with the Tropical Forecaster. Our group believes that in order for AI to be trustworthy, it must adhere to certain standards. We defined these standards as fairness, reproducibility, and overall accuracy/performance. In the context of our user, we must ensure that the model that they will be using equally represents the community that they are serving. Since observational data is not easily obtainable, the model must be able to generalize to the whole region and produce accuracy wind measurements so that the forecaster can served their community. When we were working through the notebook, we made sure to think about how our data is represented when passed through our model. The PCA component of the notebook took up a good amount of our time, since we were given the chance to sacrifice some accuracy in order to reduce the dimensionality and size of our dataset.
2. We set up a Zoom call for the 3 of us that were available from 1-4. We had a mix of technical and scientific backgrounds, which was very favorable. Some of our teammates were from more of an atmospheric background, which helped us understand the data that we were working with, and what features we were looking out for. Some of our members were also from a more engineering background, which allowed us to understand the technical aspect of the models and data handling that is often abstracted. We hope that more of our team members will join us in the later days, but it was a good start.
Great first post – you all are right on track!
I LOVE your discussion and really thoughtful analysis of what the end user needs and how you all can work to meet those needs. Ann Bostrom will also appreciate that you’re thinking about standards and how we can regulate AI – great policy thinking.
Your discussion about the strengths of working together as an interdisciplinary team also made me smile – this is such a great skill and perspective.
Keep up the great work!
1 Ans: We need our AI to be lawful, ethically adherent, and technically robust. The trust can be established in each stage from design to development, deployment and use. I believe AI and users need to work together to make sure AI is doing exactly what we want. This is where the end user has an important role to play. For different user the system will work differently as per their need. Without an end user we can not check how our AI is performing.
2 Ans: Due to the online platform and time issues it is very difficult to collaborate with team members. We plan to discuss more about this so we can work as a team and improve.
Sounds like you’ve learned a lot from day 1 and hopefully you can figure out a way to work together. Let us know in the help channel if there’s anything we can do to support that.
You’re doing an excellent job thinking about the use and taking a very user-centered approach to development! I love how collaborative your ideas are across developers and end users.
Looking forward to your next posts!