COMP0173 AI for Sustainable Development - Coursework #2 Machine Learning based Data Analysis
Coursework overview
Coursework #2 is a data analysis coding-based coursework. You are asked to choose a dataset related to sustainable development and do some machine learning based analysis. We will detail this more in the next sections.
You can choose an application/dataset from the ones we analysed in the first coursework or choose a completely new dataset:
- Option 1: Choose a dataset from the SustainBench paper (which includes 15 datasets related to sustainable development), please do not choose the climate action one related to brick kilns, as we will use it in class. You can choose the same dataset that you use in coursework #1. Please note that the authors from SustainBench have a GitHub project with data loaders, preprocessing code and baselines for these datasets here: https://github.com/sustainlab-group/sustainbench
- Option 2: Choose an alternative dataset of your choice (e.g. that you are interested in). Please if you choose this option motivate the dataset's relationship to sustainable development. If the dataset is part of an online competition (e.g. in Kaggle, Zindi, etc), please make sure to include information that makes us discern what you have built, vs what was built from someone else (since in some cases participants of the competition submit tutorials, etc).
Please do NOT choose the same dataset that you used as part of a different coursework for a different module (which may be possible as the coursework is up to some extent open ended), as this would be considered self-plagiarism.
Once you have chosen your dataset we will ask you to do the following:
1. Do some exploratory data analysis to get some insights about the dataset. We do not give specific guidelines here as this may depend on the dataset itself, but we ask you to justify your choices and the results of your analysis.
2. Think about the task that the dataset is most suitable for and discuss what would be the most appropriate evaluation criteria for such a machine learning task. For example, you may have a dataset where songs are labelled with different music genres. Perhaps the task for which the dataset is most suitable for is to build a multi label prediction model that can classify new songs, and the most appropriate evaluation metric could be the hamming loss (commonly used in multi label prediction, which could be used both as inspiration for a training objective for neural networks, or simply as the evaluation criteria for model selection and hyperparameter tuning).
3. Design and implement a machine learning system/pipeline. This could be either supervised or unsupervised. In either case, the pipeline you choose needs to be aligned with the task you defined before.
4. Design a set of experiments and discuss how these fit with the questions that the dataset aims to address. Do a performance & scalability analysis of your model.
5. Analyse and discuss some of the ethical implications, in connection with your exploratory data analysis and model performance (e.g. do you appreciate any imbalances in the data or initial biases that can be problematic?).
6. Discuss sustainable development relevance & impact.
We ask your code to be in python, but beyond that we do not mind what packages you use.
Submission: You do not have to submit your dataset and notebook to the coursework submission. Simply run the analysis in jupyter notebooks and then export the notebook as html or pdf. You can then submit this html/pdf file in moodle.
**[To be completed by you]**
Exploratory data analysis (15%)
Guidelines: Load the dataset and answer the following questions:
- What are the characteristics of your dataset? Do you see any trends in the data? (5%)
- Are there any challenges with the data? (missing values, outliers, imbalanced classes, biases, etc...) (5%)
- Does the data need any pre-processing to successfully apply standard machine learning models? If so, what kind? Please do the pre-processing that you consider necessary. (5%)
You can use descriptive statistics here, figures/plots, etc.
In [10]:
# your code here!
# load relevant packages, load your dataset, start your analysis
# you can have as many code cells as needed
**[Your insights go here. To be completed by you]**
Task and evaluation (10%)
Guidelines: Tell us a bit about the main task that in your view this dataset could help solve and propose and justify the evaluation criteria that would be important here, not only considering the task but your exploratory data analysis. For example, if you found that there is class imbalance in the dataset and that the most important class is the minority one, you may want to add costs in your evaluation metric to account for it, or maybe you want to use a metric specifically suited for imbalanced classification.
**[To be completed by you]**
Design and build an ML system (20%)
Guidelines: Taking into account the task and evaluation criteria set in the previous section design and build now an ML system. If you go for supervised learning, you could start by partiting your dataset in train and test here and try a model on the dataset. But please justify your choices in each case. Why did you choose a specific ML model? Why is it relevant for the task and dataset at hand? Do you envision any challenges with the use of such model?
Again, we do not ask for a specific model to be used. Instead, we will evaluate the depth and appropriatedness of your analysis. As this section (and some of the following ones) are more open ended we do not specify marks for each question.
Please note that if the dataset of your choice is too large to work with, you can simply choose a subset of it. We will not substract any marks for doing so.
**[Your insights go here. To be completed by you]**
Experimental analysis (performance & scalability) (20%)
Guidelines: Test your model here. You can do hyper-parameter tuning and any ablation studies you consider important. How does your model perform? Is there any room for improvement? If so, what do you think it's needed? Comment as well on how does the model compare to previous baselines (yours or from the literature).
# your experimental study here
**[Your insights go here. To be completed by you]**
Ethical considerations (15%)
Guidelines: Analyse and discuss the ethical dimensions of the application: bias, fairness, interpretability, etc. Some of these may not be relevant, but we leave this for you to decide which ones would be the relevant ones to consider for the problem at hand. Here you could do a sensitivity/interpretability analysis of the model, to study the effect of different variables or examine whether there are any biases (e.g. the model performs best for certain group of examples in your dataset).
**[Your insights go here. To be completed by you]**
Sustainable development relevance & impact (15%)
Guidelines: Discuss what are the current challenges surrounding this dataset in terms of sustainable development and how can ML help overcome them? Additionally, bring out all of the things you have managed to do and understand from the experiments and connect them to sustainable development. Discuss what are the challenges of deploying such a model and how it could impact our progress towards achieving the sustainable development goals.
**[To be completed by you]**
