G6061 Fundamentals of Machine Learning Assignment: Photo Classification
Submission
You should submit precisely 3 files individually (do NOT zip them):
1. Two page report (only references may appear on a third page) in format using either thepdf latex Download latex or word Download word template provided.
2. Code (a version that will run - either an notebook (preferred) or a script; do NOT submit any data)..ipynb.py
3. Class predictions for the test data. This must be in exactly the same format as the provided example file and must be named .sample_valid_predictions.csvpredictions.csv
This piece of coursework is worth 80% of your mark for this module.
The Brief
For this assignment, you will carry out a binary classification task, and write a report on this. Please read this brief carefully and also the Marking Criteria and Requirements below.
The data come from photos, and your task is to come up with a machine learning method for classifying the photos according to whether their content is happy or sad. The data you are given for each photo consists of 3456 features. 3072 of these were extracted from a deep Convolutional Neural Network (CNN) [1], and the remaining 384 are gist features [2]. (You are given all these features as a 1-dimensional array, so you will not be performing any feature extraction on raw images.)
There are two files of training data. The first contains 400 samples with all the data present (no missing or null values). The second contains 2750 samples, which have some missing data, as indicated by a (not a number). The training data have class labels, 1 for happy, and 0 for sad. In addition, there is also a confidence label for each sample. The class labels were assigned based on decisions from 3 people viewing the photos. When they all agreed, the class label could be considered certain, and a confidence of 1 was written down. If they didn't all agree, then the classification decided on by the majority was assigned, but with a confidence of only 0.66.NaN
There is one file of test data, containing 1000 samples. You must generate predictions for the class labels of these data. (Note that, as with the second training set, the samples in the test data set contain some missing features.)
Your job is to obtain the best predictions you can, and to justify your methods. You should provide reasons for which classifier or combination of classifiers you use, how you do model selection (training-validation split or cross validation), and how you handle the specific issues with these data (large number of features, missing data, the presence of confidence labels for the classes of the training data). We value creative approaches!
You may make use of any classifier, such as: single-layer perceptron, multi-layer perceptron, SVM, random forest, logistic regression, etc. You are not required to code classifiers from scratch, and you can use any machine learning toolbox you like, such as scikit-learn.
The data can be obtained here . (For those of you using Colab, we recommend uploading the data to your Google Drive for quick and easy access.)
And here is a link to some tips for getting started
Good luck!
Marking criteria and requirements
Your report must contain the following sections.
1. Approach (10 marks)
Present a high-level description and explanation of the the machine learning approach you have used (e.g. multi-layer perceptron, logistic regression, etc. or a combination thereof). You should cover how the method(s) work(s) and the key assumptions on which the approach depends, justifying your choices. Pay close attention to characteristics of the data set, for example: high dimensionality.
2. Methods (30 marks)
Describe in detail what you did, and include references to appropriate literature.
- How did you train and test your classifier(s)?
- How did you do model selection?
- Did you rescale the data? (See preprocessing lecture, and https://scikit-learn.org/stable/modules/preprocessing.html)
- Did you do feature selection? You are provided with two types of features: CNN features and gist features. Are they equally important? (See the lectures and e.g. https://en.wikipedia.org/wiki/Feature_selection)
- How did you deal with missing data? Did you do something with the confidence labels?
3. Results and Discussion (30 marks)
Use graphs and/or tables to illustrate the results of your model selection. For example,
- Show how the choice of classifier hyper-parameters affect the performance of the classifier, using a validation set.
- Show changing performance for different training sets. How useful, relatively, are the incomplete training data? How useful is it to take account of the training label confidence?
In the Discussion:
- If you think that there might be ways of getting better performance, then explain how.
- If you feel that you could have done a better job of evaluation, then explain how.
- What lessons have been learned?
4. Coding (20 marks) and Accuracy of your predictions (10 marks)
Please make sure we will be able to run your code as is. High quality code with good structure and comments will be marked favourably.
We will compute the accuracy of your class predictions for the test data (as percentage correct), and give you a mark out of 10 for this. Those of you with the most accurate predictions will score 10/10. Those of you with the least accurate predictions will score 5/10. If you do not submit a class predictions file in the correct format, then you score 0/10.
Footnotes:
[1] Extracted from the activation layer of fc7CaffeNet http://caffe.berkeleyvision.org (Links to an external site.)
[2] http://people.csail.mit.edu/torralba/courses/6.870/papers/IJCV01-Oliva-Torralba.pdf
