Project: Improving Classification through Self-supervised Learning
Project: Improving Classification through Self-supervised Learning
Overview
This project is one of the options which you can optin for undertaking summative assessment, carrying 25%
of the total module marks. In this project, you will applyself-supervised learning to improve the classification perf
ormance. In this project, you will: 1) build a classificationmodel; 2) improve performance through pretraining; 3) w
rite a report.
Submission
TBC
You should submit a zip file containing the following:
-
Your code;
-
A 5-pages report (IEEE double column format) explaining your code, visualising your results you obtained and discussing your observations. If you want you can include additional pages only for visualisations and references (this is optional and you won’t lose any marks if you do not include additional pages). However, the main text of your report should fit in the first 5 pages and the additional pages (if any) should only include visualisations with short captions and references.
Please note that for all the visualisations and tables that you include in your report, it is important to include a reference in the main text (typically using a Figure or Table number).
Background
Melanoma Classification
While melanoma is the least common type of skin cancer, it tragically accounts for 75% of deaths from the disease, making it the deadliest form. Despite this, over 100,000 new cases of melanoma are expected in 2020, with nearly 7,000 people losing their lives to it. Just like with other cancers, early and accurate detection is crucial for effective treatment. This is where data science could play a vital role, potentially aiding in identifying melanoma early and improving patient outcomes. You will participate in the Kaggle’s competition “SIIM-ISIC Melanoma Classification”. In this dataset, there are 33126 records for training, each has a lesion’s image and metadata. The images and metadata are located in ‘jpeg‘ and ‘train.csv‘ respectively. The metadata includes the diagnosis information, including patients id, sex, age, the anatomic site, the lesion diagnosis, a target (1 for melanoma and 0 for others), and so on.
Figure 1: Submit your results.
To submit the results, you must train a model based on the training set and predict the probability for each sample in the test set. The final result will be evaluated on area under the ROC curve through kaggle where you can upload a ‘submission.csv’ or select a notebook (recommended), as shown in Figure 1.
There are many excellent works available in kaggle to help you understand the melanoma classification, such as, 1st-Place-Solution and SIIM-Transformer. Note that, your goal is to understand how self-supervised learning facilitates classification, thus, you can use one suitable project as your baseline. To speed up the training process, you may use a lower resolution or fewer training epochs.
1
Self-supervised Learning
Self-supervised learning (SSL) is a technique in machine learning where a model trains itself on tasks using the data itself to generate supervisory signals, instead of relying on external labels provided by humans. Essentially, the model creates its own learning objectives from the data it’s given. You are going to use the images in the training set without labels to get a better starting point. One advantage of SSL is to improve data-efficiency [3], which is a major hinder in medical problems due to the privacy concern.
Successful self-supervised learning methods include MoCo [6], SiT [1], GMML [2], MAE [5], and so on. Theses methods can be roughly categorized into two branches: contrastive learning and masked image modeling. You may use packages to build models quickly through pytoch image models - timm and MAE. You may use the Efficient Self- supervised Learning Library to train your model from scratch. There are more repos you can refer: SiT and TinyViT. You may choose one of above methods to pretrain your model on the training set (feel free to use any SSL methods).
Once you pretrained your model to obtain the weights by ‘torch.save’ and ‘model.state dict()’, you will finetune the model with the classification script at the last section. The only difference is that your model starts from the pretrained weights instead of random initialization.
Guideline
-
Load and preprocess the dataset. Optional: accelerating your data loading through FFCV.
-
Determine your backbone network. Popular architectures include ResNet [7] and ViT [4]. Choose proper model
size according to your computational resources.
-
Train your model with random initialization to classify melanoma to obtain a baseline.
-
Pretrain your model to obtain pretrained weights.
-
Finetune your model with the pretrained weights to obtain an improve model.
-
Evaluate the baseline model and the improved model on test set to generate ‘submission.csv’ files and submit the files through Kaggle to get the final scores.
-
Tips: Use fewer epochs or lower resolution to accelerate training if you are in short of GPUs. Discuss your Observations
It is important to describe or explain which project you refer to, what kind of modification you made and why. It is common that your reproduce is lower than the original one, especially when you have insufficient GPUs. Performance is not the only criterion, instead, the efforts you present in the report masters more. You need to explain the mechanism of the SSL method you apply and study the key factor to improve pretraining.
Extra credit
Extra credit will be awarded if one could potentially perform additional tasks related to the main recognition task. These additional tasks might include but not limited to visualisation, interface design etc.
MARKING CRITERIA
The project will be assessed giving the 50% weight (12.5 marks) to technical report and 30% weight (7.5 marks) to functionality and 20% weight (5 marks ) to code quality according to the following criteria:
REPORT QUALITY [50 marks]
Whether the results are well presented and discussed.
In particular:
• Is the report well written and clear?
• Is the report well structured?
• Are the figures clear and well explained?
• Does the report provide a clear explanation of what has been done to solve the problem? • Is there a sufficient discussion regarding observations on the produced results?
The distribution of the marks within the report are as follows:
2
• Abstract: 10 marks,
• Introduction: 10 marks,
• Literature (minimum 5 papers): 15 marks,
• Methodology: 25 marks,
• Experiments: 30 marks,
• Conclusion and future work: 10 marks
FUNCTIONALITY [30 marks]
Whether the submitted program performs as specified.
In particular did the code implement all the steps specified in the previous sections?
CODE QUALITY [20 marks]
Quality and efficiency of the coding, including appropriate use of documentation.
In particular:
• Is the code efficient?
• Is the code extensible and maintainable?
• Is the code clear and well documented?
