CS6475 Computational Photography Project 1: Object Removal
Project 1: Object Removal The goal of this project is to replicate the results of a published Computational Photography research paper by following the methods described in the paper. For this project, you will implement your own code to accomplish object removal. Object removal remains a growing area with many new methods and papers emerging every year. Google's Magic Eraser is a good example of an object removal tool found on mobile devices! The research paper covered in this project is much simpler, but at the time it was written, it helped to generate several improvements in the object removal process. |
Replication provides a deeper understanding of the research process and results. While previous results are helpful as a guide towards implementation, successful replication is challenging. The instructions from the published paper are not always clear and implementation details may be missing or ambiguous, which makes it difficult or impossible to achieve exactly the same results.
You will reach this goal by working through a research paper in the face of uncertainty. It is up to you to read and interpret the paper yourself (do not depend on others to do it for you). Where the description is imperfect, you will need to make assumptions and experiment to find an implementation that produces results that you find acceptable. We do not expect results to match exactly, but it is possible to come very close. You may have high level discussions with your classmates on Ed about the project, but do NOT share code.
All of the instructors for CS6475 want you to succeed in this course, which includes observing a level of honesty, integrity, and excellence befitting a graduate student of the Georgia Institute of Technology. This project and the challenges that come with it are representative of the kind of work done at the graduate level by researchers the world over. It is important to embrace the experience of making and clearly stating assumptions that allow you to move forward in the face of ambiguity, and documenting that process.
Make sure that you read the entire README, the research paper, and the LaTeX template before beginning to code in the provided python skeleton so you will not be surprised by requirements.
Caution
Please remember that you may not use any part of this algorithm or code from another source in print, online, or in person (not from anywhere!).
Every semester there are students who attempt to sneak plagiarised code past our TAs and automated systems. From the Course Policies, Section I: "Please note that Academic Integrity is taken very seriously here at Georgia Tech. Failure to uphold the honor code has severe consequences, including but not limited to receiving a “fail” grade on the assignment/project and severe disciplinary action from the Office of Student Integrity. We will continue to watch for instances unworthy of any student and hope that students follow both class and institute policy."
Instructions
You are required to write your own code that will produce results equivalent to those in the following paper:
Criminisi, A., Perez, P., & Toyama, K. (2004). Region Filling and Object Removal by Exemplar-Based Image Inpainting. IEEE Transactions on Image Processing, 13(9).
What will be evaluated? 1. Completion of the Project
Develop a working implementation of object removal using the exemplar-based inpainting algorithm from the paper. You will be required to:
Take your own pictures for use in this project
Create your own mask images for the target regions
Implement your own working code
Create a computational pipeline flowchart
Complete a final LaTex report
Scope Change - If you are unable to complete the original scope of the paper and need to decrease the scope in order to complete the project, then make sure that you discuss the scope change in your report. You will lose points for reduced scope, but it is much better to have a reduced project with an intermediate output rather than just a proposed pipeline.
2. Creating Your Own Image Sets
You are required to generate at least three sets of ORIGINAL input and ouput results. The quality of your final results are taken into consideration during grading, so make sure that your top three sets are your BEST top three. Each image set must include:
One original input image taken by you.
A mask you made for the object you are removing from your original input image.
A final result generated by your own working code.
Optional: If you have additional sets (whether good or bad) that you want to share and discuss that do not fit in other sections of your report, you may use the Appendix.
Image Basic Requirements:
Images must be 3-channel RGB color in .jpg or .png formats only. The autograder
will not accept any other formats.
Images must not exceed 1MB in pixel-size (height x width). NOTE: this is not the size that is listed under image properties; those are compressed sizes and may be a small fraction of the pixel-size. Images of 200KB to 500KB pixel-size will work well and will process even quicker. Images over 1MB will have a deduction. This requirement will help ensure reasonable runtimes for code, as all of your image sets will be run using your code for verification.
Each input image must have its basic EXIF metadata. Make sure that you keep the image's exposure time, aperture, and ISO. When including or sharing images, make sure there are no data contained in the EXIF metadata that you do not want shared (i.e. GPS). If there is, make sure you strip it out before submitting your work or sharing your photos with others. You may use the piexif library in a separate python file to work with exif data. Make sure that you do not use an app that strips all EXIF metadata from the image.
Image Content Requirements:
Each image must have a non-trivial object removed (e.g. people or other significant objects in the image). The object(s) you choose to remove cannot be simple textured areas. Large objects that approach 10% of the pixel-size may be difficult subjects and should be avoided.
Geometric images, such as Figure 7 and Figure 13 in the paper, are not allowed to be included in lieu of original images that you took with a camera. You may experiment with geometric images during your development and include your results and observations as extra material in your report, but again, it will not be considered as part of the required image sets.
Mask Requirements:
The mask is a single channel black & white image with a range of (0, 255). You may convert your mask to other range types in your code if you wish.
Mask requirements are similar to Assignment 2 - Pyramid Blending. For each set, you are required to create your own custom, non-trivial, black & white "mask" image that defines the regions you want to remove. Do NOT submit a mask that you found online and avoid creating simple/trivial masks. You may start with a simple set for your own testing purposes, but you should present at least three non-trivial sets in your final submission.
You may use photo editing software to design your mask. Example programs are Gimp (open source, free), Photoshop, or many others. In the report, you will be asked to discuss the steps that you took to create your mask, so if you used software, make note of the steps you took and take screenshots if necessary.
If you implement your own code to create your mask, you must discuss your implementation in your report.
You CANNOT use outside software for any other parts of this project.
3. Original Source Code
You are expected to write YOUR OWN code as part of your project, and explain it in the report. You MAY NOT use any existing implementations of any algorithms as a starting point. You must write your own implementation following the algorithm(s) presented in the paper. Including any part of the algorithm or code from any other source is forbidden and will be treated as plagiarism and an honor code violation. This includes porting an existing implementation between languages. Citing such a source does not remove the violation.
All of your code must be in object_removal.py . A skeleton is provided for you so that your function signatures and returns will work in the autograder. The TAs will run your code with your own image sets to verify your result images. Your submission must include all source code necessary to produce your results. We cannot stress this enough - make sure that the results that you present in your report are generated by running your submitted code with your submitted input images. If our results from running your submitted code/images do not match up with the results you present, we will investigate and there may be substantial deductions.
Your image sets are numbered 1, 2, and 3. Zero is saved for use by the autograder.
Watch for function variable setnum that you may use to cue variations in your code for your three images. Window size is a critical input that you may need to vary for best results in your your image sets ( setnum=1,2,3 ). We are reserving setnum=0 for use by the autograder. We will be using little images/masks with small window values in the autograder. Your code will take far less time to run. Hard-coding the inputs could cause your code to fail.
Class Environment & Libraries:
You must use Python, and begin with the class environment, CS6475. Make sure that you use the cs6475.yml that we provided in the course repository. Using different versions of python or different versions of any of the other specified libraries may result in code crashes when we verify your code and images. You may check your env installation by running pip freeze > requirements.txt in your code environment. If you have loaded other libraries, we cannot guarantee that your environment is OK. Dependencies are often updated when libraries are uploaded.
The basic set of permitted libraries are listed in the import section of object_removal.py . You may NOT import any additional libraries in your code; this
causes the autograder to stop with no score and an error report. It counts as a submission.
Forbidden library functions in OpenCV are those associated with inpainting. There is further guidance in object_removal.py . You may not use functions that do the majority of the work for you. The presence of these functions will result in significant deductions and will be reviewed for further action, up to a honor code violation for substantial usage.
Performance Tips
Your report requires presentation of interim results and failures. Keep some images
aside in a folder for this!
A main.py file is included that will generate all of the required images from your code in object_removal.py . You are not required to use it, but it follows the function call syntax that we will use in the autograder and in our automated full-code run with image production. You may revise it, but it would be a good idea to keep a clean version around to check your code before submissions.
It is critical to start with small toy images while you develop your methods. Location record-keeping for the object being removed is fundamentally important, it may be easier to develop this with pencil and paper.
You can iterate and test faster by making reduced size copies of your input images to test your code, but your final results must be generated using full size images.
You may use cv2 and numpy generic border/contour, masking, matching, and filtering methods, and many other single functions. If there is any doubt about using a function, please use a public post on Ed to ask about it so that everyone in the class is aware of yes/no verdicts for code.
Python with good profiling/optimization will be reasonably fast on the given images. Vectorize your Python code wherever possible for efficient array operations as long as they do not directly implement the algorithm(s).
Consider writing intermediate and final result matrices to disk for manual inspection or iterative development to save time. Consider adding pre-allocated buffers and assigning to them with np.copy when possible. This avoids unnecessary heap allocation/deallocations that could significantly slow down processing, depending on how badly the heap is fragmented.
You may use code and image handling methods from the previous assignments.
There are time limits - see Autograder Information and Code Runtime Limits. If your laptop/ordinary desktop can produce the result images within the 2 hour time limit, then your code should be fast enough. Last term, the mean run time for all 3 images was 20 min, median was 6 minutes.
4. Computational Pipeline & Code Explanation
Provide a complete computational pipeline of your project:
For each step, name the function(s) or algorithm(s) used.
Include any steps completed with outside packages/code with a note.
Indicate which steps are manual (i.e. not a part of your automatic pipeline).
Include any incomplete step(s) with a note that says 'not completed'.
The flowchart is shown here and in the report template as an example. This section holds the single flowchart image along with text explaining your process. Draw.io is one free software tool to generate flowcharts, or you may use others you are familiar with.
Clearly explain the purpose of each flowchart step, and explain how your code accomplishes it. Do NOT re-explain the algorithm here! In the text explanation, show pseudocode or a SHORT (2-5 lines) snippet of critical code that accomplishes that pipeline step and EXPLAIN the code.
5. Required Report Sections
You must use the provided LaTeX template Project1_Object_Removal.zip . There is a page limit for each section, so make sure to read the LaTeX template. Your report must be 10 pages or less. Make sure to provide a detailed discussion in each of the required sections.
Here is a summary of what each section will cover. There are more detailed directions for each section in the LaTeX template, so again, make sure to carefully read the template.
SECTION 1:
Project Goals: Discussion of the original scope and any scope changes.
SECTION 2:
Algorithm Discussion: Explain the major algorithm from the paper. Show that you
understand this in your OWN words. Do NOT copy from the technical paper.
SECTION 3:
Computational Pipeline & Code: Present your computational pipeline flowchart and
explanation of each step.
SECTION 4:
Project Development: A detailed narrative of your development process that includes: Problems & Issues you faced along the way, and how you resolved them. Includes the presentation of good and failed interim results (from various parts of your work and pipeline). Keep some of your early work images in a folder for this purpose!
Retrospective discussion of what you would do differently. This can cover any part of your project, but make sure to explain the issue, and specifics on what you would investigate or do differently.
