CS 6475: Computational Photography - A4: High Dynamic Range Imaging
Synopsis
This assignment focuses on the core algorithms behind computing HDR images based on the paper . Debevek & Malik's main concern was generating radiance maps that must be produced before images are displayed. Mapping the wide range of the HDR radiance map to fit the limited visible range of a display or print medium is the final step in computing the basicHDR image. Our basicHDR image made from the radiance map demonstrates a nice, but muted result that needs further processing to generate a pleasing image. The few output images in the Debevek & Malik paper (Figure 8e and 8f, with adaptive histogram compression) were generated using Greg Ward Larson's 1997 methodology which used histogram compression to generate the outputs. Ward's method is complex, and the tech paper is long. Instead, we are taking a simpler path to tone-mapping with histogram manipulation.
We start with a basic histogram equalization technique to illustrate the methodology in Cromwell's paper, "Contrast Enhancement Through Localized Histogram Equalization". The resulting image, with the histogram and cumulative density plots will make it very clear what the method does.
To aid with this assignment review these resources:
Lesson 05-04 HDR
Chapter 10.2: Szeliski, R. (2010). Computer Vision: Algorithms and Applications
You MAY take pictures for your own image sets for this assignment. If you are interested in taking your own image set, see section 2. Alternatively, you have a choice of using an image set from the web.
General Instructions and Tips
Significant usage of linear algebra is required. The notational conventions & overall structure are explained in the paper and we provide significant extra resources in the github download.
Images in the
images/source/sampledirectory are provided for testing. A sample output is also provided for comparison use. Due to randomness in the process, your image may be very slightly darker or lighter, but the coloring should be the same. Only your basic HDR image from the sample set is included in the report. The rest of the report images with be from your chosen image set.It is essential to put the images in exposure order and name them in this order.
main.pyexpects it. This is done for you in the input/sample/ folder images of the home scene. Your own images and exposure times must be setup in the same way. For these sample images, the exposure info is given in main.py and repeated here (darkest to lightest):EXPOSURE TIMES = np.float64([1/160.0, 1/125.0, 1/80.0, 1/60.0, 1/40.0, 1/15.0])Read the
main.pyfile before you start, there are tools and plot resources included for you. Main.py is submitted, and you may make changes to the file. Also review the LaTeX report file to get an understanding of what images you will need, and the questions.Running
main.pywill execute your HDR pipeline. The script will look inside each subfolder underimages/source/and attempt to apply the HDR procedure to the contents, and save the outputs toimages/output/. (For example,images/source/samplewill produce output inimages/output/sample.)Downsample your input images to 500kB or less for testing. This assignment will be assembling all of your input images into one image, and it is memory intensive. Some image sets contain a dozen or more images, and may cause problems with your own computer or the resource-limited autograder.
Be careful regarding arithmetic overflow on function outputs. You will be working in wildly different ranges than 0-255 as we follow the work of Devebec & Malik. The radiance maps will have ranges of several magnitudes. Do not blindly threshold, clip or convert to uint8 unless it is called for, or you may drastically reduce your radiance map and ruin your output results.
1. Code Guidance
1a. Implement Basic HDR image functions
The following functions are require to generate the basic HDR image from an input image set. The Debevek & Malik paper has specific information that will help you in these functions.
linearWeight: Determine the weight of a pixel based on its intensity.sampleIntensities: Randomly sample pixel intensity exposure populations for each possible pixel intensity value from the exposure stack.computeResponseCurve: Find the camera response curve for a single color channel by finding the least-squares solution to an overdetermined system of equations. This function is difficult. There are additional resources provided in the repo: constraintMatrixEqns.pdf and mat_A_example.png. Passing the AG does NOT guarantee that your mat_A is correct, as randomness in sampleIntensities affects your mat_A and subsequent results. Print or view your mat_A and ensure that its general pattern is similar to these references. Poor results in this assignment are often related to mat_A coding errors.computeRadianceMap: Use the response curve to calculate the radiance map for each pixel in the current color channel.
At this point, you can use the main.py function computeHDR to generate your basicHDR result.
basic HDR images are required for both the sample images and your own image set.
Basic Histogram & Cumulative Density plots or both the sample images and your own set are required in the Report. They can be generated in main.py.
Radiance map plots can be generated as a part of the production of the HDR by main.py. This image plot is not
required in the report, but you may compare its output to Figures 5 and 8d in Debevek & Malik to understand the logarithmic
magnitudes of the radiance values in your image.
1b. Implement Histogram Equalization (HE)
After you complete the (1a) functions and generate the basicHDR image from your input set, complete the
following functions. These functions take the basic HDR image and apply histogram equalization to it. Review for background (first section, through the procedure steps).
computeHistogram: Convert a BGR colorspace image to Hue-Saturation_Value (HSV) colorspace. Generate a histogram of the V-channel of the image.computeCumulativeDensity: Calculate a cumulative density function (CDF) of single channel histogram.applyHistogramEquilization: Using the V-channel CDF, generate the histogram-equalized HDR (heHDR)image.
heHDR Histogram & Cumulative Density plots of the heHDR image are required in the Report.
1c. Implement Weighted Threshold Histogram Equalization (WTHE)
Use method for this HDR image. To improve on the basic HE image, you will implement WTHE. This method begins with the same histogram, but applies an enhanced power curve which can be tuned by two added variables. Effectiveness will be verified when the wtheHDR image is evaluated.
Because there are choices in setting up this function, this function will only have its return image tested for shape and data type, and that all values are within the range of 0-255 tested by the autograder.
computeWTHE: Calculate the new weighted thresholded PDF, the normalized CDF, and generate the wtheHDR image.You should experiment with the root and value variables via main.py.
wtheHDR Histogram & Cumulative Density plots of wtheHDR are required in the Report.
1d. Implement Color Saturation Enhancement for the bestHDR
-'applyColorEnhancement` Using your wtheHDR image, experiment with enhancing the colors in your image. This will done by applying a curve to the Saturation channel (S) of an HSV image. You are working to produce more life-like colors, similar to those found in individual input images. The end result is your bestHDR. Effectiveness will be verified when the bestHDR image is evaluated.
Because there are choices in setting up this function, this function will only have its return image tested for shape and data type, and that all values are within the range of 0-255 tested by the autograder.
Your bestHDR image should show clear improvement over your basicHDR and histEQ images. Clear improvement means the colors are enhanced realistically. Brighter colors should be apparent in one or more of the input images. Clarity is improved, and fine detail areas enhanced. Halos and other artifacts should not be introduced.
If your image takes a pastel church image and turns it into a Gothic Halloween image it may be very cool, but you have missed the mark.
bestHDR Histogram & Cumulative Density plots are required for the bestHDR in the Report.
1d. Submitting Code to the Autograder
The docstrings of each function contain detailed instructions. You are encouraged to write your own unit tests based on the requirements. The hdr_tests.py file is provided to get you started. Your code will be evaluated on input and output type (e.g. float64, uint8, etc.), array shapes, and values.
The autograder timeout has been extended to 20 minutes to allow for extra run time for your HDR functions.
When you are ready, submit your code to the Gradescope autograder for scoring. We will enforce the following penalties for excessive submissions:
from 1 to 30 submissions -> No penalty
over 30 but <= 40 submissions -> -5 penalty
over 40 but <= 50 submissions -> -10 penalty
more than 50 submissions -> -20 penalty
2. Find or take an HDR Image Set
2a. Making the required HDR Images
You will produce four different HDRs with your set: basicHDR, heHDR, wtheHDR, and bestHDR
Once you have passed the AG with your code using the sample images, it's time to find/take a good set of HDR images. Only one HDR is produced with the sample images. All the rest are from your image set. Find an image set on the web (2b), or take your own series of images (2c). Obtain an image set with a minimum of 4 images. Your set must have exposure times varying while other settings are held constant.
Notes: (1) Exposure Values (EV) are NOT the same as exposure times, and they will not work. (2) The sample images are quite small, but big enough for good results. Keep your input images under 1MB.
2b. Image Sources and Settings
A good source of HDR image input sets:
In the past, we used Herrmann, K. (Oct 15 2020), Farbspiel Photography, "HDR pics to play with", (https://farbspiel-photo.com/learn/hdr-pics-to-play-with), however he has gotten out of the business. We will provide some of these images on github, with the same license requirements that he previously used: cite his name and URL in your code and report. There seem to be few other options right now.
Search the web for other good sets, sources change over time. If you find other good sets, please post the URL on Ed.
Several students may use the same image set. Some images are more difficult; large sets can be slow.
You must cite the source of your images in References.
2c. Taking your own images
If you would like to take your own set, images with a very large range of brightness are the best. Examine some websites so that you understand what type of scene works well. You must have at least 4 input images, more is better. At a minimum, the camera must have manual exposure time control, where other settings (particularly aperture and ISO) are held constant, and a tripod. A remote control would help in maintaining camera position between shots. Exposure times are required for your code, and aperture and ISO must be reported. Dark indoor scenes with bright outdoors generally work well for this, or outdoor scenes with overly-bright and dark areas. HDR image sets must be well aligned and meet critical settings requirements. A simple openCV alignment tool is provided in main.py. Other image_stacking programs in python are available, and you may share URLs for these on ED. You may use outside code sources (Gimp, Photoshop, etc) for image alignment, also called image_stacking for this purpose only. There is a report question on alignment, keep track of what you try.
There is no extra credit for this, but you will earn our respect!
2d. Requirements for your image set
Caution: Some past students have tried to use image sets that changed the ISO values instead of exposure. That is a different method, it will not work here, and you will fail the assignment.
You MUST use at least 4 input images to create your HDR images. In most cases, using more input images is better. The entire set of input images along with your resulting HDR images must be submitted in Resources.
Every input image must have EXIF: exposure time, aperture, and ISO setting. There is a deduction if your images do not have all three items. Because you may be using web images, it will be sufficient to record web image values in the report table.
Enter the EXPOSURE_TIMES (NOT exposure values - EV, they are NOT times) into the main.py code. Images must be numbered in order from darkest (input-01) to lightest (input-xx). Times must be in the same order as the image files to get a correct HDR result. Order from darkest to lightest. Your results can be amazingly bad if you don't follow this rule.
There are large sets with big images available. We recommend 500kB or less per image for testing. Resize is your friend. When you are happy, you may go back to a larger images size for your official results.
Image alignment is critical for HDR. Whether you take your own images or use a set from the web, ensure that your images are aligned and cropped to the same dimensions. We have provided an alignment tool in main.py in the main function that you can enable in the calling line to perform simple alignments. You may use a commercial program such as Gimp (open source) or Photoshop ($$) to help with this step, or write your own alignment code if our simple tool does not work for you. This is the only permitted usage of outside software packages in this assignment. Note what you do in your report.
3. Download & Complete the Report
Use the A4_HDR_report_template_term.zip report template provided in the class repository. The LaTeX template specifies all of the images, data, and questions you must answer. Follow the instructions in the to upload it in Overleaf. If you have a different favorite LaTeX program, you may use it; the template should be generally compatible.
Save your report as report.pdf
The total size of your report+resources must be less than 30MB for this project. If your submission is too large, you can reduce the scale of your images. You may resize your images using the resize function in main.py.
The total length of your report is 6 pages maximum. This report is expected to be five pages for most students, unless you decide to submit extra observations and images in the appendix (no extra credit, but the TA staff enjoys interesting work and unusual fails!)
You are required to use the LaTex format for your report. Different report organization or changing section titles and questions will receive a deduction. You may add additional relevant information or images within the structure of the report, as long as all requirements are met.
You are required to provide References. Everyone is expected to have at least four references; the two required papers and two additional references; these should include StackOverflow answers, valuable tutorials, blogs, Ed posts, and other paper or textbook references. Provide URLs and site or page names, typical book information and page numbers, etc. We should be able to easily view your references if needed.
4. Submit the Report
Submit the same report.pdf to both:
Gradescope as A4-HDR Report
Canvas > Assignments to A4: HDR Report
After you upload your PDF to Gradescope, you will be taken to the "Assign Questions and Pages" section that has a Question Outline on the left hand side. These outline items are determined by the Instructors. For each question - select the question, and then select ALL pages that go with that question. This is important to do correctly because any pages that are not selected for the corresponding question might get missed during grading.
5. Submit the Resources
Your resources.zip must include your code, the input images you chose, and your HDR result images
for the input set of your choice. Only one sample set image is used, your basic sampleHDR.
Follow the filename protocol below, the autograder will be expecting these exact filenames.
Gather the following files, and zip the files together. DO NOT place the files inside a folder before you zip them. Submit your zip file to Gradescope A4-HDR Resources.
Filenames for Resources (15 required files): *Your images must be one of the following types: png, jpg, or jpeg. PNG images generally give better results, as they have little compression.
hdr.pysubmit only one code file that produces all of your imagesmain.pysubmit the version of main.py that generates your imagesinput_00all input images must have basic EXIF (exposure time, aperture, ISO)input_01input_02input_03include all remaining images in your set, numbered in ordersampleHDR: the basic HDR image from the provided sample home image setbasicHDR: basic HDR imageheHDR: histogram-equilized HDR imagewtheHDR: weighted and thresholded HDRimagebestHDR: bestHDR image (after color enhancement)histCDF_basicHDR: histogram-CDF plot for basicHDR imagehistCDF_heHDR: histogram-CDF plot for heHDR imagehistCDF_wtheHDR: histigram-CDF plot for wtheHDR imagehistCDF_bestHDR: histogram-CDF plot for bestHDR image
Notes:
Total Submissions Size: 30MB The total size of your project (report.pdf + resources.zip) MUST be less than this limit. If your submission is too large, you can reduce the scale of your images.
When sharing images, make sure there is no data contained in the EXIF data that you do not want shared (i.e. GPS). If there is, make sure you strip it out before submitting your work or sharing your photos with others. We only require that your submitted images include aperture, exposure time, and ISO.
9. Criteria for Evaluation
Your submission will be graded based on:
Correctness of required code
Creativity & overall quality of results
