Low Light Image Combining Using Python (1)

Low light images are photos taken while it is dark. Low light images are usually noisy since there is not enough light coming in. To reduce the noise, a commonly used way is to combine multiple images. For example, such a method has been used in Google pixel camera night mode. Here we will show a Python script which does simple low light image combining.

The pipeline includes three steps: raw image alignment, raw image combining, post processing.

Raw image is the direct output of the image sensor. In normal photo, each pixel has three color channels as R/G/B. But in raw image, each pixel only has one color channel. Bayer pattern is widely used in raw image, and it is also used in this experiment. The color pattern used here is shown below which is one kind of Bayer patterns. As one can see, out of 24 pixels, half of them are green, 25% is red and 25% is blue. The reason that green channel occupies the most pixels is because human eye is the most sensitive to green color.

   0 1 2 3 4 5
0 G R G R G R
1 B G B G B G
2 G R G R G R
3 B G B G B G

What raw image alignment does is to align two raw images. Since multiple images are taken at different times, due to hand movement, there will be small shift of the image. Without alignment, adding images together will create blurry photo. Our method for image alignment is straightforward. To align two images, we will divide both images to blocks with equal dimension. By calling one image "target" and the other image "candidate", we will swipe the candidate block through the target block in both x and y axis search directions. The scope of the swipe is called search range. For each swipe position, the L1 distance will be calculated which is the delta of the pixel values of the candidate and target blocks. Out of all swipe positions, the one with smallest L1 distance will be selected. For more details of this operation, you can refer to the Python source code here.

The reason that image alignment should be done at the very beginning of the image pipeline on raw image is for both performance and computation time. In term of performance, additional processing of the signal at the later stages such as interpolation/correlation may degrade the performance of alignment. In term of computation time, raw image has one color in a pixel but at later stages after interpolation, there will be three colors in each pixel. Calculating alignment on one color channel is faster than calculating that on three channels. Due to the structure of Bayer pattern, we only calculate candidate of even number of pixel shift such as 0/2/4 etc.

The step of raw image combining is to add candidate and target blocks together. Since they are already aligned, adding them together will enhance the signal while reducing the noise. Here we apply equal weight to each image. However, performance could be improved by applying different weights.

Between raw image and the photo we usually see, post processing needs to be done. Libraw library is used for post processing. The major steps of post processing is shown below:

Next we will show the outcome of combining. Raw images are obtained using Pro camera mode of a Samsung Galaxy S9 device. The camera settings are ISO = 800, 1/4 shutter speed, and F1.5. S9 has two aperture modes: F2.4 and F.15. F1.5 is a larger aperture for low light photo. Figure below shows the difference before (left) and after (right) the combining. The image quality is clearly improved. To get the right figure, we combine eight low light images. Bilateral filtering can be used to further enhance the image quality.

Source code of the Python script and raw image files can be found here: https://github.com/legendzhangn/blog/tree/master/lowlight_image_combine

LibRaw Post Processing Pipeline Translated Into Python (2) -- Low Light Photo

In a previous post, we introduced a Python script which emulates the pipeline of LibRaw. LibRaw is a commonly used library for raw image conversion.

Then we found that compared with LibRaw, our Python script does not work well for raw image taken under low light. While LibRaw still shows the objects albeit quite noisy, our original Python script does not show anything. After some digging, it is found that LibRaw has an auto bright function which is not available in the previous Python script. As shown below, for low light image, the difference made by using auto bright function is quite stark.




Thereafter, our Python script gets updated and now it supports auto bright as well. The main difference between low light image and normal light image is how Gamma mapping is done. For low light image, Gamma curve peaks much faster than for normal light image. The exact Gamma curve of low light image depends on its histogram. The updated script can be found here, and it shows how this new Gamma curve is generated.

An Android App for Photo Brightening

I published an Android app used for photo brightening named Brighter Photo. Using a low light photo as the input, the app can post process it and make it look brighter. It is used only for post processing but not as a camera. The app is available at Google Play store and the link is here

Figure below shows what the app can do. The left half is a photo taken at night and the right half is the same photo after processed by Brighter Photo app.

To use the app, the GUI has three buttons: Load, Save and Setting. "Load" button can load photo from Android gallery. After loading, post processing immediately starts. Post processing takes 10-30 seconds depending on the image size. It runs slower than some of the similar apps but produces better results. "Save" button allows to save the results. Note that to enable saving, the storage permission needs to be enabled in Android setting for this app.

This Youtube video shares how to use the app.

Hope that this introduction is useful and you can enjoy using this app. Any feedback is welcomed.

A Simple Example of Image Processing Using Java

In this post, we share a simple example of image processing using Java. This example benefits from Java tutorials available from Internet.

This example has three steps: 1) read the input image; 2) image processing; 3) display the processed image.

For step 1, an image is read into image buffer. 

    File file = new File("Lenna.png");
    BufferedImage image = null;

  // Read image
    try
    {
        image = ImageIO.read(file);
    }
    catch (IOException e)
    {
        e.printStackTrace();
    }
    System.out.println("done");

Step 2 has a simple operation of image processing. It does 2D convolution filtering for the image using filt3x3 function in the ConvolutionMatrix class.

    // 2D convolution for the image
    double[][] config = {{1,2,1}, {0,0,0}, {-1,-2,-1}};
    ConvolutionMatrix imageConv = new ConvolutionMatrix(3);
    imageConv.applyConfig(config);
    image2 = imageConv.filt3x3(image, imageConv);

Step 3 displays the modified image

    // Display the modified image
    ImageIcon icon=new ImageIcon(image);
    JFrame frame=new JFrame();
    frame.setLayout(new FlowLayout());
    frame.setSize(image.getWidth(),image.getHeight()); //Window.setSize(int width, int height)
    JLabel lbl=new JLabel();
    lbl.setIcon(icon);
    frame.add(lbl);
    frame.setVisible(true);
    frame.setDefaultCloseOperation(JFrame.EXIT_ON_CLOSE);

Java-based image processing is used in many places. Hopefully some people can benefit from this tutorial. The code can be found from here

LibRaw Post Processing Pipeline Translated Into Python

LibRaw is a widely used open source library for raw image conversion. In many people's opinion, the most valuable part of LibRaw library is that it can handle various types of raw image formats, which is not commonly available to other libraries. But people are also interested in its post processing pipeline, which converts a interleaved RGB dot map to a colorful picture by using procedures such as white balance adjustment, demosaicing, gamma mapping, etc. People want to understand LibRaw's post processing pipeline but the code can be hard to deciphered. That is why I wrote a Python script which appears to match with LibRaw's post processing pipeline quite well. The hope is that by reading this Python script, it is easier for folks to understand what actually happens in LibRaw.

I should admit that some corners have been cut in this Python script. For example, the output of the script is only half the size of the input, which means no need for me to write a full-fledged demosaicing which matches the size of the input. However, what I found over the time is that quite often, the most puzzling part of post processing to people include me is how inputs from different color channels are scaled and mixed together. The scaling and mixing parts are covered in this Python script.

Here is the approach we take when writing this script: the input of this script is a raw .dng image. The image is taken using Samsung Galaxy S9 phone. The decoding before post processing is done using Python rawpy library. Python rawpy library is LibRaw wrapped in Python. After decoding, post processing is done using two parallel paths. The first path is to use postprocess() function provided by rawpy. Under the hood, it calls dcraw_process() from dcraw library written by Dave Coffin. The first path is used for reference. The second path is my own pipeline written in Python. The second path takes rawpy decoding outputs as inputs including raw image and some parameters. At the end of the script, we compare the reference and reconstructed image by my code and find them to be almost the same.

Now let me introduce the post processing code I wrote. It includes three main steps: 1) scale the input by white balance, 2) mix R/G/B by using color matrix, 3) Gamma scaling

In step 1, raw image containing RGB info is scaled by white balance (wb_scale). Since Bayer pattern is used for raw image input, there are two green channels (color4[:,:,1] and color4[:,:,3]). The white balance scale comes from the parameter of camera_whitebalance in rawpy decoding output. One can use another parameter of daylight_whitebalance but it makes the final output image yellowish. Since the next step of pipeline only needs three inputs, there is minor step called mix green which mixes two green channels as: color4[:,:,1] = (color4[:,:,1] + color4[:,:,3])/2.

color4[:,:,0] = raw_py.raw_image[0::2,1::2]*wb_scale[0]
color4[:,:,1] = raw_py.raw_image[0::2,0::2]*wb_scale[1]
color4[:,:,2] = raw_py.raw_image[1::2,0::2]*wb_scale[2]
color4[:,:,3] = raw_py.raw_image[1::2,1::2]*wb_scale[3]

In step 2, color_matrix is applied. I believe the purpose is convert camera sensor input to standard RGB image. color_matrix is also among rawpy decoding outputs. While white balance changes from photo to photo, color_matrix is camera's inherent property and it stays the same with the same camera setting.

pic_temp[:,:,0] = color_matrix[0,0]*color4[:,:,0] + color_matrix[0,1] * color4[:,:,1] + color_matrix[0,2] * color4[:,:,2]
pic_temp[:,:,1] = color_matrix[1,0]*color4[:,:,0] + color_matrix[1,1] * color4[:,:,1] + color_matrix[1,2] * color4[:,:,2]
pic_temp[:,:,2] = color_matrix[2,0]*color4[:,:,0] + color_matrix[2,1] * color4[:,:,1] + color_matrix[2,2] * color4[:,:,2]

Step 3 is Gamma mapping. Gamma mapping is a legacy from CRT era to compensate the nonlinear distortion of CRT monitor. But it outlives CRT. The parameter used in Gamma curve is Gamma(0.45, 4.5).

pic_temp = gamma_curve[pic_temp]

Finally we compare the results between reference image by rawpy's postprocess() function and reconstructed image by our code. It turns out that max delta between them is 2 output 255. Thus we claim that these two images are almost the same. The code/image can be found here.

Image White Balancing with Python

Below is a photo I took in the viewing deck of Taipei 101. Taipei 101 has a glass wall which is green color. Thus, all photos taken inside the building through the wall has a green cast including this one.

To remove the cast of green color, we use the technique of image white balancing. First, we need a reference object with known color. Fortunately this photo has clouds (pointed by the arrow). Clouds are good references since they are white. By averaging the pixels of cloud (mean(img_array[400,500:550,:])), the mean of R/G/B is [201.5, 254.9, 253.9]. Since white color has R/G/B channels roughly equal, to make it white, 52 needs to be added to R channel of the whole image. Like DC cancellation in communication, white balancing removes the bias of the signal.

Code below shows how to do white balancing

filename = 'IMG_9254.JPG'
img = image.load_img(filename)
img_array = np.array(image.img_to_array(img),dtype=np.uint8)
img_shape=img_array.shape
plt.figure()
plt.imshow(img_array)

img_array2 = img_array
img_array2 = np.array(img_array2, dtype=np.uint16)
img_array2[:,:,0] = img_array2[:,:,0] + 52 # white balancing
img_array2 = np.clip(img_array2, 0, 255)

The newly generated image after white balancing, img_array2,  is shown below. The color does look more natural. In case that you want to repeat this, code/image can be found here.

Note on Implementation of “Fast Noise Variance Estimation” by J. Immerkær

Noise variance estimation is a fundamental task of image processing. Among various approaches of noise variance estimate, fast noise variance estimation method proposed by J. Immerkær in [1] stands out because of its good balance between complexity and accuracy.

Based on J. Immerkær's method, to estimate the standard deviation of image noise \(\sigma\), we can use equations below:

\(F=
\begin{bmatrix}
1 & -2 & 1\\
-2 & 4 & -2\\
1 & -2 & 1\\
\end{bmatrix}
\)

\(\sigma=\frac{\sqrt{\pi/2}}{6(W-2)(H-2)}\sum_{I}|I(x,y)*F|\)

\(F\) is the high-pass filter. To estimate noise variance, filter \(F\) will be applied on the whole image \(I(x,y)\) with convolution. The sum of absolute value of convolution results is normalized by width of height of the image, which is \((W-2)(H-2)\). "-2" counts for the boundary effect. The summation is also scaled by \(\frac{\sqrt{\pi/2}}{6}\), which is explained next.

Assume a 3x3 block of \(I(x,y)\) with each element of \(x_{i}\) to be a Gaussian random variable with mean \(\mu\) and standard deviation \(\sigma\)
\(I=
\begin{bmatrix}
x_{1} & x_{2} & x_{3}\\
x_{4} & x_{5} & x_{6}\\
x_{7} & x_{8} & x_{9}\\
\end{bmatrix}
\)
Then \(|I*F|=|x_{1}-2*x_{2}+x_{3}-2*x_{4}+4*x_{5}-2*x_{6}+x_{7}-2*x_{8}+x_{9}|\)
Since \(E(|I*F|^2)=36\sigma^2\), \(|I*F|=6|s|\) where s is Gaussian random variable with mean 0 and standard deviation \(\sigma\). We can derive the distribution of \(y=|s|\) and it is \(\frac{2}{\sqrt{2\pi}\sigma}e^{-\frac{y^2}{2\sigma^2}}\)  for \(y>=0\) and 0 for \(y < 0\). <0 .="" br=""> 

Since \(E(|s|)=\sqrt{\frac{2}{\pi}}\sigma\), it explains why the summation is also scaled by \(\frac{\sqrt{\pi/2}}{6}\).


[1] J. Immerkær, “Fast Noise Variance Estimation”, Computer Vision and Image Understanding, Vol. 64, No. 2, pp. 300-302, Sep. 1996