LibRaw Post Processing Pipeline Translated Into Python

LibRaw is a widely used open source library for raw image conversion. In many people's opinion, the most valuable part of LibRaw library is that it can handle various types of raw image formats, which is not commonly available to other libraries. But people are also interested in its post processing pipeline, which converts a interleaved RGB dot map to a colorful picture by using procedures such as white balance adjustment, demosaicing, gamma mapping, etc. People want to understand LibRaw's post processing pipeline but the code can be hard to deciphered. That is why I wrote a Python script which appears to match with LibRaw's post processing pipeline quite well. The hope is that by reading this Python script, it is easier for folks to understand what actually happens in LibRaw.

I should admit that some corners have been cut in this Python script. For example, the output of the script is only half the size of the input, which means no need for me to write a full-fledged demosaicing which matches the size of the input. However, what I found over the time is that quite often, the most puzzling part of post processing to people include me is how inputs from different color channels are scaled and mixed together. The scaling and mixing parts are covered in this Python script.

Here is the approach we take when writing this script: the input of this script is a raw .dng image. The image is taken using Samsung Galaxy S9 phone. The decoding before post processing is done using Python rawpy library. Python rawpy library is LibRaw wrapped in Python. After decoding, post processing is done using two parallel paths. The first path is to use postprocess() function provided by rawpy. Under the hood, it calls dcraw_process() from dcraw library written by Dave Coffin. The first path is used for reference. The second path is my own pipeline written in Python. The second path takes rawpy decoding outputs as inputs including raw image and some parameters. At the end of the script, we compare the reference and reconstructed image by my code and find them to be almost the same.

Now let me introduce the post processing code I wrote. It includes three main steps: 1) scale the input by white balance, 2) mix R/G/B by using color matrix, 3) Gamma scaling

In step 1, raw image containing RGB info is scaled by white balance (wb_scale). Since Bayer pattern is used for raw image input, there are two green channels (color4[:,:,1] and color4[:,:,3]). The white balance scale comes from the parameter of camera_whitebalance in rawpy decoding output. One can use another parameter of daylight_whitebalance but it makes the final output image yellowish. Since the next step of pipeline only needs three inputs, there is minor step called mix green which mixes two green channels as: color4[:,:,1] = (color4[:,:,1] + color4[:,:,3])/2.

color4[:,:,0] = raw_py.raw_image[0::2,1::2]*wb_scale[0]
color4[:,:,1] = raw_py.raw_image[0::2,0::2]*wb_scale[1]
color4[:,:,2] = raw_py.raw_image[1::2,0::2]*wb_scale[2]
color4[:,:,3] = raw_py.raw_image[1::2,1::2]*wb_scale[3]

In step 2, color_matrix is applied. I believe the purpose is convert camera sensor input to standard RGB image. color_matrix is also among rawpy decoding outputs. While white balance changes from photo to photo, color_matrix is camera's inherent property and it stays the same with the same camera setting.

pic_temp[:,:,0] = color_matrix[0,0]*color4[:,:,0] + color_matrix[0,1] * color4[:,:,1] + color_matrix[0,2] * color4[:,:,2]
pic_temp[:,:,1] = color_matrix[1,0]*color4[:,:,0] + color_matrix[1,1] * color4[:,:,1] + color_matrix[1,2] * color4[:,:,2]
pic_temp[:,:,2] = color_matrix[2,0]*color4[:,:,0] + color_matrix[2,1] * color4[:,:,1] + color_matrix[2,2] * color4[:,:,2]

Step 3 is Gamma mapping. Gamma mapping is a legacy from CRT era to compensate the nonlinear distortion of CRT monitor. But it outlives CRT. The parameter used in Gamma curve is Gamma(0.45, 4.5).

pic_temp = gamma_curve[pic_temp]

Finally we compare the results between reference image by rawpy's postprocess() function and reconstructed image by our code. It turns out that max delta between them is 2 output 255. Thus we claim that these two images are almost the same. The code/image can be found here.