1 Lossy Compression of Color Mosaic Images Stephanie Kwan and Karen Zhu demosaicking on the compressed mosaic image later. Two Abstract—“Raw” color mosaic images produced by consumer different approaches will be explored and compared with digital cameras pose interesting lossy compression problems. conventional demosaicking-first, compression-later Conventionally, “raw” color mosaic images are first demosaicked approaches. and color-balanced, before being compressed for storage or transmission. However, the new compression-first demosaicking- In the next section, the conventional demosaicking-first, later approach has attracted much attention due to its efficiency. compression-later approaches are outlined and serve as This paper presents two lossy compression methods for reference points for the new compression-first, demosicking- compression-first demosaicking-later approach. The resulting later approaches. In section III, existing compression-first reconstructed images are compared with the ones compressed demosaicking-later designs are discussed in brief. Section IV using the conventional methods. It is shown that the proposed introduces our new approaches to this problem and discusses methods have much advantage over the conventional methods. the algorithms in detail. Experimental setup is given in section V, while results and analysis are given in section VI. I. INTRODUCTION Finally, suggestions for future works are outlined in section VII. M OST digital cameras are equipped with sensors that can only detect the intensity but not the frequency of incoming light. A color filter array (CFA) is placed in front of II. THE CONVENTIONAL DEMOSAICKNG-FIRST COMPRESSION- LATER APPROACHES the light sensors to enable the intensity information of one Conventionally, the “raw” mosaic image data is first color component, commonly red, green, or blue to be interpolated with a demosaicking algorithm, followed by a recorded. The resulting image contains samples of these three color balancing scheme to obtain a continuous tone color primary colors, interleaved in a two-dimensional (2-D) grid, RGB image. Then, a compression scheme is applied to reduce or color mosaic pattern. The most commonly used CFA the size of the image for transmission or storage. To view the pattern is the Bayer pattern (Fig. 1). To obtain the true compressed image, a matching decompression scheme is continuous tone color image from the mosaic image, a process applied, and then the image is displayed. Fig. 2 below called color-demosaicking is used to estimate the value of the illustrates this process with the RGB image compressed using other two colors at the same pixel . a lossy compression scheme. Color RGB image Raw mosaic Demosaic Balance X image transmission, Compressed storage Lossy Fig. 1. Bayer Pattern RGB image Decompression Compression X1’ With limited storage space and processing power in digital camera, image data compression is a key component for Fig. 2. Conventional compression scheme for “raw” mosaic image. digital cameras design. Conventionally, the “raw” mosaic image is first interpolated with a demosaicking algorithm, and In order to use the path illustrated in Fig. 2 as the reference then compressed for storage. Since the demosaicking process for comparison with the two new approaches later discussed does not increase the information content of the original in section IV, two specific lossy compression methods are image, but only introduce redundant data, performing used and implemented. compression after demosaicking is no more efficient then 1) RGB Compression: The RGB image obtained after color performing compression directly on the “raw” mosaic image. balancing is split into three separate channels. Each channel, In fact, performing compression after demosaicking usually red, green, and blue, is compressed independently using 5/3 introduces longer processing time and larger storage wavelet transform  and coded with the SPIHT algorithm requirement . . Bit allocation is performed such that the distortion of each The goal of this paper is to explore the scheme of lossily channel is set to an equal value. compressing “raw” mosaic image first, and performing 2) YCrCb Compression: The RGB image obtained after 2 color balancing is transformed into the YCrCb color space. wavelet transform is then proposed for doing compression The luminance and the chrominance channels are then directly on mosaic image without de-interleaving the RGB compressed independently using 5/3 wavelet transform and components. Excellent results have been shown using this coded with the SPIHT algorithm. approach. To evaluate the quality of the compression methods, The reason behind the good performance of wavelet Composite Peak Signal to Noise Ratio (CPSNR) is used. transform on “raw” color mosaic image is discussed in detail in . The basic idea is that the four subbands generated by III. EXISTING COMPRESSION-FIRST DEMOSAICKING-LATER one level wavelet decomposition have clear connections to the SCHEMES Bayer Pattern mosaic image. For example, applying the 2D In contrast to the conventional approach, the compression- low-pass filter of the 5/3 wavelet transform to the Bayer first demosaicking-later approach directly compresses the mosaic image, the LL band can be interpreted as the “raw” mosaic image. The basic idea of this approach is luminance channel of the full color image. On the other hand, outlined in Fig. 3 below. the 2D 5/3 high-pass filter has the effect of spectral decorrelation. The HH subband resulted thus contains the Raw mosaic Lossy mosaic image details of the green channel and a highly smoothed color Compression Decompression Image Y’ difference signal. Y The results presented in  are very interesting since it suggests that interleaved compression performs as well as the Compressed Color Demosaic deinterleaved compression, if not better. However, Zhang and RGB image Balance X2’ Wu  only focused on the lossless compression of the raw Fig. 3. Compression-first demosaicking-later scheme for “raw” mosaic mosaic image, while in real life lossy compression is more image. often required. In the next section, two lossy compression schemes based on the lossless approaches suggested in  are There are a number of literatures in this area, each using a proposed. different compression method. They can be roughly categorized into two groups: ones that separate the “raw” IV. PROPOSED LOSSY COMPRESSION SCHEME mosaic image into RGB or YCrCb sub-channels for In this section we propose two methods for lossily compression, and ones that do not separate the “raw” mosaic compressing color mosaic images, the deinterleaved approach image, but compress it directly. The terms deinterleaved and and the interleaved approach. interleaved will be used to represent these two groups respectively. A. Deinterleaved Lossy Compression A. Deinterleaved Compression As illustrated in Fig. 4, the (square 2N x 2N) “raw” mosaic Most of the early works in the area of mosaic image image is first separated into three subimages. Each subimage compression used deinterleaved compression. Tsai  only contains one of three colors, namely red, green, or blue. proposed a scheme of separating the “raw” mosaic image into The approach used to separate the “raw” mosaic image into the three color groups, red, green, and blue, and compress the red, green, and blue subimages are the same as the each group separately with adaptive discrete cosine transform approach used in , and is outlined below. (ADCT). Xie et al.  and Koh et al.  not only separated The red and blue subimages are created by simply merging the “raw” image data into RGB groups, they also transformed neighboring pixels of the same color to form square the RGB groups into the YCrCb groups. After that, Xie et al. subimages that has a size of N x N.  compressed each group with JPEG-LS, while Koh et al.  compress each group using JPEG baseline. Recently, B0, 0 G0,1 B0, 2 G0,3 B0, 4 G0,5 B0, 6 G0, 7 Zhang and Wu  have compared the above approaches by G1, 0 R1,1 G1, 2 R1,3 G1, 4 R1,5 G1, 6 R1, 7 applying both JPEG-LS and JPEG2000 compression on B2, 0 G2,1 B2, 2 G2 , 3 B2, 4 G2 , 5 B2, 6 G2 , 7 separate RGB groups, and concluded that JPEG-LS, which uses Differential Pulse Code Modulation (DPCM) coding, G3, 0 R3,1 G3, 2 R3,3 G3, 4 R3,5 G3,6 R3,7 performs better then JPEG2000, which uses wavelet B4, 0 G4,1 B4, 2 G4 , 3 B4, 4 G4 , 5 B4, 6 G4 , 7 transform. G5, 0 R5,1 G5, 2 R5,3 G5, 4 R5,5 G5,6 R5, 7 B. Interleaved Compression B6, 0 G6,1 B6, 2 G6,3 B6, 4 G6,5 B6, 6 G6, 7 In addition to deinterleaved compression, Zhang and Wu G7 , 0 R7 ,1 G7 , 2 R7 ,3 G7 , 4 R7 ,5 G7 , 6 R7, 7  also explored methods for doing interleaved compression. They first observed that JPEG2000 out performs JPEG-LS when applied to interleaved “raw” mosaic image data, proving that wavelet decomposition has much advantage over DPCM on interleaved color pixels. A unique so-called Mallat packet 3 Sub-image R Sub-image B R1,1 R1,3 R1,5 R1,7 B0,0 B0, 2 B0, 4 B0,6 Raw Wavelet R3,1 R3,3 R3,5 R3,7 B2,0 B2, 2 B2, 4 B2,6 mosaic image Separate into 3 Transform Y subimages R, G, B (5/3 wavelet) Lossy R5,1 R5,3 R5,5 R5,7 B4,0 B4, 2 B4, 4 B4,6 Compression R7 ,1 R7 ,3 R7 ,5 R7 ,7 B6,0 B6, 2 B6, 4 B6,6 Combine subimages SPHITE +Arithmetic back to mosaic pattern Encoding Y’ The green pixels are arranged in a quincunx pattern. If we simply shift the columns of green pixels by 1 pixel, and merge Demosaic the green pixels into a rectangular structure, a lot of false high + Inverse Wavelet SPHITE + Arithmetic Color Balancing Transform Decoding frequencies will be created. Therefore, first, we create an (5/3 wavelet) image that has the same size as the original image, but only Decompression contains the green components. Locations where pixels are Final RGB image non-green are set to 0. X2’ 0 G0,1 0 G0 , 3 0 G0,5 0 G0 , 7 Fig. 4. Deinterleaved compression process of color mosaic image. G1, 0 0 G1, 2 0 G1, 4 0 G1, 6 0 0 G2,1 0 G2, 3 0 G2 , 5 0 G2, 7 B. Interleaved Lossy Compression G3, 0 0 G3, 2 0 G3, 4 0 G3 , 6 0 Fig. 5 illustrates the process of performing lossy compression directly on the interleaved color mosaic image. 0 G4,1 0 G4, 3 0 G4 , 5 0 G4, 7 This approach is significantly simpler then the previous one G5, 0 0 G5 , 2 0 G5, 4 0 G5 , 6 0 due to the fact that the “raw” mosaic image is not separated 0 G6,1 0 G6 , 3 0 G6 , 5 0 G6 , 7 into three subimages, but compressed directly. The mosaic G7 , 0 0 G7 , 2 0 G7 , 4 0 G7 , 6 0 image is lossily compressed using the same scheme used in Deinterleaved Lossy Compression mentioned previously. It is transformed using the Mallet packet wavelet transform, and This image is then passed through a low-pass filter H to lossily compressed using SPHITE, together with an arithmetic avoid the generation of unwanted high frequencies. coder. Fig. 6 shows the “raw” color mosaic image of ⎡0 0 1 ⎤ “isochart” image, and Fig. 7 shows a five level Mallet packet H = ⎢0 2 4⎥ 1⎢ ⎥ wavelet decomposition of it. 4 ⎢0 0 1 ⎥ ⎣ ⎦ This lowpass-filtered green subimage is then subsampled Raw (2:1) to keep the number of data points the same, and at the mosaic image Wavelet SPHITE +Arithmetic same time transform the green subimage into a 2N x N Y Transform Encoding (5/3 wavelet) rectangular array. Since the Mallet packet wavelet transform takes in square Lossy Compression images, we first do a 1D wavelet transform on the rectangular green subimage to separate the image into 2 subbands, each mosaic image Inverse Wavelet SPHITE + Arithmetic Y’ Transform which is a square of N x N. Then for each of these subbands, (5/3 wavelet) Decoding we perform Mallet packet wavelet transform and the subsequent compression in the same manner as the red and Decompression Demosaic blue subimages. + To reconstruct the compressed color mosaic image, Color Balancing arithmetic and SPHIT decoding is first performed on the three separate subimages, followed by inverse Mallet packet Final RGB image transform with inverse 5/3 wavelet. The three subimages are X2’ then combined to form a close approximation of the original “raw” mosaic image. At this point, the color mosaic image Fig. 5. Interleaved compression process of color mosaic image. obtained can be compared with the original raw mosaic image to evaluate the Peak Signal to Noise Ratio (PSNR) that results from the compression process. The last step is to perform demosaicking and color balancing on the processed color mosaic to display a normal continuous tone RGB color image. 4 to green rather than red or blue, attempts have been made to distribute distortions unequally in the subimages to see if an unequal distribution of distortion would result in images of better visual quality. We tried constraining the distortion ratio of R:G:B = 1:2:1, 2:1:2 as well as 1:3:1. Among these ratios, R:G:B ratio of 2:1:2 and R:G:B ratio of 1:1:1 give the best results. D. Comparison Schemes Three types of comparison are carried out in our experiment to show different aspects of this investigation. 1) CPSNR comparison of Conventional approaches: As outlined in section II, two methods are implemented the conventional approach, namely the RGB compression scheme and the YCrCb compression scheme. Referring to Fig. 2, the Fig. 6. “raw” color mosaic image of Isochart. RGB image X before the lossy compression block and the RGB image X1’ after the decompression block are compared. The CPSNR between these two images is calculated using equation (1). The corresponding compression ratio is also calculated from the bitrate needed to code X1’. ⎛ ⎞ ⎜ ⎟ (1) ⎜ 255 2 ⎟ CPSNR = 10 log10 ⎜ 3 W H ⎟ ∑∑∑ [I (i, j, k ) − I (i, j, k )] 1 ⎜ ⎟ 2 ⎜ 3HW in out ⎟ ⎝ k =1 i =1 j =1 ⎠ 2) PSNR comparison between mosaic images: It is hard to compare the CPSNR values in the proposed schemes outlined in section IV due to the involvement of the demosaicking and color balancing process in the path. Instead, PSNR is calculated between the original raw mosaic image Y and the reconstructed mosaic image Y’ in Fig. 3. The PSNR calculation is done by using equation (2). The corresponding Fig. 7. 5 Level Mallet packet wavelet decomposition of Isochart. compression ratio is calculated from the bitrate of coding Y’. ⎛ 255 2 ⎞ V. EXPERIMENTAL SETUP ⎜ d ⎟ PSNR = 10 log10 ⎜ ⎟ (2) ⎝ ⎠ A. Test Images 3) Visual comparison between the conventional and Test images used for our experiment are “raw” mosaic proposed approaches: Due to the very different system setup, images taken from Nikon D70 digital camera, which has a CPSNR comparison between conventional and proposed Bayer Pattern color filter array. A Matlab program ISET approaches would not generate meaningful results. Visual provided by Joyce Ferrell is used to extract data information comparison is carried out between the final RGB pictures of the raw mosaic images. displayed to determine the difference in overall quality. B. Demosaicking Algorithm and Color Balancing A simple Laplacian demosaicking algorithm and ‘Gray World’ color balancing algorithm supplied by the ISET tool VI. RESULTS AND ANAYLYSIS was used to perform demosaicking and color balancing on all The results presented in this section are based on the test test images. image “People” and the setup outlined in the previous section. Fig. 10 (a) shows the “People” image in its uncompressed, C. R:G:B distortion ratio experiment demosaicked and color balanced form. This would correspond In Deinterleaved Lossy Compression, a natural way to to the RGB image X in Fig. 2. This serves as a reference to the performing bit allocation among the red, green and blue best quality output a compression scheme can produce. subimages is to distribute distortion evenly among the three subimages so that each of them have the same average distortion. However, since the human visual system is more sensitive 5 A. CPSNR Comparison of Conventional Approaches image “people” compressed using the conventional approach with RGB subchannels compressed independently. Due to the CPSNR vs Compression Ratio of Demosaicking-First Compression-Later Design high compression ratio, the color picture quality is severely 37 YCrCb compressed deteriorated. Blocking artifacts caused by subband coding at 36 RGB compressed low rates can be seen clearly. 35 On the other hand, Fig. 10 (c) shows the same image being compressed with the proposed deinterleaved approach. With 34 very similar compression ratio, it has much better overall CPSNR (dB) 33 quality and little artifacts. This comparison clearly shows that 32 the proposed deinterleaved approach outperforms the conventional RGB compression. 31 30 29 28 2 4 6 8 10 12 14 16 18 20 22 Compression Ratio Fig. 8. CPSNR compression between the conventional approaches of test image “People” The blue line representing compression done in the YCrCb color space is higher then the red line representing compression done in RGB space. This is expected behavior, (a) since color transform from RGB space to YCrCb space decorrelates the interdependent red, green and blue channels, thus resulting in a coding gain. B. PSNR Comparison between mosaic images PSNR vs Compression Ratio of Raw Mosaic Image 44 42 Raw mosaic image compressed directly R, G, B compressed separately 40 (b) (c) 38 PSNR (dB) 36 34 32 30 28 5 10 15 20 25 30 35 40 Compression Ratio Fig. 9. PSNR compression between the conventional approaches of test image (d) (e) “People” Fig. 10. “People” results. (a) Uncompressed, demosaicked and color balanced image (b) Compressed using conventional approach, with RGB compressed The blue line, representing compression performed directly separately. Compression ratio=14.39, CPSNR=29.33, distortion ratio of on “raw” color mosaic image is higher then the red line, R:G:B=1:1:1. representing compression on separate RGB channels of “raw” (c) Compressed using proposed deinterleaved approach. Compression ratio=14.3, CPSNR=28.897, distortion ratio of R:G:B=1:1:1. color mosaic image. The reason for this result is that as (d) Compressed using conventional approach, with YCrCb compressed discussed in section III, the 5/3 wavelet transform decorrelates separately. Compression ratio=15.09, CPSNR=30.62, distortion ratio of the interdependent red, green and blue colors in the mosaic Y:Cr:Cb=1:1:1. image, thus giving rise to the coding gain observed. (e) Compressed using proposed interleaved approach. Compression ratio=14.45, CPSNR=30.074. C. Visual Comparison 2) Comparison of Conventional YCrCb Compression and 1) Comparison of Conventional RGB Compression and Proposed Interleaved Compression: Comparing Fig. 10 (d) Proposed Deinterleaved Compression: Fig. 10 (b) shows the and Fig. 10 (e) shows that the proposed interleaved approach 6 gives much better result then the YCrCb compressed similar visual quality. Superiority of images from one method approach, especially in terms of color distribution. While Fig. to the other largely depends on personal preference and the 10 (d) shows large blocks of underlying red and blue tone, choice of image. However, since the interleaved approach Fig. 10 (e) has rather natural color scheme and little noticeable doesn’t require the separation of the mosaic image into red, artifact. blue and green subimages, it has a simpler algorithm and often 3) Comparison of Deinterleaved Compression and requires much less processing time. Thus, since the resulting Interleaved Compression: By comparing Fig. 10 (c) and Fig. compressed images are similar, the Interleaved approach, 10 (e), it is observed that under similar compression ratio, the having an advantage in processing time, maybe a better choice deinterleaved approach and the interleaved approach produce overall. color images of similar visual quality. It is hard to judge which of the approaches is more superior to the other one. APPENDIX Comparison of these two approaches using other test images also results in the same conclusion that under the same The following figures show the experimental results from compression ratio, both approaches result in images of similar additional test images that we have used for comparison. visual quality. For the results of other images, please refer to the Appendix. VII. FUTURE WORKS During the progress of this work, we have identified several interesting directions to extend this work. We will briefly discuss them in this section. 1) In the proposed deinterleaved approach, wavelet transform is applied to each of the three color sub- channels. Although wavelet transform has a clear advantage for interleaved “raw” mosaic color image, it (a) may not be the optimal choice for the de-interleaved image data since we are not exploiting its advantage of decorralating the three colors. Using other transform coders, such as DPCM, might give better results for the Deinterleaved approach. 2) While doing the experiment, it is noticed that the demosaic algorithm play a very important role in the final color image quality. Different demosaic algorithms can be explored to see what type of demosaic algorithms will be the most suitable one for our specific compression (b) (c) scheme. 3) In this paper, we have implemented the conventional compression pipelines such that the transform and compression schemes are the same as that of our proposed approaches. Further work can go into using well established standards, such as JPEG2000, to compress images in the conventional pipeline, and compare them to results of our proposed approaches. VIII. CONCLUSION We have proposed two lossy compression schemes for (d) (e) “raw” color mosaic images based on the lossless compression Fig. 16. “Isochart” results. (a) Uncompressed, demosaicked and color balanced image schemes outlined in . Our test results show that using a (b) Compressed using conventional approach, with RGB compressed compression-first, demosaicking-later design, both separately. Compression ratio=12.72, CPSNR=19.4, distortion ratio of deinterleaved and interleaved compression methods R:G:B=1:1:1. (c) Compressed using proposed deinterleaved approach. Compression outperform the conventional demosaicking-first compress- ratio=12.9, CPSNR=19.28, distortion ratio of R:G:B=1:1:1. later scheme. The visual quality of the final compressed color (d) Compressed using conventional approach, with YCrCb compressed images is significantly better then the ones produced using the separately. Compression ratio=12.72, CPSNR=28.09, distortion ratio of Y:Cr:Cb=1:1:1. conventional approach. (e) Compressed using proposed interleaved approach. Compression Between the two proposed methods, deinterleaved and ratio=13.02, CPSNR=27.93. interleaved compression result in compressed color images of 7 DIVISION OF WORK The amount of work is about equally shared among the two. The following is a rough breakdown. Stephanie: - Implement reference paths - Implement deinterleaved algorithm - Wavelet lifting - presentation Karen: (a) - Implement interleaved algorithm - Mallet packet wavelet decomposition - SPIHT coding - report REFERENCES  Ning Zhang and Xiaolin Wu. “Lossless Compression of Color Mosaic Images,” IEEE Trans. ImageProcessing, vol. 15, No. 6, pp. 1379–1388, June 2006.  Y. Tim Tsai. “Color Image Compression for Single-Chip Cameras,” IEEE Trans. Electron Devices, vol. 38, No. 5, pp. 1226–1232, May 1991. (b) (c)  Michael D. Adams, Faouzi Kossentini. “Reversible Integer-to-Integer Wavelet Transforms for Image Compression: Performance Evaluation and Analysis,” IEEE Trans. Image Processing, vol. 9, No. 6, pp. 1010– 1024, June 2000.  David Taubman, and Michael Marcellin, “JPEG2000 Image Compression Fundamentals, Standards, and Practice” Boston: Kluwer Academic Publishers, 2002.  Xiang Xie, GuoLin Li, ZhiHua Wang, Chun Zhang, DongMei Li and XiaoWen Li. “A Novel Method of Lossy Image Compression for Digital Image Sensors with Bayer Color Filter Arrays,” in Conf. Rec. 2005 IEEE Int. Conf. Circuits and Systems, pp. 4995–4998.  Chin Chye Koh, Jayanta Mukherjee, and Sanjit K. Mitra. “New Efficient Methods of Image Compression in Digital Cameras with Color Filter Array,” IEEE Trans. Consumer Electronics, vol. 49, No. 4, pp. 1448– 1456, November 2003. (d) (e) Fig. 17. “women” results. (a) Uncompressed, demosaicked and color balanced image (b) Compressed using conventional approach, with RGB compressed separately. Compression ratio=11.67, CPSNR=29.38, distortion ratio of R:G:B=1:1:1. (c) Compressed using proposed deinterleaved approach. Compression ratio=11.44, CPSNR=29.34, distortion ratio of R:G:B=1:1:1. (d) Compressed using conventional approach, with YCrCb compressed separately. Compression ratio=11.42, CPSNR=34.35, distortion ratio of Y:Cr:Cb=1:1:1. (e) Compressed using proposed interleaved approach. Compression ratio=11.36, CPSNR=32.03. ACKNOWLEDGMENT We would like to thanks Markus Flierl, Joyce Farrell, and Professor Brian Wandell for their great help on this project.