Fast Raw Bayer CFA Sensor Compression on CUDAThere are many image processing tasks in camera applications for which it is strictly required to store RAW frames without converting them to RGB, that can be done later or offline. This is the case for many applications including for street view, aerial imaging, and satellite imaging applications. To that end, the RAW Bayer Codec was built to compress RAW data prior to storing data on an SSD. The RAW Bayer codec is intended for the compression and storage of raw image data captured by cameras using a Bayer filter. This data follows a Bayer pattern (RGGB, BGGR, GRBG, or GBRG), in which each pixel captures only one color channel. Lossless or visually lossless compression could be used to preserve the original data for later processing. Applications for RAW Bayer Compression
We can sometimes use lossless encoding on the RAW images we capture. The compression ratio (CR) of these algorithms is usually around two, which isn't too much. Even so, this is a mathematically lossless algorithm, meaning that each image will be restored exactly as it was right after it was taken (before compression). Here are some possible lossless algorithms: Lossless JPEG (1–16 bits), JPEG-LS (1–16 bits), and JPEG2000 (up to 16 bits). Sometimes you've got to use lossless compression, especially for scientific stuff. Although it makes sense to apply lossy encoding to RAW images to achieve a much better compression ratio, we must control the degradation of image quality, which is essential for such algorithms. Even minor distortions in the RAW-encoded frames could result in significant artifacts that would only be apparent after the RAW-to-RGB conversion. This approach could achieve a compression ratio much greater than two times, so we need to look for an optimized solution that takes into account both image quality and compression ratio requirements.
Benefits of RAW Bayer Compression
Key Features of RAW Bayer Codec
We can integrate RAW Bayer Codec into your imaging system to perform the whole job fully on GPU. Please check the description of our GPU Image & Video Processing SDK to evaluate what we can do on GPU. Image Quality and CR ConsiderationsFor RAW bayer frames from conventional 12-bit image sensors from Sony, Gpixel, On-Semi, and CMOSIS, we usually achieve a compression ratio of CR~2 for lossless algorithms. This is because useful data also includes noise, which could be important in scientific applications that requires lossless data compression. Lossless encoding also has lower performance compared to lossy codecs. Applying lossy encoding means that by default we get both losses and better performance, which depends on the image content, utilized parameters, and hardware. It's important to know how to control the level of loss to store high-quality images for a particular task. In most cases, we can compress RAW frames at a CR of 5–10 times and achieve a PSNR of 50–55 dB or better. The PSNR is computed for the original RAW bayer image and the same image after processing/encoding/decoding/restoring. We understand that PSNR metrics has limited use, and we also perform visual image quality assesment to choose appropriate default processing parameters for a project. We visually compare not RAW images, but RGB which were processed with the same parameters. For example, we shouldn't see any significant differences at viewing source and processed images at 500% zoom. For applications which require the best image quality we could limit CR by the range of 4-5 and in that case we can achieve PSNR more than 60 dB. We need to emphasize that our RAW Bayer Codec with lossy encoding is not just a codec. This is a sophisticated solution with a wide range of capabilities, extending far beyond data compression and decompression. It strikes a balance between preserving raw data and performing additional image processing to improve CR and minimize image distortion. Thanks to our technology, we can achieve significantly better CR and PSNR than conventional JPEG and JPEG2000 codecs for raw bayer images. We can also customize our software to optimize results for specific applications once we understand the project requirements. Numerical criteria for image quality estimationRAW Bayer Codec can work both in lossless and lossy modes. In lossy mode we get some image distortions and it's important to measure the level of losses and to evaluate them both visually and numerically. And we need to check not only a signal, but also a noise to be sure that we can reproduce an original image with high precision, and we can restore both signal and noise at the same time. That's why we've calculated the following:
For available raw frame with ColorChecker from DALSA/FLIR camera (20 MPix, 4512 x 4512, 12-bit, RGGB) we've applied lossy raw bayer encoding with CR = 10 and have got PSNR = 45.3 dB, Standard deviation (sigma) has changed less then 2%, SNR change was less then 0.15 dB. We've done measurements for each patch of the ColorChecker. GPU pipeline for RAW decoding, processing and visualization
The above pipeline is computationally demanding and it's not easy to achieve high performance for that task. For visualizing we need to reach up to 60 fps, and for most resolutions this could be done on NVIDIA GPU in realtime. We utilize our GPU Image Processing SDK for the above pipeline and it can handle data rates up to 2-3 GPix/s on GeForce RTX 4090. Such a solution is available for all NVIDIA GPUs: mobile, laptop, desktop, server. |