diff --git a/README.md b/README.md index ac28265..170c99b 100644 --- a/README.md +++ b/README.md @@ -31,9 +31,9 @@ ISP pipeline for `InfiniteISP_ReferenceModel v1.0` 3. **Dataset Processing**: The model facilitates execution for multiple images with different or same configuration files. -4. **Video Processing**: The model also features a video processing script that allows for sequential frame processing, with operational 3A Statistics data flowing between frames. - +4. **Video Processing**: The model also includes a video processing feature that allows for sequential frame processing, with operational 3A Statistics data flowing between frames. +5. **Support Multiple RAW formats**: The model supports renowned raw formats, for example, DNG (predominantly employed by Android devices), NEF (specific to Nikon devices), and CR2 (Canon's signature format). It can also process the .raw image format, a one-dimensional pixel data array with no metadata header. ## Objectives @@ -59,6 +59,7 @@ The table below provides a feature list of the model. The version `1.0` of the m | Gamma Correction |Implements a LUT from config | | Auto Exposure | [Auto Exposure](https://www.atlantis-press.com/article/25875811.pdf)
- AE stats calculations based on skewness | | Color Space Conversion | YCbCr digital
- BT 601
- Bt 709
|YCbCr digital
- BT 601
- Bt 709
| +| Edge Enhancement / Sharpening | Simple unsharp masking with strength control| | Noise Reduction | [Non-local means filter](https://www.ipol.im/pub/art/2011/bcm_nlm/article.pdf)
- Implements intensity level difference through a LUT| | RGB Conversion | Converts YCbCr digital image to RGB| | Invalid Region Crop | Crops image to a fixed size| @@ -103,19 +104,20 @@ RAW_DATA = './in_frames/normal/data' ## How to Run on Pipeline on Multiple Images/Dataset -There are two scripts that run Infinite-ISP on multiple images: +There is another script [isp_pipeline_multiple_images.py](isp_pipeline_multiple_images.py) that runs Infinite-ISP on multiple images with two modes: -1. [isp_pipeline_dataset.py](isp_pipeline_dataset.py) +1. DATASET PROCESSING
Execute multiple images. Raw image should have its own config file with name `-configs.yml` where `` is raw filename otherwise the default configuration file [configs.yml](config/configs.yml) is used. + For raw image format such as, NEF, DNG and CR2 we have also provided a funcationality to extract sensor information provided in these raw files metadata and update default config file. -2. [video_processing.py](video_processing.py) -
Each image in the dataset is considered as video frame in sequence. All images use the same configuration parameters from [configs.yml](config/configs.yml) and 3A Stats calculated on a frame are applied to the next frame. +2. VIDEO MODE +
Each image in the dataset is considered as video frame in sequence. All images use the same configuration parameters from [configs.yml](config/configs.yml) and 3A stats calculated on a frame are applied to the next frame. After cloning the repository and installing all the dependencies follow the following steps: -1. Set `DATASET_PATH` to dataset folder. For example if images are in in [in_frames/normal/data](in_frames/normal/data) folder +1. Set `DATASET_PATH` to dataset folder. For example if images are in [in_frames/normal/data](in_frames/normal/data) folder ```python DATASET_PATH = './in_frames/normal/data' ``` @@ -124,17 +126,17 @@ DATASET_PATH = './in_frames/normal/data' ```shell git submodule add -git submodule update –-init –-recursive +git submodule update --init --recursive ``` -4. After adding git repository as a submodule update `DATASET_PATH` variable in [isp_pipeline_dataset.py](isp_pipeline_dataset.py) to `./in_frames/normal/`. Git does not allow to import a repository’s subfolder using a submodule. You can only add an entire repository and then access the folder. If you want to use images from a subfolder of a submodule modify the `DATASET_PATH` variable in [isp_pipeline_dataset.py](isp_pipeline_dataset.py) or [video_processing.py](video_processing.py) accordingly. +4. After adding git repository as a submodule update `DATASET_PATH` variable in [isp_pipeline_multiple_images.py ](isp_pipeline_multiple_images.py ) to `./in_frames/normal/`. Git does not allow to import a repository’s subfolder using a submodule. You can only add an entire repository and then access the folder. If you want to use images from a subfolder of a submodule modify the `DATASET_PATH` variable in [isp_pipeline_multiple_images.py](isp_pipeline_multiple_images.py) accordingly. ```python DATASET_PATH = './in_frames/normal/' ``` -5. Run `isp_pipeline_dataset.py` or `video_processing.py` +5. Run `isp_pipeline_multiple_images.py` 6. The processed images are saved in [out_frames](out_frames/) folder. ## Test Vector Generation diff --git a/config/configs.yml b/config/configs.yml index 5596481..fffa0dc 100644 --- a/config/configs.yml +++ b/config/configs.yml @@ -124,6 +124,12 @@ color_space_conversion: conv_standard: 2 is_save: false +sharpen: + is_enable: true + sharpen_sigma: 5 + sharpen_strength: 1 + is_save: false + 2d_noise_reduction: is_enable: false window_size: 9 diff --git a/docs/UserGuide.md b/docs/UserGuide.md index b18b679..388eb49 100644 --- a/docs/UserGuide.md +++ b/docs/UserGuide.md @@ -153,7 +153,15 @@ Below parameters are present each ISP pipeline module they effect the functional | color_space_conversion | Details | |------------------------|------------------------------------------------------------------------------------ | | conv_standard | The standard to be used for conversion
- `1` : Bt.709 HD
- `2` : Bt.601/407 | - + +### Edge Enchancement / Sharpening + +| Sharpening | Details | +|--------------------|---------------------------------------------------| +| is_enable | When enabled applies the sharpening | +| sharpen_sigma | Define the Standard Deviation of the Gaussian Filter | +| sharpen_strength | Controls the sharpen strength applied on the high frequency components | + ### 2d Noise Reduction | 2d_noise_reduction | Details | diff --git a/docs/algorithm-description.pdf b/docs/algorithm-description.pdf index 5c537ba..4550080 100644 Binary files a/docs/algorithm-description.pdf and b/docs/algorithm-description.pdf differ diff --git a/docs/assets/Indoor1.png b/docs/assets/Indoor1.png index b5247a9..24b5034 100644 Binary files a/docs/assets/Indoor1.png and b/docs/assets/Indoor1.png differ diff --git a/docs/assets/Outdoor1.png b/docs/assets/Outdoor1.png index f23d93a..1a397a5 100644 Binary files a/docs/assets/Outdoor1.png and b/docs/assets/Outdoor1.png differ diff --git a/docs/assets/Outdoor2.png b/docs/assets/Outdoor2.png index a0daee4..a41a237 100644 Binary files a/docs/assets/Outdoor2.png and b/docs/assets/Outdoor2.png differ diff --git a/docs/assets/Outdoor3.png b/docs/assets/Outdoor3.png index f14c3c0..3a7a4bb 100644 Binary files a/docs/assets/Outdoor3.png and b/docs/assets/Outdoor3.png differ diff --git a/docs/assets/Outdoor4.png b/docs/assets/Outdoor4.png index 7d8d624..505acd1 100644 Binary files a/docs/assets/Outdoor4.png and b/docs/assets/Outdoor4.png differ diff --git a/docs/assets/infinite-isp-architecture-initial.png b/docs/assets/infinite-isp-architecture-initial.png index 910d0ee..7e1aceb 100644 Binary files a/docs/assets/infinite-isp-architecture-initial.png and b/docs/assets/infinite-isp-architecture-initial.png differ diff --git a/in_frames/normal/data/Indoor1_2592x1536_10bit_GRBG-configs.yml b/in_frames/normal/data/Indoor1_2592x1536_10bit_GRBG-configs.yml index 0c18223..2f919b2 100644 --- a/in_frames/normal/data/Indoor1_2592x1536_10bit_GRBG-configs.yml +++ b/in_frames/normal/data/Indoor1_2592x1536_10bit_GRBG-configs.yml @@ -125,6 +125,12 @@ color_space_conversion: conv_standard: 2 is_save: false +sharpen: + is_enable: true + sharpen_sigma: 3 + sharpen_strength: 2 + is_save: false + 2d_noise_reduction: is_enable: True window_size: 5 diff --git a/in_frames/normal/data/Outdoor1_2592x1536_10bit_GRBG-configs.yml b/in_frames/normal/data/Outdoor1_2592x1536_10bit_GRBG-configs.yml index e9a3d52..9f3a25f 100644 --- a/in_frames/normal/data/Outdoor1_2592x1536_10bit_GRBG-configs.yml +++ b/in_frames/normal/data/Outdoor1_2592x1536_10bit_GRBG-configs.yml @@ -124,8 +124,14 @@ auto_exposure: color_space_conversion: conv_standard: 2 - is_save: False - + is_save: false + +sharpen: + is_enable: true + sharpen_sigma: 3 + sharpen_strength: 2 + is_save: false + 2d_noise_reduction: is_enable: True window_size: 5 diff --git a/in_frames/normal/data/Outdoor2_2592x1536_10bit_GRBG-configs.yml b/in_frames/normal/data/Outdoor2_2592x1536_10bit_GRBG-configs.yml index 6de5d75..b65cd32 100644 --- a/in_frames/normal/data/Outdoor2_2592x1536_10bit_GRBG-configs.yml +++ b/in_frames/normal/data/Outdoor2_2592x1536_10bit_GRBG-configs.yml @@ -126,6 +126,12 @@ color_space_conversion: conv_standard: 2 is_save: false +sharpen: + is_enable: true + sharpen_sigma: 3 + sharpen_strength: 2 + is_save: false + 2d_noise_reduction: is_enable: True window_size: 5 diff --git a/in_frames/normal/data/Outdoor3_2592x1536_10bit_GRBG-configs.yml b/in_frames/normal/data/Outdoor3_2592x1536_10bit_GRBG-configs.yml index b5e1b2e..985c3e2 100644 --- a/in_frames/normal/data/Outdoor3_2592x1536_10bit_GRBG-configs.yml +++ b/in_frames/normal/data/Outdoor3_2592x1536_10bit_GRBG-configs.yml @@ -127,6 +127,12 @@ color_space_conversion: conv_standard: 2 is_save: false +sharpen: + is_enable: true + sharpen_sigma: 3 + sharpen_strength: 2 + is_save: false + 2d_noise_reduction: is_enable: True window_size: 5 diff --git a/in_frames/normal/data/Outdoor4_2592x1536_10bit_GRBG-configs.yml b/in_frames/normal/data/Outdoor4_2592x1536_10bit_GRBG-configs.yml index c1571ca..a7d616b 100644 --- a/in_frames/normal/data/Outdoor4_2592x1536_10bit_GRBG-configs.yml +++ b/in_frames/normal/data/Outdoor4_2592x1536_10bit_GRBG-configs.yml @@ -126,6 +126,12 @@ color_space_conversion: conv_standard: 2 is_save: false +sharpen: + is_enable: true + sharpen_sigma: 3 + sharpen_strength: 2 + is_save: false + 2d_noise_reduction: is_enable: True window_size: 5 diff --git a/infinite_isp.py b/infinite_isp.py index 42c09fd..21b366b 100644 --- a/infinite_isp.py +++ b/infinite_isp.py @@ -24,6 +24,7 @@ from modules.color_correction_matrix import ColorCorrectionMatrix as CCM from modules.color_space_conversion import ColorSpaceConversion as CSC from modules.yuv_conv_format import YUVConvFormat as YUV_C +from modules.sharpen import Sharpening as SHARP from modules.noise_reduction_2d import NoiseReduction2d as NR2D from modules.rgb_conversion import RGBConversion as RGBC from modules.invalid_region_crop import InvalidRegionCrop as IRC @@ -75,6 +76,7 @@ def load_config(self, config_path): self.parm_gmc = c_yaml["gamma_correction"] self.parm_ae = c_yaml["auto_exposure"] self.parm_csc = c_yaml["color_space_conversion"] + self.parm_sha = c_yaml["sharpen"] self.parm_2dn = c_yaml["2d_noise_reduction"] self.parm_rgb = c_yaml["rgb_conversion"] self.parm_irc = c_yaml["invalid_region_crop"] @@ -187,9 +189,26 @@ def run_pipeline(self, visualize_output=True): csc = CSC(gamma_raw, self.platform, self.sensor_info, self.parm_csc) csc_img = csc.execute() + # ===================================================================== + # Sharpening + sharp = SHARP( + csc_img, + self.platform, + self.sensor_info, + self.parm_sha, + self.parm_csc["conv_standard"], + ) + sharp_img = sharp.execute() + # ===================================================================== # 2d noise reduction - nr2d = NR2D(csc_img, self.sensor_info, self.parm_2dn, self.platform) + nr2d = NR2D( + sharp_img, + self.sensor_info, + self.parm_2dn, + self.platform, + self.parm_csc["conv_standard"], + ) nr2d_img = nr2d.execute() # ===================================================================== @@ -201,12 +220,24 @@ def run_pipeline(self, visualize_output=True): # ===================================================================== # crop image to 1920x1080 or 1920x1440 - irc = IRC(rgbc_img, self.platform, self.sensor_info, self.parm_irc) + irc = IRC( + rgbc_img, + self.platform, + self.sensor_info, + self.parm_irc, + self.parm_csc["conv_standard"], + ) irc_img = irc.execute() # ===================================================================== # Scaling - scale = Scale(irc_img, self.platform, self.sensor_info, self.parm_sca) + scale = Scale( + irc_img, + self.platform, + self.sensor_info, + self.parm_sca, + self.parm_csc["conv_standard"], + ) scaled_img = scale.execute() # ===================================================================== diff --git a/modules/auto_exposure.py b/modules/auto_exposure.py index ada3b50..19d5427 100644 --- a/modules/auto_exposure.py +++ b/modules/auto_exposure.py @@ -48,7 +48,7 @@ def apply_window_offset_crop(self): """ Get AE Stats window by cropping the offsets """ - offsets = np.ceil(self.stats_window_offset / 4) * 4 + offsets = self.stats_window_offset top = int(offsets[0]) bottom = None if offsets[1] == 0 else -int(offsets[1]) left = int(offsets[2]) @@ -104,7 +104,7 @@ def get_luminance_histogram_skewness(self, img): Zwillinger, D. and Kokoska, S. (2000). CRC Standard Probability and Statistics Tables and Formulae. Chapman & Hall: New York. 2000. Section 2.2.24.1 """ - + img_orig = np.copy(img) # First subtract central luminance to calculate skewness around it img = img.astype(np.float64) - self.center_illuminance @@ -120,18 +120,29 @@ def get_luminance_histogram_skewness(self, img): if self.is_debug: print(" - AE - Actual_Skewness = ", skewness) - sign_m3 = np.sign(m_3) - - m_2 = m_2 >> 6 - m_3 = abs(m_3) >> 9 - - approx_sqrt_m_2 = approx_sqrt(m_2) - new_skewness, _ = get_approximate(m_3 / (m_2 * approx_sqrt_m_2), 16, 8) - new_skewness = sign_m3 * new_skewness + # all integer calc + img_int = img_orig.astype(np.int64) - self.center_illuminance + img_int_size = img_int.size + m_2_int = np.sum(np.power(img_int, 2)).astype(np.int64) + m_3_int = np.sum(np.power(img_int, 3)).astype(np.int64) + m_2_int = np.int64(m_2_int / img_int_size) + m_3_int = np.int64(m_3_int / img_int_size) + sign_m3_int = np.sign(m_3_int) + # all integer calc + + m_2_int = m_2_int >> 6 + m_3_int = abs(m_3_int) >> 9 + + approx_sqrt_m_2_int = approx_sqrt(m_2_int) + new_skewness_int = ( + np.int64((m_3_int * 256) / (m_2_int * approx_sqrt_m_2_int)) / 256 + ) + new_skewness_int = sign_m3_int * new_skewness_int if self.is_debug: - print(" - AE - Approx_Skewness = ", new_skewness) - return new_skewness + print(" - AE - Approx_Skewness Int = ", new_skewness_int) + + return new_skewness_int def execute(self): """ diff --git a/modules/auto_white_balance.py b/modules/auto_white_balance.py index 1f0fb95..d58d9fb 100644 --- a/modules/auto_white_balance.py +++ b/modules/auto_white_balance.py @@ -34,7 +34,7 @@ def apply_window_offset_crop(self): """ Get AWB Stats window by cropping the offsets """ - offsets = np.ceil(self.stats_window_offset / 4) * 4 + offsets = self.stats_window_offset top = int(offsets[0]) bottom = None if offsets[1] == 0 else -int(offsets[1]) left = int(offsets[2]) diff --git a/modules/bayer_noise_reduction.py b/modules/bayer_noise_reduction.py index 467bb05..6ed8feb 100644 --- a/modules/bayer_noise_reduction.py +++ b/modules/bayer_noise_reduction.py @@ -459,6 +459,7 @@ def save(self): "Out_bayer_noise_reduction_", self.platform, self.sensor_info["bit_depth"], + self.sensor_info["bayer_pattern"] ) def execute(self): diff --git a/modules/black_level_correction.py b/modules/black_level_correction.py index 4f189e5..ada45ff 100644 --- a/modules/black_level_correction.py +++ b/modules/black_level_correction.py @@ -134,6 +134,7 @@ def save(self): "Out_black_level_correction_", self.platform, self.sensor_info["bit_depth"], + self.sensor_info["bayer_pattern"] ) def execute(self): diff --git a/modules/color_correction_matrix.py b/modules/color_correction_matrix.py index ed13071..b2c197b 100644 --- a/modules/color_correction_matrix.py +++ b/modules/color_correction_matrix.py @@ -61,6 +61,7 @@ def save(self): "Out_color_correction_matrix_", self.platform, self.bit_depth, + self.sensor_info["bayer_pattern"] ) def execute(self): diff --git a/modules/color_space_conversion.py b/modules/color_space_conversion.py index 0538226..f0cf2fb 100644 --- a/modules/color_space_conversion.py +++ b/modules/color_space_conversion.py @@ -97,6 +97,7 @@ def save(self): self.img, "Out_color_space_conversion_", self.platform, + self.conv_std ) def execute(self): diff --git a/modules/crop.py b/modules/crop.py index f1b8b7d..d1c38d1 100644 --- a/modules/crop.py +++ b/modules/crop.py @@ -110,6 +110,7 @@ def save(self, filename_tag): filename_tag, self.platform, self.sensor_info["bit_depth"], + self.sensor_info["bayer_pattern"] ) def execute(self): diff --git a/modules/dead_pixel_correction.py b/modules/dead_pixel_correction.py index c2711a9..929e4b2 100644 --- a/modules/dead_pixel_correction.py +++ b/modules/dead_pixel_correction.py @@ -438,6 +438,7 @@ def save(self): "Out_dead_pixel_correction_", self.platform, self.bpp, + self.sensor_info["bayer_pattern"] ) def execute(self): diff --git a/modules/demosaic.py b/modules/demosaic.py index c22e3d7..785d321 100644 --- a/modules/demosaic.py +++ b/modules/demosaic.py @@ -209,6 +209,7 @@ def save(self): "Out_demosaic_", self.platform, self.sensor_info["bit_depth"], + self.sensor_info["bayer_pattern"] ) def execute(self): diff --git a/modules/digital_gain.py b/modules/digital_gain.py index ae2348c..6b5c56d 100644 --- a/modules/digital_gain.py +++ b/modules/digital_gain.py @@ -79,6 +79,7 @@ def save(self): "Out_digital_gain_", self.platform, self.sensor_info["bit_depth"], + self.sensor_info["bayer_pattern"] ) def execute(self): diff --git a/modules/gamma_correction.py b/modules/gamma_correction.py index 0fd4682..c5db87c 100644 --- a/modules/gamma_correction.py +++ b/modules/gamma_correction.py @@ -64,6 +64,7 @@ def save(self): "Out_gamma_correction_", self.platform, self.sensor_info["bit_depth"], + self.sensor_info["bayer_pattern"] ) def execute(self): diff --git a/modules/invalid_region_crop.py b/modules/invalid_region_crop.py index 3d23300..483128b 100644 --- a/modules/invalid_region_crop.py +++ b/modules/invalid_region_crop.py @@ -22,7 +22,7 @@ class InvalidRegionCrop: as the input image. """ - def __init__(self, img, platform, sensor_info, parm_irc): + def __init__(self, img, platform, sensor_info, parm_irc, conv_std): self.img = img.copy() self.enable = parm_irc["is_enable"] self.platform = platform @@ -31,6 +31,7 @@ def __init__(self, img, platform, sensor_info, parm_irc): self.parm_irc = parm_irc self.bit_depth = sensor_info["bit_depth"] self.sensor_info = sensor_info + self.conv_std = conv_std self.is_valid = False def get_idx_for_rtl(self): @@ -153,6 +154,7 @@ def save(self): "Out_invalid_region_crop_", self.platform, self.bit_depth, + self.sensor_info["bayer_pattern"] ) else: save_output_array_yuv( @@ -160,6 +162,7 @@ def save(self): self.img, "Out_invalid_region_crop_", self.platform, + self.conv_std ) def execute(self): diff --git a/modules/noise_reduction_2d.py b/modules/noise_reduction_2d.py index aa8d5c2..62d0e9e 100644 --- a/modules/noise_reduction_2d.py +++ b/modules/noise_reduction_2d.py @@ -20,13 +20,14 @@ class NoiseReduction2d: 2D Noise Reduction """ - def __init__(self, img, sensor_info, parm_2dnr, platform): + def __init__(self, img, sensor_info, parm_2dnr, platform, conv_std): self.img = img.copy() self.enable = parm_2dnr["is_enable"] self.is_save = parm_2dnr["is_save"] self.platform = platform self.sensor_info = sensor_info self.parm_2dnr = parm_2dnr + self.conv_std = conv_std self.is_progress = platform["disable_progress_bar"] self.is_leave = platform["leave_pbar_string"] self.save_lut = platform["save_lut"] @@ -161,6 +162,7 @@ def save(self): self.img, "Out_2d_noise_reduction_", self.platform, + self.conv_std ) def execute(self): diff --git a/modules/oecf.py b/modules/oecf.py index 24bbc63..c18623c 100644 --- a/modules/oecf.py +++ b/modules/oecf.py @@ -77,6 +77,7 @@ def save(self): "Out_oecf_", self.platform, self.sensor_info["bit_depth"], + self.sensor_info["bayer_pattern"] ) def execute(self): diff --git a/modules/rgb_conversion.py b/modules/rgb_conversion.py index b7a1d6a..76ebc71 100644 --- a/modules/rgb_conversion.py +++ b/modules/rgb_conversion.py @@ -85,6 +85,7 @@ def save(self): "Out_rgb_conversion_", self.platform, self.bit_depth, + self.sensor_info["bayer_pattern"] ) else: save_output_array_yuv( @@ -92,6 +93,7 @@ def save(self): self.img, "Out_rgb_conversion_", self.platform, + self.conv_std ) def execute(self): diff --git a/modules/scale.py b/modules/scale.py index 5daa81d..79c18db 100644 --- a/modules/scale.py +++ b/modules/scale.py @@ -15,12 +15,13 @@ class Scale: """Scale color image to given size.""" - def __init__(self, img, platform, sensor_info, parm_sca): + def __init__(self, img, platform, sensor_info, parm_sca, conv_std): self.img = img.copy() self.enable = parm_sca["is_enable"] self.is_save = parm_sca["is_save"] self.sensor_info = sensor_info self.platform = platform + self.conv_std = conv_std self.parm_sca = parm_sca self.get_scaling_params() @@ -81,6 +82,7 @@ def save(self): "Out_scale_", self.platform, self.sensor_info["bit_depth"], + self.sensor_info["bayer_pattern"] ) else: save_output_array_yuv( @@ -88,6 +90,7 @@ def save(self): self.img, "Out_scale_", self.platform, + self.conv_std ) def execute(self): diff --git a/modules/sharpen.py b/modules/sharpen.py new file mode 100644 index 0000000..f5b6825 --- /dev/null +++ b/modules/sharpen.py @@ -0,0 +1,111 @@ +""" +File: sharpen.py +Description: Simple unsharp masking with frequency and strength control. +Code / Paper Reference: +Author: xx-isp (ispinfinite@gmail.com) +""" + +import time +import numpy as np +from scipy import ndimage + +from util.utils import save_output_array_yuv + + +class Sharpening: + """ + Sharpening + """ + + def __init__(self, img, platform, sensor_info, parm_sha, conv_std): + self.img = img + self.enable = parm_sha["is_enable"] + self.sensor_info = sensor_info + self.parm_sha = parm_sha + self.is_save = parm_sha["is_save"] + self.platform = platform + self.conv_std = conv_std + + def gaussian_kernel(self, size_x, size_y=None, sigma_x=5, sigma_y=None): + """ + Generate a Gaussian kernel for convolutions for Sharpening Algorithm + """ + if size_y is None: + size_y = size_x + if sigma_y is None: + sigma_y = sigma_x + + assert isinstance(size_x, int) + assert isinstance(size_y, int) + + x_0 = size_x // 2 + y_0 = size_y // 2 + + x_axis = np.arange(0, size_x, dtype=float) + y_axis = np.arange(0, size_y, dtype=float)[:, np.newaxis] + + x_axis -= x_0 + y_axis -= y_0 + + exp_part = x_axis**2 / (2 * sigma_x**2) + y_axis**2 / (2 * sigma_y**2) + return 1 / (2 * np.pi * sigma_x * sigma_y) * np.exp(-exp_part) + + def apply_sharpen(self): + """Sharpens an image using the unsharp mask algorithm. + + Args: + image: A numpy array of shape (height, width) representing the image to be sharpened. + kernel_size: The size of the Gaussian kernel to use for blurring the image. + sigma: The standard deviation of the Gaussian kernel. + + Returns: + A numpy array of shape (height, width) representing the sharpened image. + """ + + sigma = self.parm_sha["sharpen_sigma"] + kernel_size = int(2 * np.ceil(2 * sigma) + 1) + + kernel = self.gaussian_kernel(kernel_size, kernel_size, sigma, sigma) + + kernel = (kernel * (2**20)).astype(np.int) + + luma = (self.img[:, :, 0]).astype(np.int) + # Filter the luma component of the image with a Gaussian LPF + # Smoothing magnitude can be controlled with the sharpen_sigma parameter + smoothened = ndimage.correlate(luma, kernel, mode="mirror") + smoothened = smoothened / (2**20) + + # Sharpen the image with upsharp mask + # Strength is tuneable with the sharpen_strength parameter + sharpened = luma + ((luma - smoothened) * self.parm_sha["sharpen_strength"]) + + self.img[:, :, 0] = np.uint8(np.clip(sharpened, 0, 255)) + return self.img + + def save(self): + """ + Function to save module output + """ + if self.is_save: + save_output_array_yuv( + self.platform["in_file"], + self.img, + "Out_Shapening_", + self.platform, + self.conv_std + ) + + def execute(self): + """ + Applying sharpening to input image + """ + print("Sharpening = " + str(self.enable)) + + if self.enable is True: + start = time.time() + s_out = self.apply_sharpen() + print(f" Execution time: {time.time() - start:.3f}s") + self.img = s_out + + self.save() + return self.img diff --git a/modules/white_balance.py b/modules/white_balance.py index 2b5fa09..1faf830 100644 --- a/modules/white_balance.py +++ b/modules/white_balance.py @@ -79,6 +79,7 @@ def save(self): "Out_white_balance_", self.platform, self.sensor_info["bit_depth"], + self.sensor_info["bayer_pattern"] ) def execute(self): diff --git a/modules/yuv_conv_format.py b/modules/yuv_conv_format.py index 72de740..52ca0fc 100644 --- a/modules/yuv_conv_format.py +++ b/modules/yuv_conv_format.py @@ -75,6 +75,7 @@ def save(self): f"Out_yuv_conversion_format_{self.param_yuv['conv_type']}_", self.platform, self.sensor_info["bit_depth"], + self.sensor_info["bayer_pattern"] ) # restore the original save format self.platform["save_format"] = save_format diff --git a/util/utils.py b/util/utils.py index dbba31d..c773e1e 100644 --- a/util/utils.py +++ b/util/utils.py @@ -269,7 +269,9 @@ def rev_yuv(arr): return out_arr -def save_output_array(img_name, output_array, module_name, platform, bitdepth): +def save_output_array( + img_name, output_array, module_name, platform, bitdepth, bayer_pattern +): """ Saves output array [raw/rgb] for pipline modules """ @@ -290,8 +292,12 @@ def save_output_array(img_name, output_array, module_name, platform, bitdepth): if platform["save_format"] == "png" or platform["save_format"] == "both": + # for 1-channel raw: convert raw image to rgb image + if len(output_array.shape) == 2: + output_array = apply_cfa(output_array, bitdepth, bayer_pattern) + # convert image to 8-bit image if required - if output_array.dtype != np.uint8 and len(output_array.shape) > 2: + if output_array.dtype != np.uint8: shift_by = bitdepth - 8 output_array = (output_array >> shift_by).astype("uint8") @@ -299,7 +305,183 @@ def save_output_array(img_name, output_array, module_name, platform, bitdepth): plt.imsave(filename + ".png", output_array) -def save_output_array_yuv(img_name, output_array, module_name, platform): +def masks_cfa_bayer(img, bayer): + """ + Generating masks for the given bayer pattern + """ + + # dict will be creating 3 channel boolean type array of given shape with the name + # tag like 'r_channel': [False False ....] , 'g_channel': [False False ....] , + # 'b_channel': [False False ....] + channels = dict( + (channel, np.zeros(img.shape, dtype=bool)) for channel in "rgb" + ) + + # Following comment will create boolean masks for each channel r_channel, + # g_channel and b_channel + for channel, (y_channel, x_channel) in zip( + bayer, [(0, 0), (0, 1), (1, 0), (1, 1)] + ): + channels[channel][y_channel::2, x_channel::2] = True + + # tuple will return 3 channel boolean pattern for r_channel, + # g_channel and b_channel with True at corresponding value + # For example in rggb pattern, the r_channel mask would then be + # [ [ True, False, True, False], [ False, False, False, False]] + return tuple(channels[c] for c in "rgb") + +def apply_cfa(img, bit_depth, bayer): + """ + Demosaicing the given raw image + """ + # 3D masks accoridng to the given bayer + mask_r, mask_g, mask_b = masks_cfa_bayer(img, bayer) + raw_in = np.float32(img) + + # Declaring 3D Demosaiced image + demos_out = np.empty((raw_in.shape[0], raw_in.shape[1], 3)) + + # 5x5 2D Filter coefficients for linear interpolation of + # r_channel,g_channel and b_channel channels + # These filters helps to retain corresponding pixels information using + # laplacian while interpolation + + # g_channel at r_channel & b_channel location, + g_at_r_and_b = ( + np.float32( + [ + [0, 0, -1, 0, 0], + [0, 0, 2, 0, 0], + [-1, 2, 4, 2, -1], + [0, 0, 2, 0, 0], + [0, 0, -1, 0, 0], + ] + ) + * 0.125 + ) + + # r_channel at green in r_channel row & b_channel column -- + # b_channel at green in b_channel row & r_channel column + r_at_gr_and_b_at_gb = ( + np.float32( + [ + [0, 0, 0.5, 0, 0], + [0, -1, 0, -1, 0], + [-1, 4, 5, 4, -1], + [0, -1, 0, -1, 0], + [0, 0, 0.5, 0, 0], + ] + ) + * 0.125 + ) + + # r_channel at green in b_channel row & r_channel column -- + # b_channel at green in r_channel row & b_channel column + r_at_gb_and_b_at_gr = np.transpose(r_at_gr_and_b_at_gb) + + # r_channel at blue in b_channel row & b_channel column -- + # b_channel at red in r_channel row & r_channel column + r_at_b_and_b_at_r = ( + np.float32( + [ + [0, 0, -1.5, 0, 0], + [0, 2, 0, 2, 0], + [-1.5, 0, 6, 0, -1.5], + [0, 2, 0, 2, 0], + [0, 0, -1.5, 0, 0], + ] + ) + * 0.125 + ) + + # Creating r_channel, g_channel & b_channel channels from raw_in + r_channel = raw_in * mask_r + g_channel = raw_in * mask_g + b_channel = raw_in * mask_b + + # Creating g_channel channel first after applying g_at_r_and_b filter + g_channel = np.where( + np.logical_or(mask_r == 1, mask_b == 1), + correlate2d(raw_in, g_at_r_and_b, mode="same", boundary="symm"), + g_channel, + ) + + # Applying other linear filters + rb_at_g_rbbr = correlate2d( + raw_in, r_at_gr_and_b_at_gb, mode="same", boundary="symm" + ) + rb_at_g_brrb = correlate2d( + raw_in, r_at_gb_and_b_at_gr, mode="same", boundary="symm" + ) + rb_at_gr_bbrr = correlate2d( + raw_in, r_at_b_and_b_at_r, mode="same", boundary="symm" + ) + + # After convolving the input raw image with rest of the filters, + # now we have the respective interpolated data, now we just have + # to extract the updated pixels values according to the + # position they are meant to be updated + + # Extracting Red rows. + r_rows = np.transpose(np.any(mask_r == 1, axis=1)[np.newaxis]) * np.ones( + r_channel.shape, dtype=np.float32 + ) + + # Extracting Red columns. + r_col = np.any(mask_r == 1, axis=0)[np.newaxis] * np.ones( + r_channel.shape, dtype=np.float32 + ) + + # Extracting Blue rows. + b_rows = np.transpose(np.any(mask_b == 1, axis=1)[np.newaxis]) * np.ones( + b_channel.shape, dtype=np.float32 + ) + + # Extracting Blue columns + b_col = np.any(mask_b == 1, axis=0)[np.newaxis] * np.ones( + b_channel.shape, dtype=np.float32 + ) + + # For R channel we have to update pixels at [r_channel rows + # and b_channel cols] & at [b_channel rows and r_channel cols] + # 3 pixels need to be updated near one given r_channel + r_channel = np.where( + np.logical_and(r_rows == 1, b_col == 1), rb_at_g_rbbr, r_channel + ) + r_channel = np.where( + np.logical_and(b_rows == 1, r_col == 1), rb_at_g_brrb, r_channel + ) + + # Similarly for B channel we have to update pixels at + # [r_channel rows and b_channel cols] + # & at [b_channel rows and r_channel cols] 3 pixels need + # to be updated near one given b_channel + b_channel = np.where( + np.logical_and(b_rows == 1, r_col == 1), rb_at_g_rbbr, b_channel + ) + b_channel = np.where( + np.logical_and(r_rows == 1, b_col == 1), rb_at_g_brrb, b_channel + ) + + # Final r_channel & b_channel channels + r_channel = np.where( + np.logical_and(b_rows == 1, b_col == 1), rb_at_gr_bbrr, r_channel + ) + b_channel = np.where( + np.logical_and(r_rows == 1, r_col == 1), rb_at_gr_bbrr, b_channel + ) + + demos_out[:, :, 0] = r_channel + demos_out[:, :, 1] = g_channel + demos_out[:, :, 2] = b_channel + + # Clipping the pixels values within the bit range + demos_out = np.clip(demos_out, 0, 2**bit_depth - 1) + demos_out = np.uint16(demos_out) + return demos_out + + +def save_output_array_yuv(img_name, output_array, module_name, platform, conv_std): """ Saves output array [yuv] for pipline modules """ @@ -325,6 +507,8 @@ def save_output_array_yuv(img_name, output_array, module_name, platform): # save image as .png if platform["save_format"] == "png" or platform["save_format"] == "both": + # cconvert the yuv image to RGB image + output_array = yuv_to_rgb(output_array, conv_std) plt.imsave(filename + ".png", output_array) @@ -362,6 +546,44 @@ def save_pipeline_output(img_name, output_img, config_file, tv_flag): ) +def yuv_to_rgb(yuv_img, conv_std): + """ + YUV-to-RGB Colorspace conversion 8bit + """ + + # make nx3 2d matrix of image + mat_2d = yuv_img.reshape((yuv_img.shape[0] * yuv_img.shape[1], 3)) + + # convert to 3xn for matrix multiplication + mat2d_t = mat_2d.transpose() + + # subract the offsets + mat2d_t = mat2d_t - np.array([[16, 128, 128]]).transpose() + + if conv_std == 1: + # for BT. 709 + yuv2rgb_mat = np.array([[74, 0, 114], [74, -13, -34], [74, 135, 0]]) + else: + # for BT.601/407 + # conversion metrix with 8bit integer co-efficients - m=8 + yuv2rgb_mat = np.array([[64, 87, 0], [64, -44, -20], [61, 0, 105]]) + + # convert to RGB + rgb_2d = np.matmul(yuv2rgb_mat, mat2d_t) + rgb_2d = rgb_2d >> 6 + + # reshape the image back + rgb2d_t = rgb_2d.transpose() + yuv_img = rgb2d_t.reshape(yuv_img.shape).astype(np.float32) + + # clip the resultant img as it can have neg rgb values for small Y' + yuv_img = np.float32(np.clip(yuv_img, 0, 255)) + + # convert the image to [0-255] + yuv_img = np.uint8(yuv_img) + return yuv_img + + def create_coeff_file(numbers, filename, weight_bits): """ Creating file for coefficients