8 improve save functionality for isp modules

README.md

@@ -31,9 +31,9 @@ ISP pipeline for `InfiniteISP_ReferenceModel v1.0`
 
 3. **Dataset Processing**: The model facilitates execution for multiple images with different or same configuration files.
 
-4. **Video Processing**: The model also features a video processing script that allows for sequential frame processing, with operational 3A Statistics data flowing between frames.
-
+4. **Video Processing**: The model also includes a video processing feature that allows for sequential frame processing, with operational 3A Statistics data flowing between frames.
 
+5. **Support Multiple RAW formats**: The model supports renowned raw formats, for example, DNG (predominantly employed by Android devices), NEF (specific to Nikon devices), and CR2 (Canon's signature format). It can also process the .raw  image format, a one-dimensional pixel data array with no metadata header.
 
 ## Objectives
 
@@ -59,6 +59,7 @@ The table below provides a feature list of the model. The version `1.0` of the m
 | Gamma Correction             |Implements a LUT from config |
 | Auto Exposure                                 | [Auto Exposure](https://www.atlantis-press.com/article/25875811.pdf) <br> - AE stats calculations based on skewness |
 | Color Space Conversion                        | YCbCr digital <br> - BT 601 <br> - Bt 709  <br>   |YCbCr digital <br> - BT 601 <br> - Bt 709  <br> |
+| Edge Enhancement / Sharpening                | Simple unsharp masking with strength control| 
 | Noise Reduction                               | [Non-local means filter](https://www.ipol.im/pub/art/2011/bcm_nlm/article.pdf) <br> - Implements intensity level difference  through a LUT|
 | RGB Conversion                               | Converts YCbCr digital image to RGB|
 | Invalid Region Crop    | Crops image to a fixed size|
@@ -103,19 +104,20 @@ RAW_DATA = './in_frames/normal/data'
 
 ## How to Run on Pipeline on Multiple Images/Dataset
 
-There are two scripts that run Infinite-ISP on multiple images:
+There is another script [isp_pipeline_multiple_images.py](isp_pipeline_multiple_images.py) that runs Infinite-ISP on multiple images with two modes:
 
 
-1. [isp_pipeline_dataset.py](isp_pipeline_dataset.py) 
+1. DATASET PROCESSING
     <br >Execute multiple images. Raw image should have its own config file with name `<filename>-configs.yml` where `<filename>` is raw filename otherwise the default configuration file [configs.yml](config/configs.yml) is used.
 
+    For raw image format such as, NEF, DNG and CR2 we have also provided a funcationality to extract sensor information provided in these raw files metadata and update default config file.
 
-2. [video_processing.py](video_processing.py) 
-   <br> Each image in the dataset is considered as video frame in sequence. All images use the same configuration parameters from [configs.yml](config/configs.yml) and 3A Stats calculated on a frame are applied to the next frame.
+2. VIDEO MODE
+   <br> Each image in the dataset is considered as video frame in sequence. All images use the same configuration parameters from [configs.yml](config/configs.yml) and 3A stats calculated on a frame are applied to the next frame.
 
 After cloning the repository and installing all the dependencies follow the following steps:
 
-1.  Set `DATASET_PATH` to dataset folder. For example if images are in in [in_frames/normal/data](in_frames/normal/data) folder
+1.  Set `DATASET_PATH` to dataset folder. For example if images are in [in_frames/normal/data](in_frames/normal/data) folder
 ```python
 DATASET_PATH = './in_frames/normal/data'
 ```
@@ -124,17 +126,17 @@ DATASET_PATH = './in_frames/normal/data'
 
 ```shell
 git submodule add <url> <path>
-git submodule update –-init –-recursive
+git submodule update --init --recursive
 ``` 
 
 
-4. After adding git repository as a submodule update `DATASET_PATH` variable in [isp_pipeline_dataset.py](isp_pipeline_dataset.py) to `./in_frames/normal/<dataset_name>`. Git does not allow to import a repository’s subfolder using a submodule. You can only add an entire repository and then access the folder. If you want to use images from a subfolder of a submodule modify the `DATASET_PATH` variable in [isp_pipeline_dataset.py](isp_pipeline_dataset.py)  or [video_processing.py](video_processing.py)  accordingly.
+4. After adding git repository as a submodule update `DATASET_PATH` variable in [isp_pipeline_multiple_images.py ](isp_pipeline_multiple_images.py ) to `./in_frames/normal/<dataset_name>`. Git does not allow to import a repository’s subfolder using a submodule. You can only add an entire repository and then access the folder. If you want to use images from a subfolder of a submodule modify the `DATASET_PATH` variable in [isp_pipeline_multiple_images.py](isp_pipeline_multiple_images.py)  accordingly.
 
 ```python
 DATASET_PATH = './in_frames/normal/<dataset_name>'
 ```
 
-5. Run `isp_pipeline_dataset.py` or `video_processing.py`
+5. Run `isp_pipeline_multiple_images.py`
 6. The processed images are saved in [out_frames](out_frames/) folder.
 
 ## Test Vector Generation

config/configs.yml

@@ -124,6 +124,12 @@ color_space_conversion:
   conv_standard: 2
   is_save: false
 
+sharpen:
+  is_enable: true
+  sharpen_sigma: 5
+  sharpen_strength: 1
+  is_save: false
+
 2d_noise_reduction:
   is_enable: false
   window_size: 9

docs/UserGuide.md

@@ -153,7 +153,15 @@ Below parameters are present each ISP pipeline module they effect the functional
 | color_space_conversion | Details                                                                             |  
 |------------------------|------------------------------------------------------------------------------------                                |   
 | conv_standard          | The standard to be used for conversion <br> - `1` : Bt.709 HD <br> - `2` : Bt.601/407 |   
-
+
+### Edge Enchancement / Sharpening 
+
+| Sharpening         | Details                                           | 
+|--------------------|---------------------------------------------------|
+| is_enable           | When enabled applies the sharpening | 
+| sharpen_sigma      | Define the Standard Deviation of the Gaussian Filter |   
+| sharpen_strength   | Controls the sharpen strength applied on the high frequency components  |  
+
 ### 2d Noise Reduction
 
 | 2d_noise_reduction | Details                                           | 

in_frames/normal/data/Indoor1_2592x1536_10bit_GRBG-configs.yml

@@ -125,6 +125,12 @@ color_space_conversion:
   conv_standard: 2
   is_save: false
 
+sharpen:
+  is_enable: true
+  sharpen_sigma: 3
+  sharpen_strength: 2
+  is_save: false
+
 2d_noise_reduction:
   is_enable: True
   window_size: 5

in_frames/normal/data/Outdoor1_2592x1536_10bit_GRBG-configs.yml

@@ -124,8 +124,14 @@ auto_exposure:
 
 color_space_conversion:
   conv_standard: 2
-  is_save: False
-
+  is_save: false
+
+sharpen:
+  is_enable: true
+  sharpen_sigma: 3
+  sharpen_strength: 2
+  is_save: false
+
 2d_noise_reduction:
   is_enable: True
   window_size: 5

in_frames/normal/data/Outdoor2_2592x1536_10bit_GRBG-configs.yml

@@ -126,6 +126,12 @@ color_space_conversion:
   conv_standard: 2
   is_save: false
 
+sharpen:
+  is_enable: true
+  sharpen_sigma: 3
+  sharpen_strength: 2
+  is_save: false
+
 2d_noise_reduction:
   is_enable: True
   window_size: 5

in_frames/normal/data/Outdoor3_2592x1536_10bit_GRBG-configs.yml

@@ -127,6 +127,12 @@ color_space_conversion:
   conv_standard: 2
   is_save: false
 
+sharpen:
+  is_enable: true
+  sharpen_sigma: 3
+  sharpen_strength: 2
+  is_save: false
+
 2d_noise_reduction:
   is_enable: True
   window_size: 5

in_frames/normal/data/Outdoor4_2592x1536_10bit_GRBG-configs.yml

@@ -126,6 +126,12 @@ color_space_conversion:
   conv_standard: 2
   is_save: false
 
+sharpen:
+  is_enable: true
+  sharpen_sigma: 3
+  sharpen_strength: 2
+  is_save: false
+
 2d_noise_reduction:
   is_enable: True
   window_size: 5

infinite_isp.py

@@ -24,6 +24,7 @@
 from modules.color_correction_matrix import ColorCorrectionMatrix as CCM
 from modules.color_space_conversion import ColorSpaceConversion as CSC
 from modules.yuv_conv_format import YUVConvFormat as YUV_C
+from modules.sharpen import Sharpening as SHARP
 from modules.noise_reduction_2d import NoiseReduction2d as NR2D
 from modules.rgb_conversion import RGBConversion as RGBC
 from modules.invalid_region_crop import InvalidRegionCrop as IRC
@@ -75,6 +76,7 @@ def load_config(self, config_path):
         self.parm_gmc = c_yaml["gamma_correction"]
         self.parm_ae = c_yaml["auto_exposure"]
         self.parm_csc = c_yaml["color_space_conversion"]
+        self.parm_sha = c_yaml["sharpen"]
         self.parm_2dn = c_yaml["2d_noise_reduction"]
         self.parm_rgb = c_yaml["rgb_conversion"]
         self.parm_irc = c_yaml["invalid_region_crop"]
@@ -187,9 +189,26 @@ def run_pipeline(self, visualize_output=True):
         csc = CSC(gamma_raw, self.platform, self.sensor_info, self.parm_csc)
         csc_img = csc.execute()
 
+        # =====================================================================
+        # Sharpening
+        sharp = SHARP(
+            csc_img,
+            self.platform,
+            self.sensor_info,
+            self.parm_sha,
+            self.parm_csc["conv_standard"],
+        )
+        sharp_img = sharp.execute()
+
         # =====================================================================
         # 2d noise reduction
-        nr2d = NR2D(csc_img, self.sensor_info, self.parm_2dn, self.platform)
+        nr2d = NR2D(
+            sharp_img,
+            self.sensor_info,
+            self.parm_2dn,
+            self.platform,
+            self.parm_csc["conv_standard"],
+        )
         nr2d_img = nr2d.execute()
 
         # =====================================================================
@@ -201,12 +220,24 @@ def run_pipeline(self, visualize_output=True):
 
         # =====================================================================
         # crop image to 1920x1080 or 1920x1440
-        irc = IRC(rgbc_img, self.platform, self.sensor_info, self.parm_irc)
+        irc = IRC(
+            rgbc_img,
+            self.platform,
+            self.sensor_info,
+            self.parm_irc,
+            self.parm_csc["conv_standard"],
+        )
         irc_img = irc.execute()
 
         # =====================================================================
         # Scaling
-        scale = Scale(irc_img, self.platform, self.sensor_info, self.parm_sca)
+        scale = Scale(
+            irc_img,
+            self.platform,
+            self.sensor_info,
+            self.parm_sca,
+            self.parm_csc["conv_standard"],
+        )
         scaled_img = scale.execute()
 
         # =====================================================================

modules/auto_exposure.py

@@ -48,7 +48,7 @@ def apply_window_offset_crop(self):
         """
         Get AE Stats window by cropping the offsets
         """
-        offsets = np.ceil(self.stats_window_offset / 4) * 4
+        offsets = self.stats_window_offset
         top = int(offsets[0])
         bottom = None if offsets[1] == 0 else -int(offsets[1])
         left = int(offsets[2])
@@ -104,7 +104,7 @@ def get_luminance_histogram_skewness(self, img):
         Zwillinger, D. and Kokoska, S. (2000). CRC Standard Probability and Statistics
         Tables and Formulae. Chapman & Hall: New York. 2000. Section 2.2.24.1
         """
-
+        img_orig = np.copy(img)
         # First subtract central luminance to calculate skewness around it
         img = img.astype(np.float64) - self.center_illuminance
 
@@ -120,18 +120,29 @@ def get_luminance_histogram_skewness(self, img):
         if self.is_debug:
             print("   - AE - Actual_Skewness = ", skewness)
 
-        sign_m3 = np.sign(m_3)
-
-        m_2 = m_2 >> 6
-        m_3 = abs(m_3) >> 9
-
-        approx_sqrt_m_2 = approx_sqrt(m_2)
-        new_skewness, _ = get_approximate(m_3 / (m_2 * approx_sqrt_m_2), 16, 8)
-        new_skewness = sign_m3 * new_skewness
+        # all integer calc
+        img_int = img_orig.astype(np.int64) - self.center_illuminance
+        img_int_size = img_int.size
+        m_2_int = np.sum(np.power(img_int, 2)).astype(np.int64)
+        m_3_int = np.sum(np.power(img_int, 3)).astype(np.int64)
+        m_2_int = np.int64(m_2_int / img_int_size)
+        m_3_int = np.int64(m_3_int / img_int_size)
+        sign_m3_int = np.sign(m_3_int)
+        # all integer calc
+
+        m_2_int = m_2_int >> 6
+        m_3_int = abs(m_3_int) >> 9
+
+        approx_sqrt_m_2_int = approx_sqrt(m_2_int)
+        new_skewness_int = (
+            np.int64((m_3_int * 256) / (m_2_int * approx_sqrt_m_2_int)) / 256
+        )
+        new_skewness_int = sign_m3_int * new_skewness_int
         if self.is_debug:
-            print("   - AE - Approx_Skewness = ", new_skewness)
 
-        return new_skewness
+            print("   - AE - Approx_Skewness Int = ", new_skewness_int)
+
+        return new_skewness_int
 
     def execute(self):
         """

modules/auto_white_balance.py

@@ -34,7 +34,7 @@ def apply_window_offset_crop(self):
         """
         Get AWB Stats window by cropping the offsets
         """
-        offsets = np.ceil(self.stats_window_offset / 4) * 4
+        offsets = self.stats_window_offset
         top = int(offsets[0])
         bottom = None if offsets[1] == 0 else -int(offsets[1])
         left = int(offsets[2])

modules/bayer_noise_reduction.py

@@ -459,6 +459,7 @@ def save(self):
                 "Out_bayer_noise_reduction_",
                 self.platform,
                 self.sensor_info["bit_depth"],
+                self.sensor_info["bayer_pattern"]
             )
 
     def execute(self):

modules/black_level_correction.py

@@ -134,6 +134,7 @@ def save(self):
                 "Out_black_level_correction_",
                 self.platform,
                 self.sensor_info["bit_depth"],
+                self.sensor_info["bayer_pattern"]
             )
 
     def execute(self):

modules/color_correction_matrix.py

@@ -61,6 +61,7 @@ def save(self):
                 "Out_color_correction_matrix_",
                 self.platform,
                 self.bit_depth,
+                self.sensor_info["bayer_pattern"]
             )
 
     def execute(self):

modules/color_space_conversion.py

@@ -97,6 +97,7 @@ def save(self):
                 self.img,
                 "Out_color_space_conversion_",
                 self.platform,
+                self.conv_std
             )
 
     def execute(self):

modules/crop.py

@@ -110,6 +110,7 @@ def save(self, filename_tag):
                 filename_tag,
                 self.platform,
                 self.sensor_info["bit_depth"],
+                self.sensor_info["bayer_pattern"]
             )
 
     def execute(self):

modules/dead_pixel_correction.py

@@ -438,6 +438,7 @@ def save(self):
                 "Out_dead_pixel_correction_",
                 self.platform,
                 self.bpp,
+                self.sensor_info["bayer_pattern"]
             )
 
     def execute(self):

modules/demosaic.py

@@ -209,6 +209,7 @@ def save(self):
                 "Out_demosaic_",
                 self.platform,
                 self.sensor_info["bit_depth"],
+                self.sensor_info["bayer_pattern"]
             )
 
     def execute(self):

modules/digital_gain.py

@@ -79,6 +79,7 @@ def save(self):
                 "Out_digital_gain_",
                 self.platform,
                 self.sensor_info["bit_depth"],
+                self.sensor_info["bayer_pattern"]
             )
 
     def execute(self):

modules/gamma_correction.py

@@ -64,6 +64,7 @@ def save(self):
                 "Out_gamma_correction_",
                 self.platform,
                 self.sensor_info["bit_depth"],
+                self.sensor_info["bayer_pattern"]
             )
 
     def execute(self):