HyTools is a python library for processing airborne and spaceborne imaging spectroscopy data, supporting simultaneous brightness adjustment and normalization on large datasets.
At its core it consists of functions for
- Spectral resampling, topographic, BRDF (e.g. FlexBRDF) and sunglint correction, spectral transforms, masking and more;
- A series of command line tools which combine these functions and provide a streamlined workflow for processing images (tutorial);
- Utilizing Ray to speed up group image processing, and an alternative of FlexBRDF correction without Ray,
- Reading ENVI formatted images, NEON AOP HDF files, NetCDF (EMIT and AVIRIS) images with Geographic Lookup Table (GLT) support;
- Writing ENVI and NetCDF images.
For examples see the HyTools basics ipython notebook here.
To install with pip run:
pip install hy-toolsor
python -m pip install git+https://github.com/EnSpec/hytools.gitor clone
git clone https://github.com/EnSpec/hytools.gitand install with setuptools
python setup.py install| Function | Description | |
|---|---|---|
| BRDF correction |  |
Reduce variation in reflectance due to different sun-target-sensor geometry |
| Topographic correction |  |
Reduce variation in reflectance due to topographic difference |
| Sunglint correction |  |
Reduce variation in reflectance due to mirror-like specular reflection of sunlight off the water |
| Trait mapping |  |
Use linear predictive model to map various pixel-wise traits from reflectance |
| Spectral resampling |  |
Resample spectra by various methods |
| Spectral transformation |  |
Calculate various spectral variables from pixel-wise band arithmetic operations, including PCA transformation, normalized difference indices |
| Data extraction |  |
Extract spectral data from images based on scattered locations, wavelengths, columns, rows or chunks |
| Masking |  |
Image mask generation based on thresholding |
Supports of image I/O
| Function | Key name in configuration file or parameter name in function | Format Name | Description |
|---|---|---|---|
| Read | file_type | envi neon emit ncav |
ENVI NEON AOP HDF5 EMIT NetCDF AVIRIS NetCDF |
| Write | export : image_format export_type |
envi netcdf |
ENVI NetCDF |
import hytools as ht
#Create a HyTools container object
hy_obj = ht.HyTools()
#Read and load ENVI file metadata
hy_obj.read_file('./envi_file')
#Calculate NDVI, retrieves closest wavelength to input wavelength
ir = hy_obj.get_wave(900)
red = hy_obj.get_wave(660)
ndvi = (ir-red)/(ir+red)
#or
# Calculate normalized difference index, NDVI by default
ndvi = hy_obj.ndi()
#Other options for retrieving data
band = hy_obj.get_band(10)
column = hy_obj.get_column(1)
line = hy_obj.get_line(234)
chunk = hy_obj.get_chunk(0,100,0,100)
pixels = hy_obj.get_pixels([102,434],[324,345])
# Create a writer object to write to new file
writer = ht.io.WriteENVI('output_envi',hy_obj.get_header())
#Create an iterator object to cycle though image
iterator = hy_obj.iterate(by = 'line')
# Cycle line by line, read from original data
while not iterator.complete:
#Read next line
line = iterator.read_next()
#Do some calculations.......
radiance = line * gain + offset
#Write line to file
writer.write_line(radiance,iterator.current_line)
writer.close()
When running scripts, the setting of parameters are specified in configuration file in JSON format. It controls the input and output path, the ways of correction and related settings, mask exportation, etc.. This setting makes it easier to reproduce the data processing workflow.
For small amount of target images, a GUI with several default options can be used to setup the configuration file. For larger datasets with more images, or with more customized settings, the python script for generating JSON configuration file is recommended.
By running the script below with the configuration file, correction related images or coefficients will be saved. As most of the correction models considered are semi-empirical, saving the model coefficients derived from input images can reduce computing time for future correction applications.
python image_correct.py image_correct_config.jsonIf only the Correction Model Coefficients ("coeffs") is checked, no image will be exported, but the correction model coefficients will be saved and they can be reused in other workflow runs. TOPO and BRDF JSON coefficients files will be saved in the case below, which can be applied in different downstream workflows.
When the corrected images is really needed, the workflow with a different config file can generate the results. Here the new JSON can be named "image_correct_config_precomputed.json", with the imported precomputed TOPO and BRDF JSON coefficients files for each image.
python image_correct.py image_correct_config_precomputed.json
With precomputed correction coefficients and multiple trait prediction model coefficients, multiple trait maps can be generated in a parallel manner by the script below. The trait mapping configuration file can also be generated by a script trait_estimate_json_generate.py.
python trait_estimate.py map_trait_config.json
Please visit this page for more details about the image brightness adjustment workflow and the trait mapping workflow for large amount of images. For a solution for processing images in a more distributed way, please visit this page. For I/O support on imaging spectroscopy imagery with different formats and geocoding, please visit this page.
[1] Queally, N., Ye, Z., Zheng, T., Chlus, A., Schneider, F., Pavlick, R. P., & Townsend, P. A. (2022). FlexBRDF: A flexible BRDF correction for grouped processing of airborne imaging spectroscopy flightlines. Journal of Geophysical Research: Biogeosciences, 127(1), e2021JG006622. https://doi.org/10.1029/2021JG006622
[2] Greenberg, E., Thompson, D. R., Jensen, D., Townsend, P. A., Queally, N., Chlus, A., et al. (2022). An improved scheme for correcting remote spectral surface reflectance simultaneously for terrestrial BRDF and water-surface sunglint in coastal environments. Journal of Geophysical Research: Biogeosciences, 127(1), e2021JG006712. https://doi.org/10.1029/2021JG006712
HyTools is licensed under GPL-3.0 license.