UnicodeEncodeError: 'utf-8' codec can't encode characters in position 61-63: surrogates not allowed

[xzhangatnuist]

I am trying to calculate SPI using version 2.2.0. However, it gave the error like:

UnicodeEncodeError: 'utf-8' codec can't encode characters in position 61-63: surrogates not allowed

Seems that there are some errors associated with input arguments. I tried to export the environment variable but the error is still here. Any suggestions to avoid it?

Here is how i used the module:
process_climate_indices --index SPI --periodicity monthly --netcdf_temp /share/home/xzhang/drough_data/aaa.nc --var_name_temp tas --output_file_base /share/home/xzhang/drough_data/test_pet

Many thanks

[monocongo]

Hi @xzhangatnuist thanks for the report

Can you please include more of the error message details?

Do you get the same error when you try using a different input NetCDF file?

[xzhangatnuist]

Thanks very much for the reply. Yes, I get the same error with other NETCDF files too. I have attached a picture of the error message. I tried to set export LANG=EN_us.utf-8
export LC_ALL = EN_us.utf-8
export PYTHONIOENCODING=utf-8
export PYTHONUTF8=1

But the error persists.

Best regards,

Xiao

[xzhangatnuist]

my apologizes, I have downloaded data from 6 models and two of them encountered the problem above. I tried all models today and 4 others do not have this problem. But I am not sure how to handle it. Should I change NETCDF encoding ?

Best regards,

Xiao

[monocongo]

This is a strong indication that the issue is likely not related to the core calculation logic of the climate_indices package, but rather an interaction between the I/O libraries (xarray, netCDF4), the input file's metadata, or the system's encoding/locale settings.

The error, UnicodeEncodeError: 'utf-8' codec can't encode characters...: surrogates not allowed, occurs when a string containing invalid Unicode characters (often unpaired surrogates) is passed to an operation that requires strict UTF-8 encoding, typically during file reading, writing, or logging.

---

1. Likely Causes of the Encoding Error

The most likely causes are tied to how supporting libraries read or write metadata in the NetCDF file:

External Library Dependency Bugs

The climate_indices package relies on powerful libraries like xarray and the underlying netCDF4 library to handle all file reading and writing.

• A bug in a specific version of netCDF4 or xarray might cause them to incorrectly parse a byte sequence in the file's metadata (e.g., an attribute like title, history, or a coordinate description) into an invalid Python string (containing unpaired surrogates), which then fails when Python tries to encode it back to UTF-8 for an internal operation.
• Solution: Check the dependency versions. Often, updating xarray and netCDF4 (or the compiled C library, libnetcdf) to their latest stable versions resolves these low-level I/O issues.

Malformed NetCDF Metadata

The input file (/share/home/xzhang/drough_data/aaa.nc) might contain attributes or values that were originally written using an encoding that is not compatible with UTF-8, or that contain characters that are simply corrupt.

• When the library tries to read this malformed metadata into a standard Python string, it can create the invalid surrogate characters that trigger the error.

Locale and Environment Variables

The operating system's locale settings dictate the default encoding Python uses for handling text (especially I/O).

• If the system locale (like LANG or LC_ALL) is set to an older standard (e.g., C or en_US) instead of a modern UTF-8 locale (e.g., en_US.UTF-8), Python might default to an encoding that mishandles the characters present in the NetCDF file, leading to the error. The user trying to export environment variables suggests they suspect this.

---

2. Determining the Cause (Isolation Tests)

The most effective way to determine if the issue is external is to isolate the file I/O from the climate_indices calculation logic.

Step 1: Test Reading the File with xarray

If the error occurs just by reading the file, the problem is with the file's content or the xarray/netCDF4 dependency.

1. Open the file in a minimal Python session (ideally in the same environment where the failure occurred).

   import xarray as xr
   # This single line attempts to read all file metadata and most data structure
   ds = xr.open_dataset("/share/home/xzhang/drough_data/aaa.nc") 
   print(ds)
   # If the error occurs here, the problem is isolated to xarray/netCDF4/the file.

2. If the above fails, try accessing an attribute:

   # After the open_dataset line, try to read file attributes
   print(ds.attrs)
   # The error might be triggered by a specific, problematic global attribute.

Step 2: Test the Environment (The "Fresh Install" Check)

Create a completely new, clean environment to rule out conflicting packages.

1. Create a new Conda or virtual environment.
2. Install only the core dependencies needed for the I/O: xarray and netcdf4 (and perhaps dask).
3. Re-run the xr.open_dataset() test from Step 1.

If the error persists in the clean environment, it strongly confirms that the issue is either with the input file's metadata or a bug in the specific versions of xarray/netCDF4 you are using. If the error disappears, the previous environment had a conflict or corruption.

Step 3: Test with a Known Good File

1. Download a standard, validated NetCDF file from a trusted source (e.g., a sample dataset from NOAA or a public repository).
2. Run the climate_indices script against this known good file.

• If the script runs successfully with the known good file, the problem is specifically the content/encoding of your input file (aaa.nc).
• If the script fails with the known good file, the problem is likely a fundamental environment setup issue (like the system locale) that is affecting all I/O operations.