Skip to content

Intermittent wis2box Data Ingestion Failure (CSV → BUFR4 → GeoJSON) #1166

Open

Description

@robeleq

Data ingested into the MinIO incoming bucket occasionally fails to publish to WIS2. While the pipeline operates correctly for most files, seemingly correct files fail during conversion, blocking publication. Failures originate at both the CSV-to-BUFR4 (csv2bufr) conversion stage and the subsequent BUFR4-to-GeoJSON generation stage.

We have verified that the .CSV data in MinIO is correct by running the wis2box-csv2bufr API process.

Observed Error Categories

  • .csv failed to convert to BUFR4
  • .bufr4 failed to convert to GeoJSON
  • No features generated
  • Incorrect padding

Relevant Logs

[2026-10-19T08:02:51Z] {/app/wis2box/data/csv2bufr.py:85} WARNING - input=WIGOS_0-231-0-63478_20261019T070000.csv warning=#1#airTemperature: Value (-725.85) out of valid range (193.15 - 333.15).; Element set to missing
[2026-10-19T08:02:51Z] {/app/wis2box/data/csv2bufr.py:85} WARNING - input=WIGOS_0-231-0-63478_20261019T070000.csv warning=#1#relativeHumidity: Value (-999.0) out of valid range (0 - 100).; Element set to missing
[2026-10-19T08:02:52Z] {/app/wis2box/data/csv2bufr.py:87} ERROR - file=WIGOS_0-231-0-63453_20261019T070000.csv failed to convert to BUFR4, result={'error': 'Incorrect paddingNo features generated', 'items': []}
[2026-10-19T08:02:52Z] {/app/wis2box/data/csv2bufr.py:87} ERROR - file=WIGOS_0-231-0-63471_20261019T070000.csv failed to convert to BUFR4, result={'error': 'No features generated', 'items': []}
wis2box.cron is alive
wis2box.cron is alive
wis2box.cron is alive
wis2box.cron is alive
wis2box.cron is alive
wis2box.cron is alive
[2026-10-19T09:02:51Z] {/app/wis2box/data/csv2bufr.py:85} WARNING - input=WIGOS_0-231-0-63478_20261019T080000.csv warning=#1#airTemperature: Value (-725.85) out of valid range (193.15 - 333.15).; Element set to missing
[2026-10-19T09:02:51Z] {/app/wis2box/data/csv2bufr.py:85} WARNING - input=WIGOS_0-231-0-63478_20261019T080000.csv warning=#1#relativeHumidity: Value (-999.0) out of valid range (0 - 100).; Element set to missing
wis2box.cron is alive
wis2box.cron is alive
wis2box.cron is alive
wis2box.cron is alive
wis2box.cron is alive
wis2box.cron is alive
[2026-10-19T10:02:52Z] {/app/wis2box/data/csv2bufr.py:85} WARNING - input=WIGOS_0-231-0-63478_20261019T090000.csv warning=#1#airTemperature: Value (-725.85) out of valid range (193.15 - 333.15).; Element set to missing
[2026-10-19T10:02:52Z] {/app/wis2box/data/csv2bufr.py:85} WARNING - input=WIGOS_0-231-0-63478_20261019T090000.csv warning=#1#relativeHumidity: Value (-999.0) out of valid range (0 - 100).; Element set to missing
[2026-10-19T10:02:52Z] {/app/wis2box/data/bufr2geojson.py:75} ERROR - file=WIGOS_0-231-0-63478_20261019T090000.bufr4 failed to convert to GeoJSON
wis2box.cron is alive
wis2box.cron is alive
wis2box.cron is alive
wis2box.cron is alive
wis2box.cron is alive
wis2box.cron is alive
[2026-10-19T11:02:52Z] {/app/wis2box/data/csv2bufr.py:85} WARNING - input=WIGOS_0-231-0-63478_20261019T100000.csv warning=#1#airTemperature: Value (-725.85) out of valid range (193.15 - 333.15).; Element set to missing
[2026-10-19T11:02:52Z] {/app/wis2box/data/csv2bufr.py:85} WARNING - input=WIGOS_0-231-0-63478_20261019T100000.csv warning=#1#relativeHumidity: Value (-999.0) out of valid range (0 - 100).; Element set to missing
[2026-10-19T11:02:53Z] {/app/wis2box/data/csv2bufr.py:87} ERROR - file=WIGOS_0-231-0-63475_20261019T100000.csv failed to convert to BUFR4, result={'error': 'No features generated', 'items': []}
[2026-10-19T11:02:56Z] {/app/wis2box/data/csv2bufr.py:87} ERROR - file=WIGOS_0-231-0-63474_20261019T100000.csv failed to convert to BUFR4, result={'error': 'No features generated', 'items': []}
[2026-10-19T11:02:57Z] {/app/wis2box/data/bufr2geojson.py:75} ERROR - file=WIGOS_0-231-0-63338_20261019T100000.bufr4 failed to convert to GeoJSON

Activity

  1. added this to the sprint-021 milestone on Oct 19, 2026
  2. maaikelimper commented on Oct 19, 2026

    @maaikelimper
    Member

    @alhelguera can you have a look a this ?

  3. alhelguera commented on Oct 19, 2026

    @alhelguera
    Member

    @robeleq can you provide some sample files that produce the issue?

  4. robeleq commented on Oct 22, 2026

    @robeleq
    Author

    @alhelguera Here are some of the .CSV files. I have also tried to reproduce the issue with the files, but all tests passed (CSV->BUFF and BUFF -> GEOJSON) using the wis2box-csv2bufr and bufr2geojson API processes

    WIGOS_0-231-0-63471_20261019T070000.csv
    WIGOS_0-231-0-63502_20261012T110000.csv
    WIGOS_0-231-0-63502_20261012T070000.csv
    WIGOS_0-231-0-63502_20261012T060000.csv

  5. alhelguera commented on Oct 23, 2026

    @alhelguera
    Member

    @tomkralidis found that the requests from csv2bufr are showing results from bufr2geojson and viceversa.

    This led me to think there could be a concurrency issue somewhere in the stack with a race condition enabling these exchanges. Traced to a plausible race condition which was introduced in pygeoapi commit geopython/pygeoapi@3a63f5b0cc

    Image

    In short, changing the processor to be a class level variable enabled the sharing, and subsequent exchanging of it between different requests

  6. robeleq commented on Oct 23, 2026

    @robeleq
    Author

    @tomkralidis I agree with your analysis, and it's likely that concurrent requests can overwrite the selected processor.

    self.processor = self.get_processor(process_id)

    Later, after adding the job, it passes self.processor to the execution handler. Another request could change that value in between. If request A and request B overlap, request B can replace self.processor while request A is still executing.

    The appropriate code correction should be to use a local variable throughout execute_process: and replace the remaining self.processor references inside that method with processor.

    job_id = str(uuid.uuid1())
    
    processor = self.get_processor(process_id)
    processor.set_job_id(job_id)
    
    ...
    
    mime_type, outputs, status = handler(
        processor,
        job_id,
        data_dict,
        requested_outputs,
        **extra_execute_handler_parameters
    )
    
  7. tomkralidis commented on Nov 6, 2026

    @tomkralidis
    Member

    FYI fixed in pygeoapi (geopython/pygeoapi#2479)

  8. robeleq commented on Nov 11, 2026

    @robeleq
    Author

    @tomkralidis Thanks! When can we expect this pygeoapi fix to be included in an official wis2box API release/image so we can upgrade our production node?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Type

No type

Projects

No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions