GDAL incorrectly says "file does not exist" when ran concurrently

[C-Loftus]

I have a simple program which runs many concurrent read only operations with duckdb using the go client

However every once in a while I get this issue:

IO Error: GDAL Error (4): /Users/cloftus/github/project/my_file.fgb: No such file or directory

I have tried on both local and remote in gcs and both have the same issue (feel free to test against that file if desired; it is public open data)

IO Error: GDAL Error (4): Failed to open file gcs://national-hydrologic-geospatial-fabric-reference-hydrofabric/reference_catchments_and_flowlines.fgb,: {"exception_type":"HTTP","exception_message":"Unable to connect to URL \"https://storage.googleapis.com/national-hydrologic-geospatial-fabric-reference-hydrofabric/reference_catchments_and_flowlines.fgb%2C\": 404 (Not Found).","header_Access-Control-Allow-Origin":"*","header_Date":"Tue, 16 Dec 2025 14:27:44 GMT","header_Content-Type":"application/xml; charset=UTF-8","header_Alt-Svc":"h3=\":443\"; ma=2592000,h3-29=\":443\"; ma=2592000","header_Expires":"Tue, 16 Dec 2025 14:27:44 GMT","header_X-GUploader-UploadID":"AHVrFxMHB0OjZv81bDHRqWvbKAHCvOh1aXMUa49-4hyz0KxhY3cfLLrOp-NrgY3LaAqZiVREKl8xfz4","header_Cache-Control":"private, max-age=0","response_body":"","reason":"Not Found","header_Access-Control-Expose-Headers":"Content-Type","header_Server":"UploadServer","header_Content-Length":"261","status_code":"404"}

LINE 3:                 FROM ST_Read(
             ^ 
exit status 1

I am confused why I am getting this error since:

• All my operations are read only in the same process
• I have checks before the SQL that ensures the file exists
• I don't update the file or change the connection concurrently
• My path is absolute and I don't change my current working directory at all
• It only happens around 30% of the time and has no clear pattern

Do I need to add a mutex on this operation even though it is read only?

---

Code Context

If it is relevant, this is the entire go code which runs the SQL which causes the error. I have no other duckdb interactions in my codebase at the moment

Sorry if it is verbose; the actual SQL for duckdb is quite brief.

// This function I only open once in my program
func NewS3FlatgeobufMainstemService(mainstemFlatgeobufURI string) (S3FlatgeobufMainstemService, error) {
	db, err := sql.Open("duckdb", "")
	if err != nil {
		return S3FlatgeobufMainstemService{}, err
	}
	_, err = db.Exec("INSTALL spatial; LOAD spatial;")
	if err != nil {
		return S3FlatgeobufMainstemService{}, err
	}
	return S3FlatgeobufMainstemService{duckdb: db, mainstemFlatgeobufURI: mainstemFlatgeobufURI}, nil
}

// This I run from multiple green threads (goroutines)
func (s S3FlatgeobufMainstemService) GetMainstemForWkt(wkt string) (MainstemQueryResponse, error) {
	// We first query the centroid of the geometry
	// so that we are guaranteed to only get one
	// catchment and thus only one maistem; otherwise
	// there could be multiple overlapping and thus
	// ambiguity
	centroidQuery := `
    SELECT
        ST_X(ST_Centroid(g)) AS center_x,
        ST_Y(ST_Centroid(g)) AS center_y
    FROM (
        SELECT ST_GeomFromText(CAST(? AS VARCHAR)) AS g
    )
	`
	row := s.duckdb.QueryRow(centroidQuery, wkt)
	if row.Err() != nil {
		return MainstemQueryResponse{}, row.Err()
	}
	var center_x, center_y float64
	if err := row.Scan(&center_x, &center_y); err != nil {
		return MainstemQueryResponse{}, fmt.Errorf("centroid query failed: %w", err)
	}

	// flatgeobuf requires opening with a bbox in duckdb
	// in order to subset the data; by using the same
	// value for min and max we get a specific point
	// and a guarantee of no overlaps
	mainstemSQL := `
    SELECT geoconnex_url
		FROM ST_Read(
			?,
			spatial_filter_box = ST_MakeBox2D(
				ST_Point(?, ?),
				ST_Point(?, ?)
			)
		)
	`
	result := s.duckdb.QueryRow(mainstemSQL, s.mainstemFlatgeobufURI, center_x, center_y, center_x, center_y)
	if result.Err() != nil {
		return MainstemQueryResponse{}, fmt.Errorf("mainstem query failed: %w", result.Err())
	}
	var mainstemURI sql.NullString
	if err := result.Scan(&mainstemURI); err != nil {
		return MainstemQueryResponse{}, err
	}

	/// ... rest omitted for brevity
}

[Maxxen]

Hello!

Its hard to tell what is going wrong here, but at a glance it seems like you're trying to read from:
https://storage.googleapis.com/national-hydrologic-geospatial-fabric-reference-hydrofabric/reference_catchments_and_flowlines.fgb%2C, note the extra %2C at the end, which looks like a url-encoded comma (,).

Are you sure there's nothing funky happening in the s.mainstemFlatgeobufURI?

[C-Loftus]

Thank you very much for your response!

I apologize, my previous message is unclear; the url that I am requesting doesn't have the extra comma, it just appears that way in how my golang program formatted the error.

I have verified that there is nothing weird occurring with the url, and the debugger shows gcs://national-hydrologic-geospatial-fabric-reference-hydrofabric/reference_catchments_and_flowlines.fgb as expected.

Since I posted the original issue, I've ensured my program:

• never mutates anything related to the flatgeobuf operations
• works fine as long as I put a mutex over any duckdb spatial interactions (leading me to think there is some race condition in the internal gdal bindings or gdal itself?)
• sometimes fails 50% of the time and sometimes succeeds fine (so there isn't anything syntactically wrong)
• also can fail with a local file
  • GDAL Error (4): /Users/cloftus/github/test/reference_catchments_and_flowlines.fgb: No such file or directory even though that file definitely exists on disk for me

---

I have created a minimal golang reproduction here: https://gist.github.com/C-Loftus/cbd81d4d38b93f2f3ac58ced6b0f19f7

This example has the same behavior where it can sometimes succeed and sometimes give the GDAL error saying the file doesn't exist.