Add support for includeFoldersAsPrefixes=true when listing directories.

[Kleptine]

Describe the issue
We have a bucket that contains ~1000 folders, each with ~10 files inside. Hierarchical namespaces is enabled.

Currently, listing this directory when mounted under gcsfuse takes ~3 seconds. Similarly, via a REST request, listing the directory takes 1.7s.

However, with the new includeFoldersAsPrefixes=true argument included in the REST request, listing drops over 10x to 189ms!

It would be great if gcsfuse would pass includeFoldersAsPrefixes when listing mounted directories.

System & Version (please complete the following information):

• OS: Any (tested on Debian/OSX)
• Platform: GCE VM
• Version: 3.4

[Tulsishah]

We are already making list request with includeFoldersAsPrefixes as true for HNS buckets. ref

[Kleptine]

Ah, thanks for letting me know. Yeah, I can see that gcsfuse reports a GCS request latency of ~75ms via cloud monitoring metrics, so it looks like that argument is being properly applied.

Unfortunately, the overall command ls > /dev/null under gcsfuse still takes 2 seconds. I suppose there is additional processing happening under the hood. Is that expected performance for this case?

I can't figure out why the REST API would return the results in 75ms, but gcsfuse and ls would take 2 seconds.

[Tulsishah]

The observed latency for gcsfuse ls operations is higher than a direct GCS API call because it includes the entire round trip: the application's request through the kernel, FUSE layer, and gcsfuse processing, in addition to the actual GCS API latency.

[raj-prince]

@Kleptine Could you please run with GCSFuse logs enabled? You can do that using --log-severity=trace & --log-file=logs.txt.

This will help us in triaging if in-efficiencies are in GCSFuse or kernel.

[Kleptine]

@raj-prince Sure, I've captured the log file. Is there a way I can send this to you in a secure way? I'm not fully confident in the data it contains to publish it to the web. I will also attempt to analyze it myself and see if I can discover anything.

[Kleptine]

Strangely, it looks like the request itself is the bulk of it. Over 1.57s, if I am reading the log correctly:

{"timestamp":{"seconds":1764572026,"nanos":272339495},"severity":"TRACE","message":"fuse_debug: Op 0x00000036        connection.go:453] <- OpenDir (inode 6, PID 972)"}
{"timestamp":{"seconds":1764572026,"nanos":272475928},"severity":"TRACE","message":"fuse_debug: Op 0x00000036        connection.go:548] -> OpenDir ()"}
{"timestamp":{"seconds":1764572026,"nanos":272759017},"severity":"TRACE","message":"fuse_debug: Op 0x00000038        connection.go:453] <- ReadDir (inode 6, PID 972)"}
{"timestamp":{"seconds":1764572026,"nanos":272885277},"severity":"TRACE","message":"gcs: Req              0x3: <- ListObjects(\"sessions/project/\")"}
{"timestamp":{"seconds":1764572027,"nanos":848770478},"severity":"TRACE","message":"gcs: Req              0x3: -> ListObjects(\"sessions/project/\") (1.575889016s): OK"}

This is very strange because here is the timing for making the same list-objects request via curl.

time curl -H "Authorization: Bearer $TOKEN"      "https://storage.googleapis.com/storage/v1/b/[bucket]/o?prefix=sessions/project/&delimiter=/&includeFoldersAsPrefixes=true"      > /dev/null
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 72163  100 72163    0     0   753k      0 --:--:-- --:--:-- --:--:--  757k

real    0m0.107s
user    0m0.060s
sys     0m0.007s

Although, this folder has 1912 folders in it, so it would need at least 2 REST requests to ListObjects for pagination, I think? But that's still much shorter.

For reference, here is the REST timing without includeFoldersAsPrefixes=true:

time curl -H "Authorization: Bearer $TOKEN"      "https://storage.googleapis.com/storage/v1/b/[bucket]/o?prefix=sessions/project/&delimiter=/"      > /dev/null
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 72163  100 72163    0     0  52482      0  0:00:01  0:00:01 --:--:-- 52482

real    0m1.388s
user    0m0.050s
sys     0m0.010s

[raj-prince]

The only secure option I see is to proceed through support.

Yes, you are looking at the right logs. This clearly tells that Network IO is taking up most of the time. Let me try running a similar experiment to see why we see this huge gap between GCSFuse and curl.

In the meantime, could you please also check if you are getting the complete content via curl, I mean folders with attributes.

[raj-prince]

Hi @Kleptine,

Able to reproduce the issue till some extent, I see curl is ~5x than list call in GCSFuse.

• My bucket contains /a/b/dir_{0...1000}/file_{0...10}.
• Tested on n2-std-96 vm
• curl 7.81.0
• GCSFuse v3.4.0
• Validate the list output in both cases (GCSFuse & curl).

Curl:

time curl -X GET -H "Authorization: Bearer $token"   "https://storage.googleapis.com/storage/v1/b/list-folder-test2/o?prefix=a/b/&delimiter=/&includeFoldersAsPrefixes=true" > /dev/null
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 19945  100 19945    0     0   232k      0 --:--:-- --:--:-- --:--:--  234k

real    0m0.102s
user    0m0.055s
sys     0m0.009s

GCSFuse:

time ls ~/gcs/a/b/ > /dev/null

real    0m0.559s
user    0m0.003s
sys     0m0.003s

GCSFuse logs:

{"timestamp":{"seconds":1764608649,"nanos":291132353},"severity":"TRACE","message":"gcs: Req              0xb: <- ListObjects(\"a/b/\")"}
{"timestamp":{"seconds":1764608649,"nanos":840735458},"severity":"TRACE","message":"gcs: Req              0xb: -> ListObjects(\"a/b/\") (549.608674ms): OK"}

Thanks for reporting this, I'll try to understand where in GCSFuse takes more time.

[raj-prince]

Just to validate if this is not a Go lang in-efficiency, try listing with small go-lang code.

package main

import (
        "fmt"
        "io"
        "net/http"
        "os"
        "time"
        "bytes"
)

func main() {
        // Retrieve the token from an environment variable
        // export OAUTH_TOKEN=$(gcloud auth print-access-token)
        token := os.Getenv("OAUTH_TOKEN")
        if token == "" {
                fmt.Println("Error: OAUTH_TOKEN environment variable is not set.")
                return
        }

        // Define the URL
        url := "https://storage.googleapis.com/storage/v1/b/list-folder-test2/o?prefix=a/b/&delimiter=/&includeFoldersAsPrefixes=true"

        // Create a new HTTP request
        req, err := http.NewRequest("GET", url, nil)
        if err != nil {
                fmt.Printf("Error creating request: %v\n", err)
                return
        }

        // Add the Authorization header
        req.Header.Add("Authorization", "Bearer "+token)

        // Create an HTTP client
        client := &http.Client{
                Timeout: 30 * time.Second, // Optional: Add a timeout
        }

        // Measure the time taken for the request
        startTime := time.Now()

        // Execute the request
        resp, err := client.Do(req)
        if err != nil {
                fmt.Printf("Error making request: %v\n", err)
                return
        }
        defer resp.Body.Close()

        // Discard the response body (equivalent to > /dev/null)
        // Read the body in a buffer to ensure it's fully consumed
        buf := new(bytes.Buffer)
        _, err = io.Copy(buf, resp.Body)
        // fmt.Printf("Response Body: %s\n", buf.String())
        if err != nil {
                fmt.Printf("Error discarding response body: %v\n", err)
                return
        }

        duration := time.Since(startTime)

        // Print the status code and duration
        fmt.Printf("Request completed with status: %s\n", resp.Status)
        fmt.Printf("Time taken: %s\n", duration)

        if resp.StatusCode != http.StatusOK {
                fmt.Printf("Warning: Received non-200 status code: %d\n", resp.StatusCode)
        }
}

Output:

Request completed with status: 200 OK
Time taken: 42.490012ms

[raj-prince]

Url parameter includeTrailingDelimiter=true is the culprit.

1. Curl one: https://storage.googleapis.com/storage/v1/b/list-folder-test2/o?prefix=a/b/&delimiter=/&includeFoldersAsPrefixes=true // takes 34ms
2. GCSFuse makes internally: https://storage.googleapis.com/storage/v1/b/list-folder-test2/o?alt=json&delimiter=%2F&endOffset=&fields=nextPageToken%2Cprefixes%2Citems%28size%2Cgeneration%2Cmetageneration%2Cupdated%2Cmetadata%2CcontentEncoding%2Ccrc32c%2Cname%29&includeFoldersAsPrefixes=true&includeTrailingDelimiter=true&matchGlob=&maxResults=5000&pageToken=&prefix=a%2Fb%2F&prettyPrint=false&projection=noAcl&startOffset=&versions=false // takes 576 ms
3. includeTrailingDelimiter=true is the culprit https://storage.googleapis.com/storage/v1/b/list-folder-test2/o?prefix=a/b/&delimiter=/&includeFoldersAsPrefixes=true&includeTrailingDelimiter=true // takes 640 ms

[Kleptine]

Ah, that makes perfect sense, and I can reproduce it locally using:

time curl -H "Authorization: Bearer $TOKEN"      "https://storage.googleapis.com/storage/v1/b/[bucket]/o?prefix=sessions/&delimiter=/&includeFoldersAsPrefixes=true&includeTrailingDelimiter=true&maxResults=5000" > /dev/null
real    0m1.218s
user    0m0.049s
sys     0m0.012s

1.2s + 600ms (gcsfuse processing time from the previous logs) is exactly the timing I'm seeing in production.

I've also reproduced the issue with a simple test bucket. You can use this script to generate a test bucket with nested files. I get nearly the exact same timings as I see in my production use case.

generate_structure.py

Let me know if you need anything else. Happy to help, as this would unblock gcsfuse for our use-case.