v4.25.1 makes large S3 containers unloadable

[galengidman]

Bug description

The v4.25.1 update ensures that empty folders are included in the asset browser by building the directory list from the database and then merging that with $container->disk()->getFolders():

eloquent-driver/src/Assets/AssetContainerContents.php

Lines 50 to 55 in b446044

	->merge(
	collect($this->container->disk()->getFolders('/', true)
	->filter(fn ($folder) => ! Str::startsWith($folder, '.'))
	)
	->map(fn ($folder) => ['path' => $folder, 'type' => 'dir'])
	)

getFolders calls $filesystem->allDirectories()

https://github.com/statamic/cms/blob/5.x/src/Filesystem/FlysystemAdapter.php#L66-L73

This translates to listContents(path: '', recursive: true) in Flysystem. For large S3 buckets a call to list all directories recursively like this will time out and in some cases even crash PHP. The asset browser will spin until the request fails and then fall back to an empty state which makes the asset screen and asset fieldtype unusable.

Downgrading to v4.25.0 fixes the issue.

How to reproduce

I assume you would need a large S3 bucket with thousands of directories and images. The bucket I'm working with 90 GB and has 478k items.

Logs

Environment

Environment
Application Name: Statamic
Laravel Version: 12.19.3
PHP Version: 8.3.22
Composer Version: 2.8.9
Environment: local
Debug Mode: ENABLED
URL: statamic.test
Maintenance Mode: OFF
Timezone: UTC
Locale: en

Cache
Config: NOT CACHED
Events: NOT CACHED
Routes: NOT CACHED
Views: NOT CACHED

Drivers
Broadcasting: log
Cache: file
Database: mysql
Logs: stack / single
Mail: log
Queue: sync
Session: file

Storage
public/storage: NOT LINKED

Statamic
Addons: 3
Sites: 1
Stache Watcher: Enabled (auto)
Static Caching: Disabled
Version: 5.58.1 PRO

Statamic Addons
statamic-rad-pack/meilisearch: 3.4.0
statamic-rad-pack/runway: 8.5.1
statamic/eloquent-driver: 4.25.1

Statamic Eloquent Driver
Asset Containers: file
Assets: eloquent
Blueprints: file
Collection Trees: file
Collections: file
Entries: file
Forms: file
Global Sets: file
Global Variables: file
Navigation Trees: file
Navigations: file
Revisions: eloquent
Sites: file
Taxonomies: file
Terms: eloquent
Tokens: file

Additional details

No response

[ryanmitchell]

This is a one-off task - it remembers it in your cache (as long as you don't have the stache watcher enabled). If you are seeing git on every request please check you dont have the watcher enabled, you really shouldnt.

If required, you could warm this cache as part of your deployment by calling \Statamic\Facades\AssetContainer::find('your-handle')->contents()->directories()

You can also add a cache store called asset_container_contents to your cache config, which allows you to only clear this as and when you want, instead of as part of a normal cache:clear.

(Somewhat related I've noticed we might be using the wrong cache store so I've opened: #464)

[galengidman]

Thanks for following up so quickly @ryanmitchell. Turning off the Stache watcher means that I can now warm this with two browser refreshes or one call to \Statamic\Facades\AssetContainer::find('assets')->contents()->directories(). Warming at the end of deployment seems like it could be workable.

That said, it's still 40 seconds plus. My bucket is largish (478k items, 90GB) but it's easy to imagine many larger. Especially easy to imagine that scenario when someone has a site big enough to need the Eloquent driver.

This might be difficult/impractical/impossible but I'm just thinking out loud. Would it be possible to rework this such that instead of doing double-entry bookkeeping all at once that we just scanned for empty directories as the current folder was being browsed? Meaning when you list / then scan the filesystem for directories in /, click into /foo and scan /foo, then into /foo/bar and scan /foo/bar etc. That way you're only ever looking one level deep at a time.

[ryanmitchell]

What you suggested is possible however, the intended approach would be as follows:

You set up cache store called asset_container_contents - and the driver uses that to store the cache. You warm it once (~40s) as suggested, and from then on it's only cleared if you do it explicitly. So you wouldn't do that on a deploy, and therefore theres no need to warm it after a deploy.

This means its a one off 40s hit.

[martyf]

Just adding a +1 to this, but with R2.

I have the new asset_container_contents and asset_meta stores set up.

With 4.30.2 they remain empty.

Rolling back to 4.25.0, everything works better (not 30 second+ delays). Upgrading again to 4.30.2 and it still works. But the new cache folders still remain empty.

[ryanmitchell]

I wouldn't expect the assets_meta cache store to contain anything as thats what the database table is storing.

I would expect asset_container_contents to have items in it if you have the stache watcher turned off - just confirming that you do?

Upgrading again to 4.30.2 and it still works. But the new cache folders still remain empty.

when you say it works, do you mean it works quickly or it works with a delay? and if a delay is that on all requests or just the first one?

[martyf]

Stache watcher is explicitly turned off (not auto), set to false.

Performance is slowish: but better than 30 seconds. It’s less than ideal but at least works - if that mixed message makes sense.

But asset_container_contents remains empty.

[ryanmitchell]

Hmm, that really strange.

I just ran some tests on my demo site against both local and s3 assets and for both asset_container_contents is being populated correctly.

If you run this in tinker app(Statamic\Assets\AssetContainerContents::class)->cacheStore()->get('asset-folder-contents-assets') do you get anything?

[martyf]

null is returned with that - locally for the same site, I get a Collection showing the expected paths.

The config/cache.php includes this in the stores array:

        'asset_container_contents' => [
            'driver' => 'file',
            'path' => storage_path('statamic/asset-container-contents'),
        ],

And FWIW, support:details has:

Cache
Config: CACHED
Events: CACHED
Routes: CACHED
Views: CACHED

Drivers
Broadcasting: log
Cache: file
Database: mysql
Logs: stack / single
Mail: postmark
Queue: redis
Session: file

Update 20 hours later
I've had to force back to 4.25.0 to get anything to load in a reasonable amount of time.

With 4.30.2 I started to experience delays again, with nothing output with the tinker command you provided earlier.

[ryanmitchell]

I don’t really have any suggestions I’m afraid - it looks like the folder list isn’t being cached on your install which is why it’s slow per request. Given it’s all standard laravel stuff that’s pretty odd.

I notice your config is cached. Just to rule that out if you don’t cache it does it make any difference?

[martyf]

Even with uncached config, still happens. Never found a solution so provisioned a new server and that works as expected with 4.30.2. Really bizarre behaviour here then... just have to be a head scratching one.