[Bug]: S3 External Storage with Hetzner is causing errors and is slow

[Io-Qo]

⚠️ This issue respects the following points: ⚠️

• This is a bug, not a question or a configuration/webserver/proxy issue.
• This issue is not already reported on Github OR Nextcloud Community Forum (I've searched it).
• Nextcloud Server is up to date. See Maintenance and Release Schedule for supported versions.
• I agree to follow Nextcloud's Code of Conduct.

Bug description

I have connected an external S3 storage (Object Storage from Hetzner) to my Nextcloud. The connection itself works fine and I can access files and upload new ones through Nextcloud but viewing files in the web interface is painfully slow and results in throwing a bunch of errors in the log files (see below). This is not related to a large amount of files, in fact some buckets I've tested and experienced this with only have a few dozen files in them. I can see in the logs that it fails to connect to the storage but it eventually seems to work after some time. I also don't get any error in the web interface, I just see the loading spinner for quite some time (a minute or so) and then everything is shown correctly.

I've tried to find the root cause of the errors but I'm unable to do so. I've checked and used the S3 storage with other clients and it works just fine without errors or lag/delays.

Steps to reproduce

1. Connect Hetzner Object Storage (Bucket) to Nextcloud
2. Create file structure
3. Try to view files through Nextcloud the web interface

Expected behavior

Reasonable performance when viewing files.

Nextcloud Server version

32

Operating system

Debian/Ubuntu

PHP engine version

PHP 8.2

Web server

Apache (supported)

Database engine version

MySQL

Is this bug present after an update or on a fresh install?

None

Are you using the Nextcloud Server Encryption module?

Encryption is Disabled

What user-backends are you using?

• Default user-backend (database)
• LDAP/ Active Directory
• SSO - SAML
• Other

Configuration report

{
    "system": {
        "instanceid": "***REMOVED SENSITIVE VALUE***",
        "passwordsalt": "***REMOVED SENSITIVE VALUE***",
        "secret": "***REMOVED SENSITIVE VALUE***",
        "trusted_domains": [
            "***REMOVED SENSITIVE VALUE***",
            "***REMOVED SENSITIVE VALUE***",
            "***REMOVED SENSITIVE VALUE***",
            "***REMOVED SENSITIVE VALUE***"
        ],
        "datadirectory": "***REMOVED SENSITIVE VALUE***",
        "dbtype": "mysql",
        "version": "32.0.2.2",
        "overwrite.cli.url": "***REMOVED SENSITIVE VALUE***",
        "dbname": "***REMOVED SENSITIVE VALUE***",
        "dbhost": "***REMOVED SENSITIVE VALUE***",
        "dbport": "",
        "dbtableprefix": "oc_",
        "mysql.utf8mb4": true,
        "dbuser": "***REMOVED SENSITIVE VALUE***",
        "dbpassword": "***REMOVED SENSITIVE VALUE***",
        "installed": true,
        "memcache.local": "\\OC\\Memcache\\APCu",
        "memcache.distributed": "\\OC\\Memcache\\Redis",
        "memcache.locking": "\\OC\\Memcache\\Redis",
        "redis": {
            "host": "***REMOVED SENSITIVE VALUE***",
            "port": 6379,
            "timeout": 0
        },
        "twofactor_enforced": "true",
        "twofactor_enforced_groups": [
            "Mitarbeiter",
            "Apotheken"
        ],
        "twofactor_enforced_excluded_groups": [
            "System",
            "ExternNo2FA",
            "guest_app",
            "Bots"
        ],
        "mail_from_address": "***REMOVED SENSITIVE VALUE***",
        "mail_smtpmode": "smtp",
        "mail_sendmailmode": "smtp",
        "mail_domain": "***REMOVED SENSITIVE VALUE***",
        "mail_smtphost": "***REMOVED SENSITIVE VALUE***",
        "mail_smtpport": "25",
        "mail_smtpsecure": "tls",
        "mail_smtpauth": 1,
        "mail_smtpname": "***REMOVED SENSITIVE VALUE***",
        "mail_smtppassword": "***REMOVED SENSITIVE VALUE***",
        "mail_smtpauthtype": "LOGIN",
        "maintenance": false,
        "default_language": "de",
        "default_locale": "de_DE",
        "theme": "",
        "loglevel": 2,
        "log.condition": [
            {
                "app": "spreed-bfp"
            }
        ],
        "default_phone_region": "DE",
        "app_install_overwrite": {
            "1": "flow_notifications",
            "2": "twofactor_admin",
            "3": "webhooks",
            "4": "oidc_login"
        },
        "defaultapp": "dashboard",
        "session_lifetime": 86400,
        "remember_login_cookie_lifetime": 1296000,
        "auto_logout": false,
        "session_keepalive": true,
        "enable_previews": true,
        "enabledPreviewProviders": [
            "OC\\Preview\\TXT",
            "OC\\Preview\\MarkDown",
            "OC\\Preview\\OpenDocument",
            "OC\\Preview\\PDF",
            "OC\\Preview\\MSOffice2003",
            "OC\\Preview\\MSOfficeDoc",
            "OC\\Preview\\PDF",
            "OC\\Preview\\Image",
            "OC\\Preview\\Photoshop",
            "OC\\Preview\\TIFF",
            "OC\\Preview\\SVG",
            "OC\\Preview\\Font",
            "OC\\Preview\\MP3",
            "OC\\Preview\\Movie",
            "OC\\Preview\\MKV",
            "OC\\Preview\\MP4",
            "OC\\Preview\\AVI"
        ],
        "preview_max_memory": 1024,
        "preview_max_x": 1024,
        "preview_max_y": 1024,
        "preview_max_filesize_image": 200,
        "trashbin_retention_obligation": "30,60",
        "onlyoffice": {
            "verify_peer_off": true,
            "jwt_header": "AuthorizationJwt"
        },
        "preview_libreoffice_path": "\/usr\/bin\/libreoffice",
        "htaccess.RewriteBase": "\/",
        "allow_user_to_change_display_name": false,
        "lost_password_link": "disabled",
        "oidc_login_provider_url": "***REMOVED SENSITIVE VALUE***",
        "oidc_login_client_id": "nextcloud",
        "oidc_login_client_secret": "***REMOVED SENSITIVE VALUE***",
        "oidc_login_auto_redirect": false,
        "oidc_login_end_session_redirect": false,
        "oidc_login_button_text": "Log in with Authelia",
        "oidc_login_hide_password_form": false,
        "oidc_login_use_id_token": true,
        "oidc_login_attributes": {
            "id": "preferred_username",
            "name": "name",
            "mail": "email",
            "groups": "groups"
        },
        "oidc_login_default_group": "oidc",
        "oidc_login_use_external_storage": false,
        "oidc_login_scope": "openid profile email groups",
        "oidc_login_proxy_ldap": false,
        "oidc_login_disable_registration": true,
        "oidc_login_redir_fallback": false,
        "oidc_login_alt_login_page": "assets\/login.php",
        "oidc_login_tls_verify": true,
        "oidc_create_groups": false,
        "oidc_login_webdav_enabled": false,
        "oidc_login_password_authentication": false,
        "oidc_login_public_key_caching_time": 86400,
        "oidc_login_min_time_between_jwks_requests": 10,
        "oidc_login_well_known_caching_time": 86400,
        "oidc_login_update_avatar": false,
        "auth.webauthn.enabled": false,
        "updater.release.channel": "stable",
        "maintenance_window_start": 1
    }
}

List of activated Apps

Enabled:
  - activity: 5.0.0-dev.0
  - announcementcenter: 7.2.2
  - app_api: 32.0.0
  - approval: 2.6.0
  - bookmarks: 16.0.1
  - bruteforcesettings: 5.0.0-dev.0
  - calendar: 6.1.1
  - cloud_federation_api: 1.16.0
  - comments: 1.22.0
  - contacts: 8.1.1
  - contactsinteraction: 1.13.1
  - dashboard: 7.12.0
  - dav: 1.34.2
  - deck: 1.16.2
  - federatedfilesharing: 1.22.0
  - federation: 1.22.0
  - files: 2.4.0
  - files_accesscontrol: 3.0.2
  - files_antivirus: 6.1.0
  - files_automatedtagging: 3.0.1
  - files_downloadlimit: 5.0.0-dev.0
  - files_external: 1.24.0
  - files_pdfviewer: 5.0.0-dev.0
  - files_reminders: 1.5.0
  - files_retention: 3.0.0
  - files_sharing: 1.24.1
  - files_trashbin: 1.22.0
  - files_versions: 1.25.0
  - flow_notifications: 3.0.0
  - fulltextsearch_elasticsearch: 32.0.0
  - groupfolders: 20.1.5
  - logreader: 5.0.0-dev.0
  - lookup_server_connector: 1.20.0
  - news: 27.2.0
  - nextcloud_announcements: 4.0.0-dev.0
  - notes: 4.12.4
  - notifications: 5.0.0-dev.0
  - oauth2: 1.20.0
  - oidc_login: 3.2.2
  - password_policy: 4.0.0-dev.0
  - passwords: 2025.12.10
  - privacy: 4.0.0-dev.0
  - profile: 1.1.0
  - provisioning_api: 1.22.0
  - related_resources: 3.0.0-dev.0
  - richdocuments: 9.0.2
  - settings: 1.15.1
  - sharebymail: 1.22.0
  - spreed: 22.0.4
  - support: 4.0.0-dev.0
  - systemtags: 1.22.0
  - text: 6.0.1
  - theming: 2.7.0
  - theming_customcss: 1.19.0
  - twofactor_admin: 4.9.0
  - twofactor_backupcodes: 1.21.0
  - twofactor_nextcloud_notification: 6.0.0-dev.0
  - twofactor_totp: 14.0.0
  - twofactor_webauthn: 2.4.1
  - updatenotification: 1.22.0
  - user_saml: 7.1.1
  - user_status: 1.12.0
  - viewer: 5.0.0-dev.0
  - weather_status: 1.12.0
  - webhook_listeners: 1.3.0
  - workflow_pdf_converter: 3.0.0
  - workflow_script: 3.0.0
  - workflowengine: 2.14.0
Disabled:
  - admin_audit: 1.22.0
  - circles: 32.0.0 (installed 27.0.1)
  - encryption: 2.20.0
  - firstrunwizard: 5.0.0-dev.0 (installed 2.12.0)
  - fulltextsearch: 31.0.0 (installed 31.0.0)
  - mail: 5.0.3 (installed 5.0.3)
  - photos: 5.0.0-dev.1 (installed 2.0.1)
  - polls: 7.4.2 (installed 7.4.2)
  - recommendations: 5.0.0-dev.0 (installed 1.6.0)
  - serverinfo: 4.0.0-dev.0 (installed 1.17.0)
  - survey_client: 4.0.0-dev.0 (installed 1.15.0)
  - suspicious_login: 10.0.0-dev.0
  - user_ldap: 1.23.0
  - webhooks: 0.4.3 (installed 0.4.3)
  - welcome: 1.2.1 (installed 1.2.1)

Nextcloud Signing status

No errors have been found.

Nextcloud Logs

{"reqId":"5liwgMd70Hw8fUUP839f","level":2,"time":"2026-01-04T19:03:29+00:00","remoteAddr":"***","user":"***","app":"PHP","method":"PROPFIND","url":"/remote.php/dav/files/***/***/","message":"fopen(httpseek://): Failed to open stream: "OC\\Files\\Stream\\SeekableHttpStream::stream_open" call failed at /var/www/nextcloud/lib/private/Files/Stream/SeekableHttpStream.php#50","userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/143.0.0.0 Safari/537.36","version":"32.0.2.2","data":{"app":"PHP"}}
{"reqId":"5liwgMd70Hw8fUUP839f","level":3,"time":"2026-01-04T19:03:29+00:00","remoteAddr":"***","user":"***","app":"files_external","method":"PROPFIND","url":"/remote.php/dav/files/***/***/","message":"Failed to read object .","userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/143.0.0.0 Safari/537.36","version":"32.0.2.2","exception":{"Exception":"Exception","Message":"Failed to read object .","Code":0,"Trace":[{"file":"/var/www/nextcloud/apps/files_external/lib/Lib/Storage/AmazonS3.php","line":446,"function":"readObject","class":"OCA\\Files_External\\Lib\\Storage\\AmazonS3","type":"->"},{"file":"/var/www/nextcloud/lib/private/Files/Storage/Wrapper/Wrapper.php","line":143,"function":"fopen","class":"OCA\\Files_External\\Lib\\Storage\\AmazonS3","type":"->"},{"file":"/var/www/nextcloud/lib/private/Files/Storage/Wrapper/PermissionsMask.php","line":98,"function":"fopen","class":"OC\\Files\\Storage\\Wrapper\\Wrapper","type":"->"},{"file":"/var/www/nextcloud/lib/private/Files/Storage/Wrapper/Wrapper.php","line":143,"function":"fopen","class":"OC\\Files\\Storage\\Wrapper\\PermissionsMask","type":"->"},{"function":"fopen","class":"OC\\Files\\Storage\\Wrapper\\Wrapper","type":"->"},{"file":"/var/www/nextcloud/lib/private/Files/Storage/Wrapper/Availability.php","line":83,"function":"call_user_func_array"},{"file":"/var/www/nextcloud/lib/private/Files/Storage/Wrapper/Availability.php","line":178,"function":"handleAvailability","class":"OC\\Files\\Storage\\Wrapper\\Availability","type":"->"},{"file":"/var/www/nextcloud/apps/files_antivirus/lib/AvirWrapper.php","line":76,"function":"fopen","class":"OC\\Files\\Storage\\Wrapper\\Availability","type":"->"},{"file":"/var/www/nextcloud/lib/private/Files/Storage/Wrapper/Wrapper.php","line":143,"function":"fopen","class":"OCA\\Files_Antivirus\\AvirWrapper","type":"->"},{"file":"/var/www/nextcloud/apps/files_accesscontrol/lib/StorageWrapper.php","line":236,"function":"fopen","class":"OC\\Files\\Storage\\Wrapper\\Wrapper","type":"->"},{"file":"/var/www/nextcloud/apps/richdocuments/lib/Storage/SecureViewWrapper.php","line":45,"function":"fopen","class":"OCA\\FilesAccessControl\\StorageWrapper","type":"->"},{"file":"/var/www/nextcloud/apps/richdocuments/lib/Service/SecureViewService.php","line":36,"function":"fopen","class":"OCA\\Richdocuments\\Storage\\SecureViewWrapper","type":"->"},{"file":"/var/www/nextcloud/apps/richdocuments/lib/DAV/SecureViewPlugin.php","line":74,"function":"shouldSecure","class":"OCA\\Richdocuments\\Service\\SecureViewService","type":"->"},{"file":"/var/www/nextcloud/apps/richdocuments/lib/DAV/SecureViewPlugin.php","line":56,"function":"isDownloadable","class":"OCA\\Richdocuments\\DAV\\SecureViewPlugin","type":"->"},{"file":"/var/www/nextcloud/3rdparty/sabre/event/lib/WildcardEmitterTrait.php","line":89,"function":"handleGetProperties","class":"OCA\\Richdocuments\\DAV\\SecureViewPlugin","type":"->"},{"file":"/var/www/nextcloud/3rdparty/sabre/dav/lib/DAV/Server.php","line":1052,"function":"emit","class":"Sabre\\DAV\\Server","type":"->"},{"file":"/var/www/nextcloud/3rdparty/sabre/dav/lib/DAV/Server.php","line":984,"function":"getPropertiesByNode","class":"Sabre\\DAV\\Server","type":"->"},{"file":"/var/www/nextcloud/3rdparty/sabre/dav/lib/DAV/Server.php","line":1664,"function":"getPropertiesIteratorForPath","class":"Sabre\\DAV\\Server","type":"->"},{"file":"/var/www/nextcloud/3rdparty/sabre/dav/lib/DAV/Server.php","line":1649,"function":"writeMultiStatus","class":"Sabre\\DAV\\Server","type":"->"},{"file":"/var/www/nextcloud/3rdparty/sabre/dav/lib/DAV/CorePlugin.php","line":346,"function":"generateMultiStatus","class":"Sabre\\DAV\\Server","type":"->"},{"file":"/var/www/nextcloud/3rdparty/sabre/event/lib/WildcardEmitterTrait.php","line":89,"function":"httpPropFind","class":"Sabre\\DAV\\CorePlugin","type":"->"},{"file":"/var/www/nextcloud/3rdparty/sabre/dav/lib/DAV/Server.php","line":472,"function":"emit","class":"Sabre\\DAV\\Server","type":"->"},{"file":"/var/www/nextcloud/apps/dav/lib/Connector/Sabre/Server.php","line":211,"function":"invokeMethod","class":"Sabre\\DAV\\Server","type":"->"},{"file":"/var/www/nextcloud/apps/dav/lib/Server.php","line":424,"function":"start","class":"OCA\\DAV\\Connector\\Sabre\\Server","type":"->"},{"file":"/var/www/nextcloud/apps/dav/appinfo/v2/remote.php","line":22,"function":"exec","class":"OCA\\DAV\\Server","type":"->"},{"file":"/var/www/nextcloud/remote.php","line":151,"args":["/var/www/nextcloud/apps/dav/appinfo/v2/remote.php"],"function":"require_once"}],"File":"/var/www/nextcloud/lib/private/Files/ObjectStore/S3ObjectTrait.php","Line":77,"message":"Failed to read object .","exception":{},"CustomMessage":"Failed to read object ."}}
{"reqId":"5liwgMd70Hw8fUUP839f","level":2,"time":"2026-01-04T19:03:29+00:00","remoteAddr":"***","user":"***","app":"richdocuments","method":"PROPFIND","url":"/remote.php/dav/files/***/***/","message":"SecureViewPlugin caught an exception that likely is ignorable. Still preventing download.","userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/143.0.0.0 Safari/537.36","version":"32.0.2.2","exception":{"Exception":"TypeError","Message":"fclose(): Argument #1 ($stream) must be of type resource, bool given","Code":0,"Trace":[{"file":"/var/www/nextcloud/apps/richdocuments/lib/Service/SecureViewService.php","line":37,"function":"fclose","args":["*** sensitive parameters replaced ***"]},{"file":"/var/www/nextcloud/apps/richdocuments/lib/DAV/SecureViewPlugin.php","line":74,"function":"shouldSecure","class":"OCA\\Richdocuments\\Service\\SecureViewService","type":"->"},{"file":"/var/www/nextcloud/apps/richdocuments/lib/DAV/SecureViewPlugin.php","line":56,"function":"isDownloadable","class":"OCA\\Richdocuments\\DAV\\SecureViewPlugin","type":"->"},{"file":"/var/www/nextcloud/3rdparty/sabre/event/lib/WildcardEmitterTrait.php","line":89,"function":"handleGetProperties","class":"OCA\\Richdocuments\\DAV\\SecureViewPlugin","type":"->"},{"file":"/var/www/nextcloud/3rdparty/sabre/dav/lib/DAV/Server.php","line":1052,"function":"emit","class":"Sabre\\DAV\\Server","type":"->"},{"file":"/var/www/nextcloud/3rdparty/sabre/dav/lib/DAV/Server.php","line":984,"function":"getPropertiesByNode","class":"Sabre\\DAV\\Server","type":"->"},{"file":"/var/www/nextcloud/3rdparty/sabre/dav/lib/DAV/Server.php","line":1664,"function":"getPropertiesIteratorForPath","class":"Sabre\\DAV\\Server","type":"->"},{"file":"/var/www/nextcloud/3rdparty/sabre/dav/lib/DAV/Server.php","line":1649,"function":"writeMultiStatus","class":"Sabre\\DAV\\Server","type":"->"},{"file":"/var/www/nextcloud/3rdparty/sabre/dav/lib/DAV/CorePlugin.php","line":346,"function":"generateMultiStatus","class":"Sabre\\DAV\\Server","type":"->"},{"file":"/var/www/nextcloud/3rdparty/sabre/event/lib/WildcardEmitterTrait.php","line":89,"function":"httpPropFind","class":"Sabre\\DAV\\CorePlugin","type":"->"},{"file":"/var/www/nextcloud/3rdparty/sabre/dav/lib/DAV/Server.php","line":472,"function":"emit","class":"Sabre\\DAV\\Server","type":"->"},{"file":"/var/www/nextcloud/apps/dav/lib/Connector/Sabre/Server.php","line":211,"function":"invokeMethod","class":"Sabre\\DAV\\Server","type":"->"},{"file":"/var/www/nextcloud/apps/dav/lib/Server.php","line":424,"function":"start","class":"OCA\\DAV\\Connector\\Sabre\\Server","type":"->"},{"file":"/var/www/nextcloud/apps/dav/appinfo/v2/remote.php","line":22,"function":"exec","class":"OCA\\DAV\\Server","type":"->"},{"file":"/var/www/nextcloud/remote.php","line":151,"args":["/var/www/nextcloud/apps/dav/appinfo/v2/remote.php"],"function":"require_once"}],"File":"/var/www/nextcloud/apps/richdocuments/lib/Service/SecureViewService.php","Line":37,"message":"SecureViewPlugin caught an exception that likely is ignorable. Still preventing download.","exception":{},"CustomMessage":"SecureViewPlugin caught an exception that likely is ignorable. Still preventing download."}}
{"reqId":"5liwgMd70Hw8fUUP839f","level":2,"time":"2026-01-04T19:03:29+00:00","remoteAddr":"***","user":"***","app":"PHP","method":"PROPFIND","url":"/remote.php/dav/files/***/***/","message":"fopen(https://***.nbg1.your-objectstorage.com/Archiv): Failed to open stream: HTTP request failed! HTTP/1.1 404 Not Found\r\n at /var/www/nextcloud/lib/private/Files/ObjectStore/S3ObjectTrait.php#74","userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/143.0.0.0 Safari/537.36","version":"32.0.2.2","data":{"app":"PHP"}}
{"reqId":"5liwgMd70Hw8fUUP839f","level":2,"time":"2026-01-04T19:03:29+00:00","remoteAddr":"***","user":"***","app":"PHP","method":"PROPFIND","url":"/remote.php/dav/files/***/***/","message":"fopen(httpseek://): Failed to open stream: "OC\\Files\\Stream\\SeekableHttpStream::stream_open" call failed at /var/www/nextcloud/lib/private/Files/Stream/SeekableHttpStream.php#50","userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/143.0.0.0 Safari/537.36","version":"32.0.2.2","data":{"app":"PHP"}}
{"reqId":"5liwgMd70Hw8fUUP839f","level":3,"time":"2026-01-04T19:03:29+00:00","remoteAddr":"***","user":"***","app":"files_external","method":"PROPFIND","url":"/remote.php/dav/files/***/***/","message":"Failed to read object Archiv","userAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/143.0.0.0 Safari/537.36","version":"32.0.2.2","exception":{"Exception":"Exception","Message":"Failed to read object Archiv","Code":0,"Trace":[{"file":"/var/www/nextcloud/apps/files_external/lib/Lib/Storage/AmazonS3.php","line":446,"function":"readObject","class":"OCA\\Files_External\\Lib\\Storage\\AmazonS3","type":"->"},{"file":"/var/www/nextcloud/lib/private/Files/Storage/Wrapper/Wrapper.php","line":143,"function":"fopen","class":"OCA\\Files_External\\Lib\\Storage\\AmazonS3","type":"->"},{"file":"/var/www/nextcloud/lib/private/Files/Storage/Wrapper/PermissionsMask.php","line":98,"function":"fopen","class":"OC\\Files\\Storage\\Wrapper\\Wrapper","type":"->"},{"file":"/var/www/nextcloud/lib/private/Files/Storage/Wrapper/Wrapper.php","line":143,"function":"fopen","class":"OC\\Files\\Storage\\Wrapper\\PermissionsMask","type":"->"},{"function":"fopen","class":"OC\\Files\\Storage\\Wrapper\\Wrapper","type":"->"},{"file":"/var/www/nextcloud/lib/private/Files/Storage/Wrapper/Availability.php","line":83,"function":"call_user_func_array"},{"file":"/var/www/nextcloud/lib/private/Files/Storage/Wrapper/Availability.php","line":178,"function":"handleAvailability","class":"OC\\Files\\Storage\\Wrapper\\Availability","type":"->"},{"file":"/var/www/nextcloud/apps/files_antivirus/lib/AvirWrapper.php","line":76,"function":"fopen","class":"OC\\Files\\Storage\\Wrapper\\Availability","type":"->"},{"file":"/var/www/nextcloud/lib/private/Files/Storage/Wrapper/Wrapper.php","line":143,"function":"fopen","class":"OCA\\Files_Antivirus\\AvirWrapper","type":"->"},{"file":"/var/www/nextcloud/apps/files_accesscontrol/lib/StorageWrapper.php","line":236,"function":"fopen","class":"OC\\Files\\Storage\\Wrapper\\Wrapper","type":"->"},{"file":"/var/www/nextcloud/apps/richdocuments/lib/Storage/SecureViewWrapper.php","line":45,"function":"fopen","class":"OCA\\FilesAccessControl\\StorageWrapper","type":"->"},{"file":"/var/www/nextcloud/apps/richdocuments/lib/Service/SecureViewService.php","line":36,"function":"fopen","class":"OCA\\Richdocuments\\Storage\\SecureViewWrapper","type":"->"},{"file":"/var/www/nextcloud/apps/richdocuments/lib/DAV/SecureViewPlugin.php","line":74,"function":"shouldSecure","class":"OCA\\Richdocuments\\Service\\SecureViewService","type":"->"},{"file":"/var/www/nextcloud/apps/richdocuments/lib/DAV/SecureViewPlugin.php","line":56,"function":"isDownloadable","class":"OCA\\Richdocuments\\DAV\\SecureViewPlugin","type":"->"},{"file":"/var/www/nextcloud/3rdparty/sabre/event/lib/WildcardEmitterTrait.php","line":89,"function":"handleGetProperties","class":"OCA\\Richdocuments\\DAV\\SecureViewPlugin","type":"->"},{"file":"/var/www/nextcloud/3rdparty/sabre/dav/lib/DAV/Server.php","line":1052,"function":"emit","class":"Sabre\\DAV\\Server","type":"->"},{"file":"/var/www/nextcloud/3rdparty/sabre/dav/lib/DAV/Server.php","line":984,"function":"getPropertiesByNode","class":"Sabre\\DAV\\Server","type":"->"},{"file":"/var/www/nextcloud/3rdparty/sabre/dav/lib/DAV/Server.php","line":1664,"function":"getPropertiesIteratorForPath","class":"Sabre\\DAV\\Server","type":"->"},{"file":"/var/www/nextcloud/3rdparty/sabre/dav/lib/DAV/Server.php","line":1649,"function":"writeMultiStatus","class":"Sabre\\DAV\\Server","type":"->"},{"file":"/var/www/nextcloud/3rdparty/sabre/dav/lib/DAV/CorePlugin.php","line":346,"function":"generateMultiStatus","class":"Sabre\\DAV\\Server","type":"->"},{"file":"/var/www/nextcloud/3rdparty/sabre/event/lib/WildcardEmitterTrait.php","line":89,"function":"httpPropFind","class":"Sabre\\DAV\\CorePlugin","type":"->"},{"file":"/var/www/nextcloud/3rdparty/sabre/dav/lib/DAV/Server.php","line":472,"function":"emit","class":"Sabre\\DAV\\Server","type":"->"},{"file":"/var/www/nextcloud/apps/dav/lib/Connector/Sabre/Server.php","line":211,"function":"invokeMethod","class":"Sabre\\DAV\\Server","type":"->"},{"file":"/var/www/nextcloud/apps/dav/lib/Server.php","line":424,"function":"start","class":"OCA\\DAV\\Connector\\Sabre\\Server","type":"->"},{"file":"/var/www/nextcloud/apps/dav/appinfo/v2/remote.php","line":22,"function":"exec","class":"OCA\\DAV\\Server","type":"->"},{"file":"/var/www/nextcloud/remote.php","line":151,"args":["/var/www/nextcloud/apps/dav/appinfo/v2/remote.php"],"function":"require_once"}],"File":"/var/www/nextcloud/lib/private/Files/ObjectStore/S3ObjectTrait.php","Line":77,"message":"Failed to read object Archiv","exception":{},"CustomMessage":"Failed to read object Archiv"}}

Additional info

No response

[solracsf]

First, it seems that S3 from Hetzner is slow and error prone. Just check Reddit and other forums.

Second, other clients may have a very agressive retry policy, where Nextcloud can't wait indefinitely for the object to return and fails fast.

Failed to read object

This clearly indicates failures from the upstream service.

[Io-Qo]

Thank you for your reply. The S3 from Hetzner had some issues when introduced but this hasn't been an issue as of lately.
As I said I've tried multiple other clients and they work at high speeds (directories load instantaneously) with no errors whatsoever anywhere. Nextcloud is the only client I'm having this issue with.

I've done some further testing and the problem seems occur because the $request variable from calling the method readObject() in /nextcloud/lib/private/Files/ObjectStore/S3ObjectTrait.php is NULL when the error is thrown by the app "richdocuments". That's certainly not intended behaviour? Obviously the fopen fails because $request->getUri() is not a object/method.

[solracsf]

Allow me to disagree: https://www.reddit.com/r/hetzner/s/P0rWqdMzpb

But maybe there is a bug somewhere, feel free to submit a PR after debugging it further.

[Io-Qo]

So I can pinpoint the error now:

Nextcloud is creating a filecache entry for folders that do not exist as objects on the S3 storage. For example I have a folder called "Archive" but that does not exist in the S3 logic but a file "Archive/File.pdf" does. So if I try to get the properties via aws command line of the folder "bucket.blabla.com/Archive" I get a 404 error, the same 404 error that also gets logged in Nextcloud logfiles. Nextcloud tries to readObject() of that folder which it treats as an storage object although it is not.

I'm not a 100% sure but I don't think folders are real objects in any S3 implementation and are just a prefix convention, therefore Nextcloud trying to apply readObject() on a non-S3-object seems to be malformed and should be considered a bug that needs fixing but I'm not sure how the entire Nextcloud thing is working here. I can see in the filecache table that the entry for the folder has the mimetype 2 so these could be excluded all together imo. Another solution would be appending a "/" to the path for mimetype 2 entries for the readObject() calls. But then again, I have no real understanding of the inner workings of Nextcloud here.

[joshtrichards]

fopen(https://***.nbg1.your-objectstorage.com/Archiv): Failed to open stream: HTTP request failed! HTTP/1.1 404 Not Found

The troubleshooting/logs can get a bit messy, particularly when there are multiple wrappers (files_accesscontrol, files_antivirus, files_richdocument SecureView, PermissionsMask, etc) like there are here. Some of the log entries are likely just byproduct failures, but not root cause related. Though some of the byproduct failures could be further exasperating things too (e.g. triggering rate limit), in addition to making the logs noisier.

At least part of the problem does appear to be the Office (richdocuments) SecureWrapper. It doesn't differentiate between folders versus files so it's atttempting an fopen() on folders needlessly (when it tries to make sure the filecache is up-to-date/available). This probably has a bigger negative side effect on S3/External Storage. Whether that is the root cause here -- or just a side effect that could be dealt with better -- is hard to say at this point in time. Either way, that should be addressed -- working on some code for that now.

I've checked and used the S3 storage with other clients and it works just fine without errors or lag/delays.

Can you clarify whether these clients are running on/co-located on the same server/network hosting Nextcloud or do you mean from a remote workstation? Just trying to understanding performance differences.

I can see in the filecache table that the entry for the folder has the mimetype 2 so these could be excluded all together imo

Are you saying mimetype field is empty? I'd expect the mimetype for Archiv to be set to httpd/unix-directory in your filecache table.

For what it's worth, the underlying S3 client in Nextcloud is the official AWS S3 SDK.

[joshtrichards]

@Io-Qo Any chance you can/are willing to test the change proposed in the following PR within your environment?

nextcloud/richdocuments#5257