Race condition when removing users

[v-scharf]

Describe the bug

related #1877

Steps to reproduce

1. run opencloud
2. run test several times USE_BEARER_TOKEN=true TEST_SERVER_URL=https://cloud.opencloud.test make test-acceptance-api BEHAT_FEATURE=tests/acceptance/features/apiSharingNgShareInvitation/shareInvitations.feature:1625

Expected behavior

no error

Actual behavior

error log:

{"level":"error","service":"storage-users","host.name":"dccd0e3abc1c","pkg":"rgrpc","traceid":"b78450a7eb199466805d9c4d430958cd","error":"node.Xattr /var/lib/opencloud/storage/users/users/7c6d6720-271e-416d-ace3-bcc34320553a user.oc.name: no data available","id":"1272256e-37f8-4a85-b2aa-3652d261cf94","time":"2025-11-19T13:09:28Z","line":"github.com/opencloud-eu/reva/v2@v2.39.3-0.20251113164418-9fd6b6864c10/pkg/storage/pkg/decomposedfs/spaces.go:317","message":"could not read node"}

{"level":"error","service":"storage-users","host.name":"dccd0e3abc1c","pkg":"rgrpc","traceid":"b78450a7eb199466805d9c4d430958cd","error":"node.Xattr /var/lib/opencloud/storage/users/users/7c6d6720-271e-416d-ace3-bcc34320553a user.oc.name: no data available","status":{"code":15,"message":"error listing spaces","trace":"b78450a7eb199466805d9c4d430958cd"},"filters":[{"type":2,"Term":{"Id":{"opaque_id":"77b1193c-84f6-43ae-b14d-a9bfa479de1d$1272256e-37f8-4a85-b2aa-3652d261cf94!1272256e-37f8-4a85-b2aa-3652d261cf94"}}},{"type":4,"Term":{"SpaceType":"+grant"}}],"time":"2025-11-19T13:09:28Z","line":"github.com/opencloud-eu/reva/v2@v2.39.3-0.20251113164418-9fd6b6864c10/internal/grpc/services/storageprovider/storageprovider.go:579","message":"failed to list storage spaces"}

Setup

Please describe how you started the server and provide a list of relevant environment variables or configuration files.

Details

OC_XXX=somevalue
OC_YYY=somevalue
PROXY_XXX=somevalue

Additional context

Add any other context about the problem here.

[michaelstingl]

Additional reproduction case from production

Issue	Problem	Expected
Orphaned index files	by-user-id/*.mpk remain after user deletion	Index cleanup on user deletion
Space deletion fails	xattr.Remove returns "no data available"	Graceful handling of missing xattr

Scenario

1. Project space with multiple user/group grants
2. Users deleted (via Web UI or API)
3. by-user-id/{user-uuid}.mpk index files not cleaned up
4. Space deletion now fails permanently
5. Error log spam on every ListStorageSpaces call — same error as CI. opencloud log contains a lot of errors #2045 (error 2 in that issue)

Affected index files

Index	Problem
by-user-id/{deleted-user}.mpk	Entire file is orphaned
by-user-id/{active-user}.mpk	Contains grants to deleted spaces
by-group-id/{group}.mpk	Contains grants to deleted spaces
by-type/project.mpk	Ghost space triggers directory recreation + error loop

Workaround

⚠️ Warning: This involves manual editing of binary index files. Mistakes can make spaces inaccessible. Always back up the entire indexes directory before making changes. Only proceed if you understand the MessagePack format and have verified which entries are ghosts.

Requires Python with msgpack (pip3 install msgpack).

# 1. Identify ghost space UUIDs
#    Compare project directories with active drives:
ls /storage/users/projects/
#    vs.
curl ... /graph/v1.0/drives | jq '.value[] | select(.driveType=="project") | .id'
#    Directories without matching active drive = ghost spaces

# 2. Stop OpenCloud (consistent index state!)
docker compose stop opencloud

# 3. Back up ALL index files before any changes
cp -a /storage/users/indexes/ /storage/users/indexes.bak-$(date +%Y%m%d)

# 4. Delete orphaned user index files (users that no longer exist)
rm /storage/users/indexes/by-user-id/{deleted-user-uuid}.mpk

# 5. Find and remove ghost entries from active indexes (MessagePack)
grep -rl "GHOST-SPACE-UUID" /storage/users/indexes/

# Edit each affected file (writes to temp file first):
python3 -c "
import msgpack
with open('file.mpk', 'rb') as f:
    data = msgpack.unpackb(f.read(), raw=False)
print('Before:', list(data.keys()))
del data['GHOST-SPACE-UUID']
print('After:', list(data.keys()))
with open('file.mpk.tmp', 'wb') as f:
    f.write(msgpack.packb(data, use_bin_type=True))
"
mv file.mpk.tmp file.mpk

# 6. Delete ghost space directories
rm -rf /storage/users/projects/{ghost-space-uuid}

# 7. Start OpenCloud
docker compose start opencloud

# 8. If something went wrong, restore backup:
# docker compose stop opencloud
# rm -rf /storage/users/indexes/
# mv /storage/users/indexes.bak-YYYYMMDD /storage/users/indexes/
# docker compose start opencloud

Simply deleting orphaned by-user-id/*.mpk files is not sufficient. Ghost entries in by-type/project.mpk and active user/group indexes must also be removed — otherwise OpenCloud recreates ghost directories and continues the error loop.

Environment: OpenCloud 5.0, Reva v2.42.3, PosixFS (also seen on 4.1 / Reva v2.41.0)

[michaelstingl]

Opened two reva PRs, one for each read site of this:

• fix(decomposedfs): skip a space whose name attribute is unset reva#710
• fix(decomposedfs): tolerate a space root with an unset name attribute reva#713

The first fixes the storage-users ListStorageSpaces listing; the second fixes the search service's tree walk, which reads the space root through the same ReadNode path. Each makes the read skip a space that still references a deleted user (a node with no user.oc.name) instead of failing, so the user.oc.name: no data available errors stop.

It is still a partial fix: the orphaned index entries are not cleaned up, only tolerated on read, and new ones still appear when a user is deleted. Preventing them (cleanup on user deletion) is a separate change.

_{🤖 drafted with Claude Code, reviewed before submitting.}