Version
payload 3.88.0
@payloadcms/db-postgres 3.88.0
- Node 22
What happens
addSessionToUser appends to the user's sessions array with a
read-modify-write of the whole document
(packages/payload/src/auth/sessions.ts):
user.sessions = removeExpiredSessions(user.sessions)
user.sessions.push(session)
await payload.db.updateOne({ id: user.id, collection, data: user, req })
Two logins for the same account that overlap both read the same array, both
write their own version, and one session is lost. logout has the same shape in
reverse (auth/operations/logout.ts filters the array and writes it back), so a
logout concurrent with a login can also delete the login's session.
The user who lost the race holds a JWT that verifies correctly against a session
that is not there. Every subsequent request is anonymous:
JWTAuthentication finds no matching sid and returns { user: null } with no
error and nothing logged. From the outside it looks like "I signed in and it
signed me straight out".
Why it matters
It is not only a load-test artefact. Any shared account — a shop floor, a
support desk — hits it, and so does one person signing in on two devices at
once. The failure is silent on both the client and the server.
Minimal reproduction
Any collection with auth: {} (sessions are on by default since the sid
claim was introduced). Fire N logins concurrently, then check each token:
const N = 8
const BASE = 'http://localhost:3000'
const logins = await Promise.all(
Array.from({ length: N }, async () => {
const res = await fetch(`${BASE}/api/users/login`, {
method: 'POST',
headers: { 'Content-Type': 'application/json', Origin: BASE },
body: JSON.stringify({ email: 'admin@example.com', password: 'test' }),
})
return /payload-token=([^;]+)/.exec(res.headers.get('set-cookie') ?? '')?.[1] ?? ''
}),
)
let working = 0
for (const token of logins) {
const me = await fetch(`${BASE}/api/users/me`, {
headers: { Cookie: `payload-token=${token}`, Origin: BASE },
})
if ((await me.json()).user) working += 1
}
console.log(`${working}/${N} sessions survived`)
Observed: 1/8, 1/8, 2/8 on three consecutive runs.
Expected: 8/8.
The Origin header is required or extractJWT refuses the cookie — unrelated
to this bug, but it makes the repro fail for the wrong reason without it.
A second effect worth fixing alongside
removeExpiredSessions only drops sessions that have expired. Nothing
bounds the list otherwise, so it grows for the life of the account and the whole
array is rewritten on every login. Eighty sessions accumulated on one account
during an afternoon of testing; by then a single login took long enough that a
two-second advisory lock around it expired mid-write, which is how we first
mistook the fix for working.
Suggested fix
Make the append atomic rather than a read-modify-write — an insert into the
sessions table for relational adapters, or $push for Mongo — so concurrent
logins do not have to read the array at all. Failing that, a bounded retry on a
version/updatedAt check would at least make the loss detectable.
What we did instead
Wrapped POST /api/users/login and POST /api/users/logout in a per-account
Redis lock so the two writers serialise, and added an afterLogin hook that
trims the list to the twenty most recent sessions. Both are marked in our code
to be removed once this is fixed upstream.
Worth noting that the lock alone was not enough at first: with it demonstrably
held on every pass, ten serialised logins still produced only six net sessions,
because by then the account had eighty sessions and the whole array rewrite was
outrunning a two-second lease. That is the second effect above, and it is why we
think the unbounded list is worth fixing alongside the race rather than after it.
Version
payload3.88.0@payloadcms/db-postgres3.88.0What happens
addSessionToUserappends to the user'ssessionsarray with aread-modify-write of the whole document
(
packages/payload/src/auth/sessions.ts):Two logins for the same account that overlap both read the same array, both
write their own version, and one session is lost.
logouthas the same shape inreverse (
auth/operations/logout.tsfilters the array and writes it back), so alogout concurrent with a login can also delete the login's session.
The user who lost the race holds a JWT that verifies correctly against a session
that is not there. Every subsequent request is anonymous:
JWTAuthenticationfinds no matchingsidand returns{ user: null }with noerror and nothing logged. From the outside it looks like "I signed in and it
signed me straight out".
Why it matters
It is not only a load-test artefact. Any shared account — a shop floor, a
support desk — hits it, and so does one person signing in on two devices at
once. The failure is silent on both the client and the server.
Minimal reproduction
Any collection with
auth: {}(sessions are on by default since thesidclaim was introduced). Fire N logins concurrently, then check each token:
Observed:
1/8,1/8,2/8on three consecutive runs.Expected:
8/8.The
Originheader is required orextractJWTrefuses the cookie — unrelatedto this bug, but it makes the repro fail for the wrong reason without it.
A second effect worth fixing alongside
removeExpiredSessionsonly drops sessions that have expired. Nothingbounds the list otherwise, so it grows for the life of the account and the whole
array is rewritten on every login. Eighty sessions accumulated on one account
during an afternoon of testing; by then a single login took long enough that a
two-second advisory lock around it expired mid-write, which is how we first
mistook the fix for working.
Suggested fix
Make the append atomic rather than a read-modify-write — an insert into the
sessions table for relational adapters, or
$pushfor Mongo — so concurrentlogins do not have to read the array at all. Failing that, a bounded retry on a
version/
updatedAtcheck would at least make the loss detectable.What we did instead
Wrapped
POST /api/users/loginandPOST /api/users/logoutin a per-accountRedis lock so the two writers serialise, and added an
afterLoginhook thattrims the list to the twenty most recent sessions. Both are marked in our code
to be removed once this is fixed upstream.
Worth noting that the lock alone was not enough at first: with it demonstrably
held on every pass, ten serialised logins still produced only six net sessions,
because by then the account had eighty sessions and the whole array rewrite was
outrunning a two-second lease. That is the second effect above, and it is why we
think the unbounded list is worth fixing alongside the race rather than after it.