Skip to content

[Security] Quota consumption TOCTOU race — user/token quota can be oversold via concurrent requests #2440

Description

@minyanyi

[Security] Quota consumption TOCTOU race — user/token quota can be oversold via concurrent requests

Summary

The quota consumption path has a time-of-check-to-time-of-use (TOCTOU) race condition. The
balance sufficiency check and the actual deduction are not atomic, and every deduction is an
unconditional UPDATE ... SET quota = quota - ? with no WHERE quota >= ? guard, no row
lock, and no transaction
. By issuing concurrent requests, a user holding a valid API token
can consume more quota than they actually have — quota is oversold and can be driven negative.

For a per-token billing gateway this is direct monetary loss for the operator. It is the same
vulnerability class as the billing race found in the downstream fork new-api (QuantumNous/new-api#6609).

Verified facts (code paths, one-api @ 8df4a26)

1. The check and the deduction are separated in time.

relay/controller/helper.go:68 preConsumeQuota:

  • helper.go:71 — reads the current user quota once via model.CacheGetUserQuota.
  • helper.go:76 — sufficiency check userQuota - preConsumedQuota < 0. This is the TOCTOU check.
  • helper.go:84-87 — if userQuota > 100*preConsumedQuota, preConsumedQuota is set to 0
    ("trusted, no need to pre-consume") and token-level pre-consumption is skipped entirely.
    In this common path the only deduction happens at the very end of the request.
  • The real deduction of the user balance only occurs after the upstream LLM call finishes, in
    relay/billing/billing.go:23 PostConsumeQuotamodel/token.go:282 PostConsumeTokenQuota.
    For streaming completions this window spans seconds to minutes.

2. Without Redis (default), nothing atomically decrements the user balance during the request.

model/cache.go:119-125:

func CacheDecreaseUserQuota(id int, quota int64) error {
	if !common.RedisEnabled {
		return nil   // <-- no-op when Redis is not configured
	}
	err := common.RedisDecrease(fmt.Sprintf("user_quota:%d", id), int64(quota))
	return err
}

When RedisIsEnabled is false (Redis is optional; REDIS_CONN_STRING unset by default), the
pre-consumption decrement is silently skipped. The only user-balance gate is the single stale
read at helper.go:71, and the balance is only touched at request end.

3. Every deduction is unconditional — there is no WHERE quota >= ? guard.

  • model/user.go:401-404 decreaseUserQuota:
    DB.Model(&User{}).Where("id = ?", id).Update("quota", gorm.Expr("quota - ?", quota))
  • model/token.go:206-215 decreaseTokenQuota:
    DB.Model(&Token{}).Where("id = ?", id).Updates(map{ "remain_quota": gorm.Expr("remain_quota - ?", quota), ... })

Neither checks the resulting balance, so quota goes negative under concurrent deduction.

4. No concurrency control exists anywhere in the quota path.

A repo-wide search of the quota path (model/token.go, model/user.go, model/cache.go,
relay/billing/billing.go, relay/controller/helper.go) finds zero occurrences of
FOR UPDATE / Locking / ForUpdate / .Transaction / sync.Mutex. The check→deduct is
never serialized.

5. BATCH_UPDATE_ENABLED amplifies the race.

main.go:86 enables config.BatchUpdateEnabled. With it, model/utils.go:38-78 aggregates
deductions in an in-memory map and flushes to the DB only every BATCH_UPDATE_INTERVAL seconds
(default 5, common/config/config.go:113). During that window the DB quota column is stale,
so the helper.go:71 / token.go:225 checks read old values while deductions are pending in
memory — the oversell window grows from "request duration" to "request duration + flush interval".

Reproduction sketch

With a user holding quota Q and a valid token, fire N concurrent /v1/chat/completions
requests each costing ~Q (or simply more concurrent requests than the balance covers). All N
read the same stale balance, all pass the sufficiency check, and all are served. Final balance
is negative (or the operator pays upstream for N - floor(Q/cost) requests that were never
billable). Enabling BATCH_UPDATE_ENABLED=true widens the window and makes reproduction easier.

# N concurrent requests against one user's token; each passes the stale balance check
seq 1 20 | xargs -P 20 -I{} curl -s https://TARGET/v1/chat/completions \
  -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" \
  -d '{"model":"...","messages":[{"role":"user","content":"..."}]}'

Impact

  • Financial: quota oversell — the gateway pays upstream providers for usage it cannot bill.
    Direct loss for every multi-tenant one-api deployment.
  • Data integrity: users.quota / tokens.remain_quota go negative; accounting is corrupted.
  • Availability: an attacker can drain the operator's upstream budget, denying service to
    legitimate users.

Prerequisite: a registered account with a self-issued token — i.e. any tenant of a multi-user
one-api instance. This is the normal threat model of a billing gateway.

Suggested fix

Make check-and-deduct atomic. Minimal change — guard the deduction and verify affected rows:

// model/user.go — decreaseUserQuota
res := DB.Model(&User{}).
    Where("id = ? AND quota >= ?", id, quota).
    Update("quota", gorm.Expr("quota - ?", quota))
if res.RowsAffected == 0 {
    return errors.New("quota insufficient")
}

Apply the same WHERE remain_quota >= ? + RowsAffected check to decreaseTokenQuota, and
wrap the user+token pair in a single DB.Transaction. Alternatively SELECT ... FOR UPDATE
on the user/token row inside a transaction. The pre-consume/post-consume accounting should be
reconciled against the same guarded deduction so the "trusted" path (helper.go:84) cannot
bypass it.

References

Environment

  • one-api commit 8df4a2670b98266bd287c698243fff327d9748cf (main, 2025-02-21)
  • Default config: BatchUpdateEnabled=false, PreConsumedQuota=500, Redis disabled.
  • Source-level verification (no third-party deployment was tested).

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions