Skip to content

feat: report remaining egress quotas #583

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
akronim26 wants to merge 21 commits into
base: main
Choose a base branch
from
Open

feat: report remaining egress quotas #583

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
21 commits
Select commit Hold shift + click to select a range
cccc2e7
repot remaining regress quotas
akronim26 Jan 24, 2026
7a6e01c
fix lint
akronim26 Jan 26, 2026
c30b39c
Merge branch 'main' into regress-quotas
akronim26 Jan 26, 2026
e2515ff
revert back changes in the worker-configuration files
akronim26 Jan 27, 2026
2f28238
change header from X-* to FB-*
akronim26 Jan 27, 2026
fc9a4b5
allow negative values in response headers
akronim26 Jan 27, 2026
11e36f7
add tests
akronim26 Jan 27, 2026
7193801
added support for too many requests
akronim26 Jan 27, 2026
de090e0
adjust tests
akronim26 Jan 28, 2026
200c7f7
Merge branch 'main' into regress-quotas
akronim26 Jan 28, 2026
91a488e
remove chnages in worker-configuration.d.ts
akronim26 Jan 28, 2026
a0ebbe4
fix changes
akronim26 Jan 28, 2026
760fca9
Revert "remove chnages in worker-configuration.d.ts"
akronim26 Jan 28, 2026
3bbe611
remove the changes from *.d.ts
akronim26 Jan 28, 2026
d288566
Merge branch 'main' into regress-quotas
akronim26 Jan 28, 2026
7ac7e62
fetch egress at the start of the response
akronim26 Jan 29, 2026
5e16c99
Merge branch 'main' into regress-quotas
akronim26 Jan 29, 2026
7ff29ee
commit requested changes
akronim26 Jan 29, 2026
e9ea5b6
chore: make requested changes
akronim26 Feb 6, 2026
73a1452
packed setting of headers into helper fn
akronim26 Feb 8, 2026
9c6307c
remove unnecessary fileds from test
akronim26 Feb 10, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3,883 changes: 3,184 additions & 699 deletions bad-bits/worker-configuration.d.ts
View file
Open in desktop

Large diffs are not rendered by default.

3,904 changes: 3,180 additions & 724 deletions indexer/worker-configuration.d.ts
View file
Open in desktop

Large diffs are not rendered by default.

62 changes: 36 additions & 26 deletions payment-settler/worker-configuration.d.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,20 +1,19 @@
/* eslint-disable */
// Generated by Wrangler by running `wrangler types --env dev` (hash: d91ee19fc7369076a2f49d54170f737b)
// Runtime types generated with workerd@1.20251213.0 2025-09-08 nodejs_compat
// Generated by Wrangler by running `wrangler types --env dev` (hash: 37113b22e6f1035329f4604b66234abe)
// Runtime types generated with workerd@1.20260120.0 2025-09-08 nodejs_compat
declare namespace Cloudflare {
interface GlobalProps {
mainModule: typeof import("./bin/payment-settler");
}
interface Env {
ENVIRONMENT: "dev" | "calibration" | "mainnet";
RPC_URL: "https://api.calibration.node.glif.io/" | "https://api.node.glif.io/";
FILBEAM_OPERATOR_CONTRACT_ADDRESS: "0x0000000000000000000000000000000000000000" | "0x5991E4F9fcEF4AE23959eE03638B4688A7e1EcfF" | "0x9E90749D298C4ca43Bb468CA859Dfe167F9CdCf2";
DEFAULT_LOCKUP_PERIOD_DAYS: 30;
FILECOIN_GENESIS_BLOCK_TIMESTAMP_MS: 1667326380000 | 1598306400000;
SETTLEMENT_BATCH_SIZE: 10 | 150;
FILBEAM_OPERATOR_PAYMENT_SETTLER_PRIVATE_KEY: string;
DB: D1Database;
TRANSACTION_QUEUE: Queue;
ENVIRONMENT: "dev";
RPC_URL: "https://api.calibration.node.glif.io/";
FILBEAM_OPERATOR_CONTRACT_ADDRESS: "0x0000000000000000000000000000000000000000";
DEFAULT_LOCKUP_PERIOD_DAYS: 30;
FILECOIN_GENESIS_BLOCK_TIMESTAMP_MS: 1667326380000;
SETTLEMENT_BATCH_SIZE: 10;
TRANSACTION_MONITOR_WORKFLOW: Workflow<Parameters<import("./bin/payment-settler").TransactionMonitorWorkflow['run']>[0]['payload']>;
}
}
Expand All @@ -23,7 +22,7 @@ type StringifyValues<EnvType extends Record<string, unknown>> = {
[Binding in keyof EnvType]: EnvType[Binding] extends string ? EnvType[Binding] : string;
};
declare namespace NodeJS {
interface ProcessEnv extends StringifyValues<Pick<Cloudflare.Env, "ENVIRONMENT" | "RPC_URL" | "FILBEAM_OPERATOR_CONTRACT_ADDRESS" | "DEFAULT_LOCKUP_PERIOD_DAYS" | "FILECOIN_GENESIS_BLOCK_TIMESTAMP_MS" | "SETTLEMENT_BATCH_SIZE" | "FILBEAM_OPERATOR_PAYMENT_SETTLER_PRIVATE_KEY">> {}
interface ProcessEnv extends StringifyValues<Pick<Cloudflare.Env, "ENVIRONMENT" | "RPC_URL" | "FILBEAM_OPERATOR_CONTRACT_ADDRESS" | "DEFAULT_LOCKUP_PERIOD_DAYS" | "FILECOIN_GENESIS_BLOCK_TIMESTAMP_MS" | "SETTLEMENT_BATCH_SIZE">> {}
}

// Begin runtime types
Expand Down Expand Up @@ -511,8 +510,10 @@ interface DurableObjectNamespaceNewUniqueIdOptions {
jurisdiction?: DurableObjectJurisdiction;
}
type DurableObjectLocationHint = "wnam" | "enam" | "sam" | "weur" | "eeur" | "apac" | "oc" | "afr" | "me";
type DurableObjectRoutingMode = "primary-only";
interface DurableObjectNamespaceGetDurableObjectOptions {
locationHint?: DurableObjectLocationHint;
routingMode?: DurableObjectRoutingMode;
}
interface DurableObjectClass<_T extends Rpc.DurableObjectBranded | undefined = undefined> {
}
Expand Down Expand Up @@ -2096,6 +2097,8 @@ interface Transformer<I = any, O = any> {
expectedLength?: number;
}
interface StreamPipeOptions {
preventAbort?: boolean;
preventCancel?: boolean;
/**
* Pipes this readable stream to a given writable stream destination. The way in which the piping process behaves under various error conditions can be customized with a number of passed options. It returns a promise that fulfills when the piping process completes successfully, or rejects if any errors were encountered.
*
Expand All @@ -2114,8 +2117,6 @@ interface StreamPipeOptions {
* The signal option can be set to an AbortSignal to allow aborting an ongoing pipe operation via the corresponding AbortController. In this case, this source readable stream will be canceled, and destination aborted, unless the respective options preventCancel or preventAbort are set.
*/
preventClose?: boolean;
preventAbort?: boolean;
preventCancel?: boolean;
signal?: AbortSignal;
}
type ReadableStreamReadResult<R = any> = {
Expand Down Expand Up @@ -2390,13 +2391,13 @@ declare abstract class TransformStreamDefaultController<O = any> {
terminate(): void;
}
interface ReadableWritablePair<R = any, W = any> {
readable: ReadableStream<R>;
/**
* Provides a convenient, chainable way of piping this readable stream through a transform stream (or any other { writable, readable } pair). It simply pipes the stream into the writable side of the supplied pair, and returns the readable side for further use.
*
* Piping a stream will lock it for the duration of the pipe, preventing any other consumer from acquiring a reader.
*/
writable: WritableStream<W>;
readable: ReadableStream<R>;
}
/**
* The **`WritableStream`** interface of the Streams API provides a standard abstraction for writing streaming data to a destination, known as a sink.
Expand Down Expand Up @@ -5515,7 +5516,7 @@ interface Ai_Cf_Qwen_Qwq_32B_Messages {
};
})[];
/**
* JSON schema that should be fufilled for the response.
* JSON schema that should be fulfilled for the response.
*/
guided_json?: object;
/**
Expand Down Expand Up @@ -5781,7 +5782,7 @@ interface Ai_Cf_Mistralai_Mistral_Small_3_1_24B_Instruct_Messages {
};
})[];
/**
* JSON schema that should be fufilled for the response.
* JSON schema that should be fulfilled for the response.
*/
guided_json?: object;
/**
Expand Down Expand Up @@ -5872,7 +5873,7 @@ interface Ai_Cf_Google_Gemma_3_12B_It_Prompt {
*/
prompt: string;
/**
* JSON schema that should be fufilled for the response.
* JSON schema that should be fulfilled for the response.
*/
guided_json?: object;
/**
Expand Down Expand Up @@ -6031,7 +6032,7 @@ interface Ai_Cf_Google_Gemma_3_12B_It_Messages {
};
})[];
/**
* JSON schema that should be fufilled for the response.
* JSON schema that should be fulfilled for the response.
*/
guided_json?: object;
/**
Expand Down Expand Up @@ -6303,7 +6304,7 @@ interface Ai_Cf_Meta_Llama_4_Scout_17B_16E_Instruct_Messages {
})[];
response_format?: Ai_Cf_Meta_Llama_4_Scout_17B_16E_Instruct_JSON_Mode;
/**
* JSON schema that should be fufilled for the response.
* JSON schema that should be fulfilled for the response.
*/
guided_json?: object;
/**
Expand Down Expand Up @@ -6533,7 +6534,7 @@ interface Ai_Cf_Meta_Llama_4_Scout_17B_16E_Instruct_Messages_Inner {
})[];
response_format?: Ai_Cf_Meta_Llama_4_Scout_17B_16E_Instruct_JSON_Mode;
/**
* JSON schema that should be fufilled for the response.
* JSON schema that should be fulfilled for the response.
*/
guided_json?: object;
/**
Expand Down Expand Up @@ -7583,7 +7584,7 @@ interface Ai_Cf_Ai4Bharat_Indictrans2_En_Indic_1B_Input {
*/
text: string | string[];
/**
* Target langauge to translate to
* Target language to translate to
*/
target_language: "asm_Beng" | "awa_Deva" | "ben_Beng" | "bho_Deva" | "brx_Deva" | "doi_Deva" | "eng_Latn" | "gom_Deva" | "gon_Deva" | "guj_Gujr" | "hin_Deva" | "hne_Deva" | "kan_Knda" | "kas_Arab" | "kas_Deva" | "kha_Latn" | "lus_Latn" | "mag_Deva" | "mai_Deva" | "mal_Mlym" | "mar_Deva" | "mni_Beng" | "mni_Mtei" | "npi_Deva" | "ory_Orya" | "pan_Guru" | "san_Deva" | "sat_Olck" | "snd_Arab" | "snd_Deva" | "tam_Taml" | "tel_Telu" | "urd_Arab" | "unr_Deva";
}
Expand Down Expand Up @@ -9531,6 +9532,15 @@ declare abstract class D1PreparedStatement {
// ignored when `Disposable` is included in the standard lib.
interface Disposable {
}
/**
* The returned data after sending an email
*/
interface EmailSendResult {
/**
* The Email Message ID
*/
messageId: string;
}
/**
* An email message that can be sent from a Worker.
*/
Expand Down Expand Up @@ -9572,19 +9582,19 @@ interface ForwardableEmailMessage extends EmailMessage {
* @param headers A [Headers object](https://developer.mozilla.org/en-US/docs/Web/API/Headers).
* @returns A promise that resolves when the email message is forwarded.
*/
forward(rcptTo: string, headers?: Headers): Promise<void>;
forward(rcptTo: string, headers?: Headers): Promise<EmailSendResult>;
/**
* Reply to the sender of this email message with a new EmailMessage object.
* @param message The reply message.
* @returns A promise that resolves when the email message is replied.
*/
reply(message: EmailMessage): Promise<void>;
reply(message: EmailMessage): Promise<EmailSendResult>;
}
/**
* A binding that allows a Worker to send email messages.
*/
interface SendEmail {
send(message: EmailMessage): Promise<void>;
send(message: EmailMessage): Promise<EmailSendResult>;
}
declare abstract class EmailEvent extends ExtendableEvent {
readonly message: ForwardableEmailMessage;
Expand Down Expand Up @@ -9617,7 +9627,7 @@ interface Hyperdrive {
/**
* Connect directly to Hyperdrive as if it's your database, returning a TCP socket.
*
* Calling this method returns an idential socket to if you call
* Calling this method returns an identical socket to if you call
* `connect("host:port")` using the `host` and `port` fields from this object.
* Pick whichever approach works better with your preferred DB client library.
*
Expand Down Expand Up @@ -9950,7 +9960,7 @@ declare module "cloudflare:pipelines" {
protected ctx: ExecutionContext;
constructor(ctx: ExecutionContext, env: Env);
/**
* run recieves an array of PipelineRecord which can be
* run receives an array of PipelineRecord which can be
* transformed and returned to the pipeline
* @param records Incoming records from the pipeline to be transformed
* @param metadata Information about the specific pipeline calling the transformation entrypoint
Expand Down Expand Up @@ -10454,7 +10464,7 @@ declare namespace TailStream {
// For Hibernate and Mark this would be the span under which they were emitted.
// spanId is not set ONLY if:
// 1. This is an Onset event
// 2. We are not inherting any SpanContext. (e.g. this is a cross-account service binding or a new top-level invocation)
// 2. We are not inheriting any SpanContext. (e.g. this is a cross-account service binding or a new top-level invocation)
readonly spanId?: string;
}
interface TailEvent<Event extends EventType> {
Expand Down
42 changes: 42 additions & 0 deletions piece-retriever/bin/piece-retriever.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -202,6 +202,27 @@ export default {
'Cache-Control',
`public, max-age=${env.CLIENT_CACHE_TTL}`,
)
try {
Comment thread
akronim26 marked this conversation as resolved.
Outdated
const contentLengthHeader =
retrievalResult.response.headers.get('content-length')
const estimatedEgress = contentLengthHeader
? Number.parseInt(contentLengthHeader, 10) || 0
: 0
const remainingCdn =
retrievalCandidate.cdnEgressQuota - BigInt(estimatedEgress)
const remainingCacheMiss =
retrievalCandidate.cacheMissEgressQuota - BigInt(estimatedEgress)
Copy link
Copy Markdown
Contributor

@bajtos bajtos Jan 27, 2026 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have mixed feelings about relying on content-length. It's not guaranteed that the response will have this header. I am concerned that the remaining egress reported by FilBeam will be unpredictable.

Also, there may be other requests in progress that will further reduce the remaining egress quotas - I think it's impossible to give the client a precise value.

I propose simplifying the report and reporting the remaining egress quota as it was at the start of the response, at the time sent back the headers.

@pyropy @juliangruber thoughts?

Copy link
Copy Markdown
Contributor

@pyropy pyropy Jan 27, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have mixed feelings about relying on content-length. It's not guaranteed that the response will have this header. I am concerned that the remaining egress reported by FilBeam will be unpredictable.

@bajtos You once mentioned that content length is encoded in CID V2, would it be possible to use that instead of the content-length headers?

Also, there may be other requests in progress that will further reduce the remaining egress quotas - I think it's impossible to give the client a precise value.

That's true, we should document that this may report an imprecise amount back to the user.

I propose simplifying the report and reporting the remaining egress quota as it was at the start of the response, at the time sent back the headers.

That might be a good start. I think these headers might be used to raise alerts and trigger top-ups (either manual or automated) and not for precise accounting so that could work.

Copy link
Copy Markdown
Contributor

@bajtos bajtos Jan 27, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have mixed feelings about relying on content-length. It's not guaranteed that the response will have this header. I am concerned that the remaining egress reported by FilBeam will be unpredictable.

@bajtos You once mentioned that content length is encoded in CID V2, would it be possible to use that instead of the content-length headers?

Yes, we can extract the piece size from the CID, but this will not work for range requests (e.g. when downloading a 1MB chunk of a video file, as browsers typically do).

Copy link
Copy Markdown
Member

@juliangruber juliangruber Jan 29, 2026 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@bajtos +1 to simplifying the report, first get it simple and correct, we can improve it later

response.headers.set(
'X-Cdn-Egress-Remaining',
Comment thread
akronim26 marked this conversation as resolved.
Outdated
String(remainingCdn < 0n ? 0n : remainingCdn),
)
response.headers.set(
'X-Cache-Miss-Egress-Remaining',
String(remainingCacheMiss < 0n ? 0n : remainingCacheMiss),
)
Comment thread
akronim26 marked this conversation as resolved.
Outdated
} catch (e) {
console.warn('Failed to compute egress remaining headers', e)
}
return response
}

Expand Down Expand Up @@ -288,6 +309,27 @@ export default {
'Cache-Control',
`public, max-age=${env.CLIENT_CACHE_TTL}`,
)
try {
Comment thread
akronim26 marked this conversation as resolved.
Outdated
const contentLengthHeader =
retrievalResult.response.headers.get('content-length')
const estimatedEgress = contentLengthHeader
? Number.parseInt(contentLengthHeader, 10) || 0
: 0
const remainingCdn =
retrievalCandidate.cdnEgressQuota - BigInt(estimatedEgress)
const remainingCacheMiss =
retrievalCandidate.cacheMissEgressQuota - BigInt(estimatedEgress)
response.headers.set(
'X-Cdn-Egress-Remaining',
String(remainingCdn < 0n ? 0n : remainingCdn),
)
response.headers.set(
'X-Cache-Miss-Egress-Remaining',
String(remainingCacheMiss < 0n ? 0n : remainingCacheMiss),
)
} catch (e) {
console.warn('Failed to compute egress remaining headers', e)
}
return response
} catch (error) {
const { status } = getErrorHttpStatusMessage(error)
Expand Down
Loading
Loading