[Lambda: Tracking AWS CE] Generate running data from AWS CE

[yangw-dev]

1. setup the lambda to track aws ce runner duration
2. set up library for clickhouse client for unit test

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Skipped Deployment

Name	Status	Preview	Updated (UTC)
torchci	⬜️ Ignored (Inspect)	Visit Preview	Jun 2, 2025 7:20pm

[huydhn]

@jeanschmidt probably won't be happen see the lambda here. Do we have a guide or code pointer on how to create and manage lambda on Terraform? I guess we could take the new scale-up/down chron lambdas as an example, so maybe @ZainRizvi or @Camyll would know

[yangw-dev]

@jeanschmidt probably won't be happen see the lambda here. Do we have a guide or code pointer on how to create and manage lambda on Terraform? I guess we could take the new scale-up/down chron lambdas as an example, so maybe @ZainRizvi or @Camyll would know

this is funny, I created the guidance without seeing this haha.

I think last time I synced with jean, he mentioned he wants to do in most cases, the code base for lambda stays in test-infra, and the resources TF in pytorch-labs/GH. Would be curious if there is an exception, for instance, some sensitive lambda etc.

[wdvr]

Can we rename everywhere ce to costexplorer? I think something like oss_ci_aws_ce is not super clear (same for the clickhouse table)

[ZainRizvi]

Small request: Can we find a better name for AWS CE? It's such an obscure term :). Ideally something that reflects it's purpose and doesn't assume it must be backed by AWS (spitballing: "billable-units"?)

[zxiiro]

Small request: Can we find a better name for AWS CE? It's such an obscure term :). Ideally something that reflects it's purpose and doesn't assume it must be backed by AWS (spitballing: "billable-units"?)

Although the code here literally does track with AWS CE so it is descriptive. As I learn more about FinOps though apparently the industry standard term for this is CUR data (Cost and Usage Reporting).

[yangw-dev]

I will rename it to CUR!

[zxiiro]

I will rename it to CUR!

@yangw-dev I don't know how deep into the rabbit hole you want to get but just in case you're interested. There's a standard specification for this kind of data too. https://focus.finops.org