Skip to content

[Lambda: Tracking AWS CE] Generate running data from AWS CE #6682

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 30 commits into from
Jun 4, 2025
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
30 commits
Select commit Hold shift + click to select a range
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
34 changes: 34 additions & 0 deletions .github/workflows/lambda-do-release-runners.yml
Original file line number Diff line number Diff line change
Expand Up @@ -146,6 +146,40 @@ jobs:
tag: ${{ inputs.tag }}
updateOnlyUnreleased: true

release-oss-ci-cur:
name: Upload Release for oss-ci-cur lambda
runs-on: ubuntu-latest
permissions:
contents: write
env:
REF: ${{ inputs.tag }}
steps:
- name: Checkout code
uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
with:
ref: ${{ inputs.tag }}

- uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5.6.0
with:
python-version: '3.10'

- name: Build deployment.zip
working-directory: aws/lambda/oss_ci_cur
run: make deployment.zip

- name: Copy deployment.zip to root
run: cp aws/lambda/oss_ci_cur/deployment.zip oss-ci-cur.zip

- uses: ncipollo/release-action@440c8c1cb0ed28b9f43e4d1d670870f059653174 # v1.16.0
with:
artifacts: "oss-ci-cur.zip"
allowUpdates: true
draft: true
name: ${{ inputs.tag }}
tag: ${{ inputs.tag }}
updateOnlyUnreleased: true


finish-release:
needs:
- release-lambdas
Expand Down
3 changes: 3 additions & 0 deletions aws/lambda/oss_ci_cur/.gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
*.zip
deployment/
venv/
19 changes: 19 additions & 0 deletions aws/lambda/oss_ci_cur/Makefile
Original file line number Diff line number Diff line change
@@ -0,0 +1,19 @@
all: run-local

clean:
rm -rf deployment
rm -rf venv
rm -rf deployment.zip

venv/bin/python:
virtualenv venv
venv/bin/pip install -r requirements.txt

deployment.zip:
mkdir -p deployment
cp lambda_function.py ./deployment/.
pip3.10 install -r requirements.txt -t ./deployment/. --platform manylinux2014_x86_64 --only-binary=:all: --implementation cp --python-version 3.10 --upgrade
cd ./deployment && zip -q -r ../deployment.zip .

.PHONY: create-deployment-package
create-deployment-package: deployment.zip
18 changes: 18 additions & 0 deletions aws/lambda/oss_ci_cur/Readme.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
# AWS Cost Explorer Data Processing
This Python script is designed to fetch and process AWS Cost Explorer data, specifically focusing on EC2 usage with Daily granularity.

This Lambda function runs daily and retrieves AWS Cost Explorer data for the date two days prior due to the delay in AWS Cost Explorer data availability.

## Usage
### Local Execution
To run the script locally, use the following command:

```bash
python lambda_function.py --clickhouse-endpoint <endpoint> --clickhouse-username <username> --clickhouse-password <password> [--not-dry-run] [--start-time <start_time>] [--end-time <end_time>]
```
* `--clickhouse-endpoint`: The ClickHouse endpoint URL.
* `--clickhouse-username`: The ClickHouse username.
* `--clickhouse-password`: The ClickHouse password.
* `--not-dry-run`: Optional. If set, the script will write results to the database.
* `--start-time`: Optional. Start time in UTC ISO8601 format (e.g., 2025-05-28T00:00:00Z). Otherwise, the script will use the current time minus 2 day.
* `--end-time`: Optional. End time in UTC ISO8601 format (e.g., 2025-05-29T00:00:00Z). Otherwise, the script will use the current time minus 1 day.
Loading