Skip to content

Commit 9bcf994

Browse files
rafafrdzrafafrdz
authored
Join Feature (#33)
1 parent 143f021 commit 9bcf994

File tree

40 files changed

+887
-289
lines changed

40 files changed

+887
-289
lines changed

‎.github/workflows/ci.yml‎

Lines changed: 16 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -31,8 +31,22 @@ jobs:
3131
- name: Create header and run scalafmt
3232
run: sbt headerCreateAll scalafmtAll
3333

34-
- name: Build and Test
35-
run: sbt -v +test
34+
- name: Build and Test with Coverage
35+
run: sbt clean coverage test coverageReport coverageAggregate
36+
37+
- name: Upload coverage reports to Codecov
38+
uses: codecov/codecov-action@v5
39+
with:
40+
token: ${{ secrets.CODECOV_TOKEN }}
41+
slug: eff3ct0/supabase-auth-scala
42+
files: target/scala-*/scoverage-report/scoverage.xml
43+
fail_ci_if_error: true
44+
45+
- name: Upload test results
46+
uses: actions/upload-artifact@v3
47+
with:
48+
name: test-results
49+
path: target/test-reports
3650

3751
# Optional: This step uploads information to the GitHub dependency graph and unblocking Dependabot alerts for the repository
3852
#- name: Upload dependency graph

‎.github/workflows/release.yml‎

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -16,6 +16,24 @@ jobs:
1616
java-version: 11
1717
cache: sbt
1818
- uses: sbt/setup-sbt@v1
19+
20+
- name: Build and Test with Coverage
21+
run: sbt clean coverage test coverageReport coverageAggregate
22+
23+
- name: Upload coverage reports to Codecov
24+
uses: codecov/codecov-action@v5
25+
with:
26+
token: ${{ secrets.CODECOV_TOKEN }}
27+
slug: eff3ct0/supabase-auth-scala
28+
files: target/scala-*/scoverage-report/scoverage.xml
29+
fail_ci_if_error: true
30+
31+
- name: Upload test results
32+
uses: actions/upload-artifact@v3
33+
with:
34+
name: test-results
35+
path: target/test-reports
36+
1937
- run: sbt ci-release
2038
env:
2139
PGP_PASSPHRASE: ${{ secrets.PGP_PASSPHRASE }}

‎README.md‎

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,8 @@
11
# Teckel
22

3-
[![Release](https://github.com/rafafrdz/teckel/actions/workflows/release.yml/badge.svg?branch=master)](https://github.com/rafafrdz/teckel/actions/workflows/release.yml) [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
3+
[![Release](https://github.com/rafafrdz/teckel/actions/workflows/release.yml/badge.svg?branch=master)](https://github.com/rafafrdz/teckel/actions/workflows/release.yml)
4+
[![codecov](https://codecov.io/gh/eff3ct0/teckel/graph/badge.svg?token=24E1IZ0K2H)](https://codecov.io/gh/eff3ct0/teckel)
5+
[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
46

57
Teckel is a framework designed to simplify the creation of Apache Spark ETL (Extract, Transform,
68
Load) processes using YAML configuration files. This tool aims to standardize and streamline ETL workflow creation by
@@ -34,6 +36,7 @@ Here's an example of a fully defined ETL configuration using a YAML file:
3436
- `Where` Example: [here](./docs/etl/where.yaml)
3537
- `Group By` Example: [here](./docs/etl/group-by.yaml)
3638
- `Order By` Example: [here](./docs/etl/order-by.yaml)
39+
- `Join` Example: [here](./docs/etl/join.yaml)
3740

3841
## Getting Started
3942

‎api/src/test/resources/data/csv/example-2.csv‎

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
ID|Date|Symbol|Adj Close|Close|High|Low|Open|Volume
2+
1|2024-11-05|ZTS|175.27|175.27|176.80|172.25|174.55|2453800.0
3+
2|2024-11-06|ZTS|170.3699951171875|170.3699951171875|178.9199981689453|169.30999755859375|178.52000427246094|5362100.0
4+
3|2024-11-07|ZTS|174.25|174.25|174.7899932861328|169.63999938964844|172.58999633789062|3781300.0
5+
4|2024-11-08|ZTS|176.82000732421875|176.82000732421875|177.10000610351562|173.22000122070312|174.25|3243400.0
6+
5|2024-11-11|ZTS|176.14999389648438|176.14999389648438|178.3800048828125|175.0|176.92999267578125|3399500.0
7+
6|2024-11-12|ZTS|173.9600067138672|173.9600067138672|176.50999450683594|173.75|175.38999938964844|2704100.0
8+
7|2024-11-13|ZTS|177.0399932861328|177.0399932861328|177.5|174.91000366210938|175.32000732421875|2375300.0
9+
8|2024-11-14|ZTS|174.6300048828125|174.6300048828125|178.97000122070312|173.80999755859375|177.47999572753906|3009800.0
10+
9|2024-11-15|ZTS|175.13999938964844|175.13999938964844|177.07000732421875|170.75|173.0|3426500.0

‎api/src/test/resources/data/csv/example.csv‎

Lines changed: 29 additions & 29 deletions
Original file line numberDiff line numberDiff line change
@@ -1,29 +1,29 @@
1-
Date|Symbol|Adj Close|Close|High|Low|Open|Volume
2-
2024-11-05|ZTS|175.27|175.27|176.80|172.25|174.55|2453800.0
3-
2024-11-06|ZTS|170.3699951171875|170.3699951171875|178.9199981689453|169.30999755859375|178.52000427246094|5362100.0
4-
2024-11-07|ZTS|174.25|174.25|174.7899932861328|169.63999938964844|172.58999633789062|3781300.0
5-
2024-11-08|ZTS|176.82000732421875|176.82000732421875|177.10000610351562|173.22000122070312|174.25|3243400.0
6-
2024-11-11|ZTS|176.14999389648438|176.14999389648438|178.3800048828125|175.0|176.92999267578125|3399500.0
7-
2024-11-12|ZTS|173.9600067138672|173.9600067138672|176.50999450683594|173.75|175.38999938964844|2704100.0
8-
2024-11-13|ZTS|177.0399932861328|177.0399932861328|177.5|174.91000366210938|175.32000732421875|2375300.0
9-
2024-11-14|ZTS|174.6300048828125|174.6300048828125|178.97000122070312|173.80999755859375|177.47999572753906|3009800.0
10-
2024-11-15|ZTS|175.13999938964844|175.13999938964844|177.07000732421875|170.75|173.0|3426500.0
11-
2024-11-18|ZTS|176.4199981689453|176.4199981689453|177.1999969482422|173.66000366210938|174.1300048828125|3172900.0
12-
2024-11-19|ZTS|175.55999755859375|175.55999755859375|176.5|173.24000549316406|174.8800048828125|2208300.0
13-
2024-11-20|ZTS|175.6699981689453|175.6699981689453|177.41000366210938|173.8300018310547|176.4199981689453|2187300.0
14-
2024-11-21|ZTS|176.7100067138672|176.7100067138672|177.66000366210938|174.5500030517578|175.6999969482422|2019500.0
15-
2024-11-22|ZTS|176.9600067138672|176.9600067138672|178.07000732421875|176.27999877929688|176.35000610351562|1854600.0
16-
2024-11-25|ZTS|178.7100067138672|178.7100067138672|178.8000030517578|176.14999389648438|177.0|4558300.0
17-
2024-11-26|ZTS|175.6999969482422|175.6999969482422|178.64999389648438|174.83999633789062|178.4499969482422|2539600.0
18-
2024-11-27|ZTS|176.74000549316406|176.74000549316406|179.27000427246094|175.0|175.27000427246094|2315800.0
19-
2024-11-29|ZTS|175.25|175.25|177.80999755859375|175.24000549316406|176.92999267578125|1543400.0
20-
2024-12-02|ZTS|176.80999755859375|176.80999755859375|176.91000366210938|173.72999572753906|175.77999877929688|2391500.0
21-
2024-12-03|ZTS|176.94000244140625|176.94000244140625|181.39999389648438|176.55999755859375|176.7100067138672|2679000.0
22-
2024-12-04|ZTS|175.32000732421875|175.32000732421875|178.5|174.5399932861328|174.60000610351562|2687000.0
23-
2024-12-05|ZTS|174.77000427246094|174.77000427246094|176.52999877929688|173.72000122070312|175.27000427246094|2442000.0
24-
2024-12-06|ZTS|176.4600067138672|176.4600067138672|177.5500030517578|174.41000366210938|174.77000427246094|2551200.0
25-
2024-12-09|ZTS|178.14999389648438|178.14999389648438|179.77999877929688|175.0800018310547|175.8800048828125|2387300.0
26-
2024-12-10|ZTS|176.7100067138672|176.7100067138672|178.50999450683594|176.1999969482422|177.8800048828125|1678200.0
27-
2024-12-11|ZTS|177.1699981689453|177.1699981689453|178.3800048828125|175.80999755859375|176.25999450683594|1782400.0
28-
2024-12-12|ZTS|178.83999633789062|178.83999633789062|179.6999969482422|176.55999755859375|176.9499969482422|1936000.0
29-
2024-12-13|ZTS|178.17999267578125|178.17999267578125|181.85000610351562|176.6300048828125|178.97999572753906|1650300.0
1+
ID|Date|Symbol|Adj Close|Close|High|Low|Open|Volume
2+
1|2024-11-05|ZTS|175.27|175.27|176.80|172.25|174.55|2453800.0
3+
2|2024-11-06|ZTS|170.3699951171875|170.3699951171875|178.9199981689453|169.30999755859375|178.52000427246094|5362100.0
4+
3|2024-11-07|ZTS|174.25|174.25|174.7899932861328|169.63999938964844|172.58999633789062|3781300.0
5+
4|2024-11-08|ZTS|176.82000732421875|176.82000732421875|177.10000610351562|173.22000122070312|174.25|3243400.0
6+
5|2024-11-11|ZTS|176.14999389648438|176.14999389648438|178.3800048828125|175.0|176.92999267578125|3399500.0
7+
6|2024-11-12|ZTS|173.9600067138672|173.9600067138672|176.50999450683594|173.75|175.38999938964844|2704100.0
8+
7|2024-11-13|ZTS|177.0399932861328|177.0399932861328|177.5|174.91000366210938|175.32000732421875|2375300.0
9+
8|2024-11-14|ZTS|174.6300048828125|174.6300048828125|178.97000122070312|173.80999755859375|177.47999572753906|3009800.0
10+
9|2024-11-15|ZTS|175.13999938964844|175.13999938964844|177.07000732421875|170.75|173.0|3426500.0
11+
10|2024-11-18|ZTS|176.4199981689453|176.4199981689453|177.1999969482422|173.66000366210938|174.1300048828125|3172900.0
12+
11|2024-11-19|ZTS|175.55999755859375|175.55999755859375|176.5|173.24000549316406|174.8800048828125|2208300.0
13+
12|2024-11-20|ZTS|175.6699981689453|175.6699981689453|177.41000366210938|173.8300018310547|176.4199981689453|2187300.0
14+
13|2024-11-21|ZTS|176.7100067138672|176.7100067138672|177.66000366210938|174.5500030517578|175.6999969482422|2019500.0
15+
14|2024-11-22|ZTS|176.9600067138672|176.9600067138672|178.07000732421875|176.27999877929688|176.35000610351562|1854600.0
16+
15|2024-11-25|ZTS|178.7100067138672|178.7100067138672|178.8000030517578|176.14999389648438|177.0|4558300.0
17+
16|2024-11-26|ZTS|175.6999969482422|175.6999969482422|178.64999389648438|174.83999633789062|178.4499969482422|2539600.0
18+
17|2024-11-27|ZTS|176.74000549316406|176.74000549316406|179.27000427246094|175.0|175.27000427246094|2315800.0
19+
18|2024-11-29|ZTS|175.25|175.25|177.80999755859375|175.24000549316406|176.92999267578125|1543400.0
20+
19|2024-12-02|ZTS|176.80999755859375|176.80999755859375|176.91000366210938|173.72999572753906|175.77999877929688|2391500.0
21+
20|2024-12-03|ZTS|176.94000244140625|176.94000244140625|181.39999389648438|176.55999755859375|176.7100067138672|2679000.0
22+
21|2024-12-04|ZTS|175.32000732421875|175.32000732421875|178.5|174.5399932861328|174.60000610351562|2687000.0
23+
22|2024-12-05|ZTS|174.77000427246094|174.77000427246094|176.52999877929688|173.72000122070312|175.27000427246094|2442000.0
24+
23|2024-12-06|ZTS|176.4600067138672|176.4600067138672|177.5500030517578|174.41000366210938|174.77000427246094|2551200.0
25+
24|2024-12-09|ZTS|178.14999389648438|178.14999389648438|179.77999877929688|175.0800018310547|175.8800048828125|2387300.0
26+
25|2024-12-10|ZTS|176.7100067138672|176.7100067138672|178.50999450683594|176.1999969482422|177.8800048828125|1678200.0
27+
26|2024-12-11|ZTS|177.1699981689453|177.1699981689453|178.3800048828125|175.80999755859375|176.25999450683594|1782400.0
28+
27|2024-12-12|ZTS|178.83999633789062|178.83999633789062|179.6999969482422|176.55999755859375|176.9499969482422|1936000.0
29+
28|2024-12-13|ZTS|178.17999267578125|178.17999267578125|181.85000610351562|176.6300048828125|178.97999572753906|1650300.0

‎api/src/test/resources/etl/group-by.yaml‎

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,23 @@
1+
input:
2+
- name: table1
3+
format: csv
4+
path: 'src/test/resources/data/csv/example.csv'
5+
options:
6+
header: true
7+
sep: '|'
8+
9+
transformation:
10+
- name: groupByTable1
11+
group:
12+
from: table1
13+
by:
14+
- Symbol
15+
agg:
16+
- sum(`adj close`) as TotalClose
17+
- max(high) as Highest
18+
- min(low) as Lowest
19+
output:
20+
- name: groupByTable1
21+
format: parquet
22+
mode: overwrite
23+
path: 'src/test/resources/data/parquet/example/group-by'

‎api/src/test/resources/etl/join.yaml‎

Lines changed: 33 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,33 @@
1+
input:
2+
- name: table1
3+
format: csv
4+
path: 'src/test/resources/data/csv/example.csv'
5+
options:
6+
header: true
7+
sep: '|'
8+
9+
- name: table2
10+
format: csv
11+
path: 'src/test/resources/data/csv/example-2.csv'
12+
options:
13+
header: true
14+
sep: '|'
15+
16+
17+
transformation:
18+
19+
- name: joinTable1
20+
join:
21+
left: table1
22+
right:
23+
- name: table2
24+
type: inner
25+
on:
26+
- table1.id==table2.id
27+
- table1.id>175
28+
29+
output:
30+
- name: joinTable2
31+
format: parquet
32+
mode: overwrite
33+
path: 'src/test/resources/data/parquet/example/join'

‎api/src/test/resources/etl/order-by.yaml‎

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,22 @@
1+
input:
2+
- name: table1
3+
format: csv
4+
path: 'src/test/resources/data/csv/example.csv'
5+
options:
6+
header: true
7+
sep: '|'
8+
9+
transformation:
10+
- name: orderByTable1
11+
order:
12+
from: table1
13+
by:
14+
- id
15+
- date
16+
order: Desc
17+
18+
output:
19+
- name: orderByTable1
20+
format: parquet
21+
mode: overwrite
22+
path: 'src/test/resources/data/parquet/example/order-by'

‎api/src/test/resources/etl/select.yaml‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
input:
2+
- name: table1
3+
format: csv
4+
path: 'src/test/resources/data/csv/example.csv'
5+
options:
6+
header: true
7+
sep: '|'
8+
9+
transformation:
10+
- name: selectTable1
11+
select:
12+
from: table1
13+
columns:
14+
- id
15+
- date
16+
17+
output:
18+
- name: selectTable1
19+
format: parquet
20+
mode: overwrite
21+
path: 'src/test/resources/data/parquet/example/select'

‎api/src/test/resources/etl/simple.yaml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -11,4 +11,4 @@ output:
1111
- name: table1
1212
format: parquet
1313
mode: overwrite
14-
path: 'src/test/resources/data/parquet/example'
14+
path: 'src/test/resources/data/parquet/example/simple'

0 commit comments

Comments
 (0)