Skip to content

Feature/rally features main merged #71

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Draft
wants to merge 77 commits into
base: main
Choose a base branch
from
Draft

Feature/rally features main merged #71

wants to merge 77 commits into from

Conversation

susanxiong
Copy link

creating this as a draft in hopes that one day rally can be pointed to the real release versions of this package...

this needs to be:

  • tested in rally
  • ideally refactored to be generic enough to be applicable to other use cases besides rally
  • and then tested in rally again :)

@jayckaiser
Move disk_to_s3 and some Slack callbacks from Rally; overload xcom_pu…
8754e17 
…ll_template to accept either a string or Operator.
@jayckaiser
Move Sharefile Airflow objects from Rally.
6a5460c
@jayckaiser
Whitespace cleanup in s3_to_postgres.py.
3957079
@jayckaiser
Refactor project to move Python callables into their own location; ad…
6c84ccf 
…d deprecation warnings when referenced from the old path.

TODO: Test this.
@jayckaiser
Move JSONL and snake_case functionality from Rally.
0963da6
@jayckaiser
Rename callables.airflow to callables.variable to prevent a name-coll…
60adfd5 
…ision; migrate gsheets logic from Rally.
@jayckaiser
Migrate S3-to-Snowflake from Rally.
f5c86e2
@jayckaiser
Update google hook import to current iteration (untested).
2bf9306
@jayckaiser
Clean up code in callables.snowflake.
114a951
@jayckaiser
Add type-hints to s3 callables where missing; minor code clean-up.
1608081
@jayckaiser
Move slack callbacks into callables subdirectory.
7c0e92d
@jayckaiser
Force line endings to LF.
6552c88
@jayckaiser
Overload Slack alert typing to accept exception strings or exceptions.
c71e2b6
@jayckaiser
Add FTP and ZIP util callables (TODO: test).
4e7dad7
@jayckaiser
Add Airflow and SQL helpers.
9cf28ca
@jayckaiser
Fix import errors.
56ebab0
@jayckaiser
Refactor project to alias original submodule pathing to new location …
133dfb2 
…in sys init (very much untested).
@jayckaiser
Fix revised xcom_pull_template logic to check the subclass of task_ids.
f8063cc
@jayckaiser
Add custom EA SFTP Hook to project.
5bbff9c
@jayckaiser
Minor cleanup before first attempted run.
5f3588d
@jayckaiser
Move DBT run-operation operator into providers submodule.
cc5ac4c
@jayckaiser
Move s3 Loop operator into providers submodule.
6cb4905
@jayckaiser
Move SSM helper class into new callables file.
3d3c1ce
@jayckaiser
Use lazy approach for rerouting deprecated imports (untested).
718f127
@jayckaiser
Fix DBT operator import.
7e53d40
@jayckaiser
Fix S3-to-Snowflake DAG import error.
2290cd5
@jayckaiser
Fix gspread type errors.
e566242
@jayckaiser
Fix gspread type errors (missed one).
c7fa7ba
@jayckaiser
Attempt refactor of deprecated imports mapping.
8c7c92a
@jayckaiser
Fix bug in path-renames.
d87af8d
jayckaiser and others added 29 commits January 23, 2024 12:52
@jayckaiser
Fix bug in DBT Run Operation Operator.
10fb4bc
@jayckaiser
Make DbtRunOperationOperator arguments a dictionary.
d89e8da
@jayckaiser
Add cleanup and TODOs across project.
a60e1d0
@jayckaiser
Fix f-string bug.
77f002b
@jayckaiser
Update credentials-extraction logic to use new built-ins.
faca28f
@jayckaiser
Revert code to try to circumvent permissions error.
728c6e5
@jayckaiser
Revert to non-public _get_field() method.
059d371
@jayckaiser
Fix bug in argument name when calling Slack insert failure alert.
ba2e69e
@jayckaiser
Migrate new DBT changes from feature/rally_features to be merged into…
bebfb00 
… main.
@jayckaiser
Add missing init file.
136eef0
@jayckaiser
Rename csv_path to local_path in jsonl helper.
23f222b
@jayckaiser
Increment version to 0.3.0.
16e5c3f
@jayckaiser
Update imports to use new callables subfolder; add callables.
a14bc52
@jayckaiser
Add Sharefile provider; add callables for gsheets, s3, snowflake, sna…
eef3ea8 
…ke_case, and jsonl; add xcom util in Airflow callable.
@jayckaiser
Merge all updates from Rally testing into original rally_features bra…
a0813d4 
…nch.
@jayckaiser
Re-remove deprecated dag_util files.
4873124
@jayckaiser
Update CHANGELOG.md (still incomplete)
46d511e
@jayckaiser
Add missing package calls in init.
ec6fa86
@jayckaiser
Fix callable routing in new-old DAG.
0661f4c
@susanxiong
Allow deletion with specified where clause
813388c
@susanxiong
Pass in a list for sum
e04aac1
@susanxiong
Fix sum logic to use 0 if None
5e87c29
@susanxiong
Merge pull request #62 from edanalytics/feature/rally_features__snowf…
f3855db 
…lake_delete

Feature/rally features  snowflake delete
@susanxiong
Update to main branch version of run_dbt dag and ea_custom_dag
ded21d5
@susanxiong
Merge branch 'main' into feature/rally_features_main_merged
b6ddecd
@susanxiong
Delete snake_case.py (unused)
58d307b
@susanxiong
Remove references to slack_conn_id
e3fd72c
@susanxiong
Replace references to source_org with tenant/tenant_code
8880bad
@susanxiong
Sync files with main
76eca24
@susanxiong susanxiong self-assigned this Apr 25, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees

@susanxiong susanxiong

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@susanxiong @jayckaiser