-
Notifications
You must be signed in to change notification settings - Fork 57
[b/390590500] Airflow dates params #766
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
...va/com/google/edwmigration/dumper/application/dumper/connector/airflow/AirflowConnector.java
Outdated
Show resolved
Hide resolved
...va/com/google/edwmigration/dumper/application/dumper/connector/airflow/AirflowConnector.java
Outdated
Show resolved
Hide resolved
...va/com/google/edwmigration/dumper/application/dumper/connector/airflow/AirflowConnector.java
Outdated
Show resolved
Hide resolved
...va/com/google/edwmigration/dumper/application/dumper/connector/airflow/AirflowConnector.java
Outdated
Show resolved
Hide resolved
...va/com/google/edwmigration/dumper/application/dumper/connector/airflow/AirflowConnector.java
Outdated
Show resolved
Hide resolved
...va/com/google/edwmigration/dumper/application/dumper/connector/airflow/AirflowConnector.java
Outdated
Show resolved
Hide resolved
.../app/src/main/java/com/google/edwmigration/dumper/application/dumper/ConnectorArguments.java
Show resolved
Hide resolved
...va/com/google/edwmigration/dumper/application/dumper/connector/airflow/AirflowConnector.java
Show resolved
Hide resolved
...va/com/google/edwmigration/dumper/application/dumper/connector/airflow/AirflowConnector.java
Outdated
Show resolved
Hide resolved
...va/com/google/edwmigration/dumper/application/dumper/connector/airflow/AirflowConnector.java
Outdated
Show resolved
Hide resolved
...va/com/google/edwmigration/dumper/application/dumper/connector/airflow/AirflowConnector.java
Outdated
Show resolved
Hide resolved
-start-date=x --lookback-days=y -> date range: [x-y, x) This usage is not intuitive, at least for me. Usually when I provide start date I want all the data to be later than this. I think |
.../app/src/main/java/com/google/edwmigration/dumper/application/dumper/ConnectorArguments.java
Outdated
Show resolved
Hide resolved
.../app/src/main/java/com/google/edwmigration/dumper/application/dumper/ConnectorArguments.java
Outdated
Show resolved
Hide resolved
.../google/edwmigration/dumper/application/dumper/annotations/RespectsArgumentQueryLogDays.java
Show resolved
Hide resolved
...va/com/google/edwmigration/dumper/application/dumper/connector/airflow/AirflowConnector.java
Outdated
Show resolved
Hide resolved
...va/com/google/edwmigration/dumper/application/dumper/connector/airflow/AirflowConnector.java
Outdated
Show resolved
Hide resolved
I do the calculation and log the actual time range to show the date range explicitly. I agree that I'm open to ideas. There is a good example: https://grafana.com/docs/grafana/latest/dashboards/use-dashboards/#set-dashboard-time-range does it look better? |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
My thoughts from discussion:
- Start should temporally be before end.
- Can we make the names sufficiently generic that we can have one set of arguments for query logs, airflow logs, etc, in the future?
- lookback implies with respect to end, so it doesn't "feel" right to say "start" and lookback (because it would be forwards from start). Can we use interval, period, something like that which is agnostic about which endpoint we anchor?
Overall, I appreciate the philosophy that is driving these clarifications, so thank you for asking.
I will keep only |
Add date range support to airflow connector.
Usage:
--lookback-days=x
-> date range:[now-x, now)
--start-date=x --end-date=y
-> date range:[x, y)
-start-date=x --lookback-days=y
-> date range:[x-y, x)
If date range is provided, sql queries will be generate for each day in the interval.