Skip to content

OpenLineage run start time can be set multiple times #2408

@collado-mike

Description

@collado-mike

In some cases, an OpenLineage run can have multiple START events sent (e.g., a Spark SQL job execution will trigger both the SQL start event and the Spark Job event or an Airflow Async operator may fire a start event at the beginning of the task start and another when the async task completes). Marquez updates the start time based on the last start event received (see code here), so we'll have an inaccurate picture of the actual execution time of the OpenLineage job run. We should update the code to correctly capture the actual start time of the event.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    Status

    No status

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions