Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[OPENLINEAGE] Introduce AirflowJobFacet and AirflowStateRunFacet #39520

Draft
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

kacpermuda
Copy link
Contributor

closes: #39467


^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

@kacpermuda kacpermuda force-pushed the ol-new-dag-facet branch 3 times, most recently from 35778b2 to ea6a570 Compare May 15, 2024 11:31
@kacpermuda kacpermuda marked this pull request as ready for review May 15, 2024 11:59
@kacpermuda kacpermuda changed the title [OPENLINEAGE] Add new DAG job structure facet [OPENLINEAGE] Introduce AirflowJobFacet and AirflowStateRunFacet May 16, 2024
@kacpermuda kacpermuda marked this pull request as draft May 17, 2024 12:23
@kacpermuda kacpermuda force-pushed the ol-new-dag-facet branch 2 times, most recently from 3d6af0d to 7c0e499 Compare May 21, 2024 14:51
@kacpermuda kacpermuda force-pushed the ol-new-dag-facet branch 3 times, most recently from 77020dd to ec4bc16 Compare May 22, 2024 17:50
@kacpermuda kacpermuda marked this pull request as ready for review May 22, 2024 17:50
@kacpermuda
Copy link
Contributor Author

I had to add some words to spelling wordlist, due to the docs error in CI here. Let me know if there is a better solution to that problem, but part of these words have already been present, just with different capitalization.

if not dag_run.dag:
return {}
return {
"airflowDag": AirflowJobFacet(
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Any ideas for a better key than airflowDag here ?
@mobuchowski @JDarDagran

@kacpermuda
Copy link
Contributor Author

I also added json specs for the facets, in a new directory. I was not sure if i should add the "$id" key, as in the OL repo, as these facets will not be uploaded to openlineage.io, so i skipped it.

Signed-off-by: Kacper Muda <mudakacper@gmail.com>
@kacpermuda kacpermuda marked this pull request as draft May 30, 2024 12:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[OpenLineage] Add new DAG job facet
3 participants