My team was able to get generic pipeline DAGs to work with Airflow 2.5.3 on this commit (kflow-ai@f9d1329). We don't have unit tests working locally, so this commit isn't ready for a PR. Perhaps we can work with an active maintainer to get this in proper shape for an upstream merge?

Off the top of my head, we need automated tests to work and potentially test with both earlier and later versions of Airflow. My understanding from this comment is we may actually want to take a different approach to keep Airflow 1 support at the same time.

Do we really need to test it with Airflow 1.x?

By the way, your code looks really good, I have almost exactly that in my local code base right now :-)

Two differences:

https://github.com/apache/airflow/pull/21653/files

days_ago will not be there anymore in Airflow 3.x, so
import pendulum

instead of

from airflow.utils.dates import days_ago

and

schedule='@once'

instead of
schedule_interval='@once'

and

start_date=pendulum.today('UTC').add(days=-1),

instead of

start_date=days_ago(1),

@ianonavy yup, we will certainly work with @lresende I am also sporadically in contact with @thesuperzapper who is involved in both KF Pipelines and airflow helm charts.

@ianonavy "Off the top of my head, we need automated tests to work and potentially test with both earlier and later versions of Airflow." we will deprecate airflow 1.x support with elyra 4.0 and I am thinking of adding more unit tests aside from the existing ones, too.

so far, I only had to modify one existing test in community elyra for it to work https://github.com/elyra-ai/elyra/pull/3167/files#diff-f4fbb5d86d69f450c9235421616ec18e28f22eb174fb97332ba1952026c0a9f6 Do you think more than that unit test change or additonal unit test adding is needed? The existing Elyra tests still all run though in PR 3167 and spcifically https://github.com/elyra-ai/elyra/actions/runs/7505710196/job/20435558496?pr=3167#step:6:506

as mentioned, we are assuming full switchover to airflow 2.x in Elyra 4.0. Discussed with and okayed by the maintainers.