Run tests for Providers also for Airflow 2.8

[potiuk]

This is a follow-up on #39513 to add support for running Provider
tests against Airlfow 2.8 installed from PyPI.

---

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

[potiuk]

OK. Just 4 tests left to fix in 2.8 :).

[potiuk]

together with OL support, so I'm trying to understand if there might be some mismatch of the provider versions there.

I think the problem is connected with upath integration. Airflow 2.8.4 uses older upath version that was not compatible with Python 3.12 - cc: @bolkedebruin

[bolkedebruin]

@potiuk Airflow 2.8.4 requires (constraints) universal-path 0.1.4 which is indeed not compatible with python 3.12. However., I am not sure if this is related. The namespace property was added past in 2.9.0, so I guess the provider is only compatible with Airflow 2.9.0+ without adjustments?

[potiuk]

@potiuk Airflow 2.8.4 requires (constraints) universal-path 0.1.4 which is indeed not compatible with python 3.12. However., I am not sure if this is related. The namespace property was added past in 2.9.0, so I guess the provider is only compatible with Airflow 2.9.0+ without adjustments?

Yes. That was my thought as well (and that's how I solved it in the PR) - In this case I check if we are retrieving openlineage facets in Airlfow < 2.9.0 in this operator and fail it explicitly if so: https://github.com/apache/airflow/pull/39606/files#diff-ee30b541cc8033613196b62a0e5d9ff0276113f27d011a9ffbcd355826205f14R87 . Note that FileTransferOperator should work fine when openlineage is disabled even in 2.8.*

Not sure if this is the best approach @mobuchowski @kacpermuda -> maybe we can make it works also in 2.8 ?

[kacpermuda]

@potiuk do you think #39755 will be enough to fix the test you mentioned and make the io provider + OpenLineage work well with Airflow 2.8 ?

[potiuk]

@potiuk do you think #39755 will be enough to fix the test you mentioned and make the io provider + OpenLineage work well with Airflow 2.8 ?

Yes. In the last rebase I 'lost" previous commit - so you could not really properly test it. but I just restored it and it looks like it's working just fine.

[kacpermuda]

I was wondering why i could not do it. I was trying to run breeze with dev providers and Airflow 2.8.4 like you mentioned, but the fab provider required 2.9.0. After i dealt with it, and run the test, it passed without any problems, but now it makes more sense. Anyway, glad that it should be solved now ^^

[potiuk]

I was wondering why i could not do it. I was trying to run breeze with dev providers and Airflow 2.8.4 like you mentioned, but the fab provider required 2.9.0. After i dealt with it, and run the test, it passed without any problems, but now it makes more sense. Anyway, glad that it should be solved now ^^

Yes. I have to upgrade the docs to add information about removing incompatible providers (and I think I can change it to run those tests locally without actually building packages - so that will simplify the process (separate PR)

[potiuk]

OK . That one is ready to be finally reviewed. All problems with compatibility with 2.8 have been solved in the test suite, also thanks to that one we found out the compatbility issue of common.io filetransfer openlineage integration for 2.8 that has been fixed in #39755 -> so thsoe tests are alredy showing it's worth 🚀

Once this one is merged, I will work on 2.7 compatibility and - in parallell - making it easier to debug and reproduce the tests with previous version of airflow locally.

[potiuk]

I would love to merge that one and work on 2.7 tests :)

[potiuk]

Anyone :) ? I am already working on 2.7 compatibility in #39862