[Bug] Partition expiration may conflict with in-flight compaction

[schnappi17]

Search before asking

• I searched in the issues and found nothing similar.

Paimon version

release-0.7

Compute Engine

flink-1.17

Minimal reproduce step

1. create partitioned table, set expired = 7d
2. mock data to always generate partition 19700101
3. sometimes it'll failover because compact deletion will conflict with the partition expiration.
   Let's say there is fileA which need to be marked as delete in a partition expiration task, and writer suppose the file still exists and pick it in a compaction task. Then compaction task will be finished with committing it as delete and will result in deleting a file which is not recorded by the base manifest.

What doesn't meet your expectations?

This caused a flink job failover

Anything else?

No response

Are you willing to submit a PR?

• I'm willing to submit a PR!