[Feature] Reduce S3 API calls

[polyzos]

Search before asking

• I searched in the issues and found nothing similar.

Motivation

Overall Apache Paimon offers low TCO, but in terms of S3 API calls it makes way more compared to Iceberg.
Using version 0.7 you can see the following in the screenshot

On the left it's the S3 API calls Apache Iceberg makes and on the right its Apache Paimon (~3k/hour I believe)

I believe @JingsongLi mentioned some improvements were made in the upcoming 08 version, but as this is important for many companies we need to keep it as low as iceberg if possible.

Solution

No response

Anything else?

No response

Are you willing to submit a PR?

• I'm willing to submit a PR!

[polyzos]

More context on this: turns out it was the effect of not having compaction running in the background., which wasn't clear in the first place.

This means that although partition/snapshot/manifest file expiration happens automatically
https://paimon.apache.org/docs/master/maintenance/manage-snapshots/#expire-snapshots
it results in more files being generated, but not physically deleted because compaction is disabled.

This means that although the generated files keep increasing, they are also never removed, but Paimon will still have to access/list more metadata files (more and more over time), in order to find out what's still relevant and what's not.

Turning compaction, in the long run will keep the requests stable.

At the same time it is also recommended to use version 0.8+ that brings many improvements as well