You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
This issue could be mitigated by adding Reshuffle before writing the files. However, we could add this as the builtin step when detecting too many bundles with small sizes.
Issue Priority
Priority: 3 (nice-to-have improvement)
Issue Components
Component: Python SDK
Component: Java SDK
Component: Go SDK
Component: Typescript SDK
Component: IO connector
Component: Beam YAML
Component: Beam examples
Component: Beam playground
Component: Beam katas
Component: Website
Component: Spark Runner
Component: Flink Runner
Component: Samza Runner
Component: Twister2 Runner
Component: Hazelcast Jet Runner
Component: Google Cloud Dataflow Runner
The text was updated successfully, but these errors were encountered:
What would you like to happen?
FileIO supports https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/io/FileIO.Write.html#withAutoSharding, which usually works well to split large bundles to smaller ones and then write the shards to files. However, in some cases, the shard sizes could be small like few KBs (1-10KBs) and the number of shards could be huge (e.g., 1M).
This issue could be mitigated by adding
Reshuffle
before writing the files. However, we could add this as the builtin step when detecting too many bundles with small sizes.Issue Priority
Priority: 3 (nice-to-have improvement)
Issue Components
The text was updated successfully, but these errors were encountered: