Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[VL] Support customize options for parquet native write #5751

Open
gaoyangxiaozhu opened this issue May 15, 2024 · 2 comments
Open

[VL] Support customize options for parquet native write #5751

gaoyangxiaozhu opened this issue May 15, 2024 · 2 comments
Labels
enhancement New feature or request

Comments

@gaoyangxiaozhu
Copy link
Contributor

Description

Currently, parquet native write doesn't support customized option as block_size, page_size etc. when write data.

There is a onging PR in velox side https://github.com/facebookincubator/velox/pull/8864/files#diff-5a2dd3766d9a74bbef58d62d96f0abfb111e8e507ce9bcecd35f69d2c8669ed7 to let support pass flushPolicy which support customize block_size, but not page_size.

Let use this issue to track for supporting customize all parquet options when write data.

@gaoyangxiaozhu gaoyangxiaozhu added the enhancement New feature or request label May 15, 2024
@gaoyangxiaozhu
Copy link
Contributor Author

@JkSelf let use this issue to track the customized block_size, page_size not support problem.

@FelixYBW
Copy link
Contributor

@gaoyangxiaozhu can you list all the parquet write parameters Spark supports and velox/arrow supports? Let's pass all supported params to Velox

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants