Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Improve][KafkaSource]Concurrent reading of Kafka partitions. #6733

Open
wants to merge 1 commit into
base: dev
Choose a base branch
from

Conversation

lightzhao
Copy link
Contributor

@lightzhao lightzhao commented Apr 19, 2024

Purpose of this pull request

If there is only one Task, then the reading of multiple partitions is synchronized, which affects performance and should be upgraded to concurrent.

Does this PR introduce any user-facing change?

How was this patch tested?

Check list

@lightzhao
Copy link
Contributor Author

@hailin0 @EricJoy2048 @Hisoka-X PTAL.

@Hisoka-X
Copy link
Member

Thanks @lightzhao for created this PR. Quick question: why not recommend users to add parallelism to job?

@lightzhao
Copy link
Contributor Author

Quick question: why not recommend users to add parallelism to job?

More concurrency requires more Task Managers, resulting in higher resource usage and costs.

@Hisoka-X
Copy link
Member

Quick question: why not recommend users to add parallelism to job?

More concurrency requires more Task Managers, resulting in higher resource usage and costs.

This change will bring uncertainty, for example, data from different splits are not written in order, and we do not guarantee the thread safety of the Collector. I prefer to use the parallelism parameter.

@lightzhao
Copy link
Contributor Author

This change will bring uncertainty, for example, data from different splits are not written in order, and we do not guarantee the thread safety of the Collector. I prefer to use the parallelism parameter.

Yes, there may indeed be issues with out of order, and the collector may also have thread safety issues.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants