Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Improve][Jdbc/CDC] Improve user permissions in table metadata load #6649

Open
wants to merge 2 commits into
base: dev
Choose a base branch
from

Conversation

hailin0
Copy link
Contributor

@hailin0 hailin0 commented Apr 7, 2024

Purpose of this pull request

[Jdbc/CDC] Improve user permissions in table metadata load

Does this PR introduce any user-facing change?

No

How was this patch tested?

existed

Check list

@hailin0
Copy link
Contributor Author

hailin0 commented Apr 7, 2024

test logs

2024-04-07 14:50:08,766 INFO  org.apache.seatunnel.connectors.seatunnel.jdbc.source.ChunkSplitter - Start splitting table QA_SOURCE.T1 into chunks...
2024-04-07 14:50:09,338 INFO  org.apache.seatunnel.connectors.seatunnel.jdbc.source.ChunkSplitter - Split table, query min max: SELECT MIN("ID"), MAX("ID") FROM (select * from QA_SOURCE.T1) tmp
2024-04-07 14:50:09,392 INFO  org.apache.seatunnel.connectors.seatunnel.jdbc.source.DynamicChunkSplitter - Splitting table QA_SOURCE.T1 into chunks, split column: ID, min: 1, max: 2, chunk size: 8096, distribution factor upper: 100.0, distribution factor lower: 0.05, sample sharding threshold: 1000
2024-04-07 14:50:15,403 INFO  org.apache.seatunnel.connectors.seatunnel.jdbc.internal.dialect.oracle.OracleDialect - Split Chunk, approximateRowCntStatement: analyze table QA_SOURCE.T1 compute statistics for table
2024-04-07 14:50:18,288 WARN  org.apache.seatunnel.connectors.seatunnel.jdbc.internal.dialect.oracle.OracleDialect - Failed to get approximate row count from table status, fallback to count rows
java.sql.SQLSyntaxErrorException: ORA-01031: 权限不足

	at oracle.jdbc.driver.T4CTTIoer11.processError(T4CTTIoer11.java:509) ~[ojdbc8-19.18.0.0.jar:19.18.0.0.0]
	at oracle.jdbc.driver.T4CTTIoer11.processError(T4CTTIoer11.java:461) ~[ojdbc8-19.18.0.0.jar:19.18.0.0.0]
	at oracle.jdbc.driver.T4C8Oall.processError(T4C8Oall.java:1104) ~[ojdbc8-19.18.0.0.jar:19.18.0.0.0]
	at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:553) ~[ojdbc8-19.18.0.0.jar:19.18.0.0.0]
	at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:269) ~[ojdbc8-19.18.0.0.jar:19.18.0.0.0]
	at oracle.jdbc.driver.T4C8Oall.doOALL(T4C8Oall.java:655) ~[ojdbc8-19.18.0.0.jar:19.18.0.0.0]
	at oracle.jdbc.driver.T4CStatement.doOall8(T4CStatement.java:229) ~[ojdbc8-19.18.0.0.jar:19.18.0.0.0]
	at oracle.jdbc.driver.T4CStatement.doOall8(T4CStatement.java:41) ~[ojdbc8-19.18.0.0.jar:19.18.0.0.0]
	at oracle.jdbc.driver.T4CStatement.executeForRows(T4CStatement.java:928) ~[ojdbc8-19.18.0.0.jar:19.18.0.0.0]
	at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1205) ~[ojdbc8-19.18.0.0.jar:19.18.0.0.0]
	at oracle.jdbc.driver.OracleStatement.executeInternal(OracleStatement.java:1823) ~[ojdbc8-19.18.0.0.jar:19.18.0.0.0]
	at oracle.jdbc.driver.OracleStatement.execute(OracleStatement.java:1778) ~[ojdbc8-19.18.0.0.jar:19.18.0.0.0]
	at oracle.jdbc.driver.OracleStatementWrapper.execute(OracleStatementWrapper.java:303) ~[ojdbc8-19.18.0.0.jar:19.18.0.0.0]
	at org.apache.seatunnel.connectors.seatunnel.jdbc.internal.dialect.oracle.OracleDialect.approximateRowCntStatement(OracleDialect.java:210) [classes/:?]
	at org.apache.seatunnel.connectors.seatunnel.jdbc.source.DynamicChunkSplitter.queryApproximateRowCnt(DynamicChunkSplitter.java:205) [classes/:?]
	at org.apache.seatunnel.connectors.seatunnel.jdbc.source.DynamicChunkSplitter.evenlyColumnSplitChunks(DynamicChunkSplitter.java:152) [classes/:?]
	at org.apache.seatunnel.connectors.seatunnel.jdbc.source.DynamicChunkSplitter.splitTableIntoChunks(DynamicChunkSplitter.java:123) [classes/:?]
	at org.apache.seatunnel.connectors.seatunnel.jdbc.source.DynamicChunkSplitter.createDynamicSplits(DynamicChunkSplitter.java:74) [classes/:?]
	at org.apache.seatunnel.connectors.seatunnel.jdbc.source.DynamicChunkSplitter.createSplits(DynamicChunkSplitter.java:62) [classes/:?]
	at org.apache.seatunnel.connectors.seatunnel.jdbc.source.ChunkSplitter.generateSplits(ChunkSplitter.java:101) [classes/:?]
	at org.apache.seatunnel.connectors.seatunnel.jdbc.source.JdbcSourceSplitEnumerator.run(JdbcSourceSplitEnumerator.java:81) [classes/:?]
	at org.apache.seatunnel.engine.server.task.SourceSplitEnumeratorTask.stateProcess(SourceSplitEnumeratorTask.java:311) [classes/:?]
	at org.apache.seatunnel.engine.server.task.SourceSplitEnumeratorTask.call(SourceSplitEnumeratorTask.java:138) [classes/:?]
	at org.apache.seatunnel.engine.server.TaskExecutionService$BlockingWorker.run(TaskExecutionService.java:641) [classes/:?]
	at org.apache.seatunnel.engine.server.TaskExecutionService$NamedTaskWrapper.run(TaskExecutionService.java:942) [classes/:?]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?]
	at java.util.concurrent.FutureTask.run$$$capture(FutureTask.java:264) [?:?]
	at java.util.concurrent.FutureTask.run(FutureTask.java) [?:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
	at java.lang.Thread.run(Thread.java:834) [?:?]
Caused by: oracle.jdbc.OracleDatabaseException: ORA-01031: 权限不足

	at oracle.jdbc.driver.T4CTTIoer11.processError(T4CTTIoer11.java:513) ~[ojdbc8-19.18.0.0.jar:19.18.0.0.0]
	... 30 more
2024-04-07 14:50:20,369 INFO  org.apache.seatunnel.engine.server.checkpoint.CheckpointCoordinator - wait checkpoint completed: 1
2024-04-07 14:50:20,487 INFO  org.apache.seatunnel.connectors.seatunnel.jdbc.internal.dialect.SQLUtils - Split Chunk, countForTable: SELECT COUNT(*) FROM QA_SOURCE.T1
2024-04-07 14:50:26,302 INFO  org.apache.seatunnel.connectors.seatunnel.jdbc.source.DynamicChunkSplitter - The distribution factor of table QA_SOURCE.T1 is 1.0 according to the min split key 1, max split key 2 and approximate row count 2
2024-04-07 14:50:26,303 INFO  org.apache.seatunnel.connectors.seatunnel.jdbc.source.DynamicChunkSplitter - Use evenly-sized chunk optimization for table QA_SOURCE.T1, the approximate row count is 2, the chunk size is 8096, the dynamic chunk size is 8096
2024-04-07 14:50:26,319 INFO  org.apache.seatunnel.connectors.seatunnel.jdbc.source.ChunkSplitter - Split table QA_SOURCE.T1 into 1 chunks, time cost: 17553ms.

@hailin0 hailin0 marked this pull request as ready for review April 7, 2024 08:49
EricJoy2048
EricJoy2048 previously approved these changes Apr 7, 2024
@Hisoka-X
Copy link
Member

Hisoka-X commented Apr 7, 2024

Shall we add a test case to cover this situation which the insufficient user permissions?

@Carl-Zhou-CN
Copy link
Member

Can we prompt users in the log to elevate permissions for better performance?

@hailin0
Copy link
Contributor Author

hailin0 commented Apr 8, 2024

Shall we add a test case to cover this situation which the insufficient user permissions?

Ok, I'll add

wuchunfu
wuchunfu previously approved these changes May 7, 2024
@hailin0 hailin0 dismissed stale reviews from wuchunfu and EricJoy2048 via 42f7fb1 May 15, 2024 05:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

6 participants