Source plugin : Jdbc [Spark]
Read external data source data through JDBC
name | type | required | default value |
---|---|---|---|
driver | string | yes | - |
jdbc.* | string | no | |
password | string | yes | - |
table | string | yes | - |
url | string | yes | - |
user | string | yes | - |
common-options | string | yes | - |
The jdbc class name
used to connect to the remote data source
In addition to the parameters that must be specified above, users can also specify multiple optional parameters, which cover all the parameters provided by Spark JDBC.
The way to specify parameters is to add the prefix jdbc.
to the original parameter name. For example, the way to specify fetchsize
is: jdbc.fetchsize = 50000
. If these non-essential parameters are not specified, they will use the default values given by Spark JDBC.
table name
The URL of the JDBC connection. Refer to a case: jdbc:postgresql://localhost/test
username
Source plugin common parameters, please refer to Source Plugin for details
jdbc {
driver = "com.mysql.jdbc.Driver"
url = "jdbc:mysql://localhost:3306/info"
table = "access"
result_table_name = "access_log"
user = "username"
password = "password"
}
Read MySQL data through JDBC
jdbc {
driver = "com.mysql.jdbc.Driver"
url = "jdbc:mysql://localhost:3306/info"
table = "access"
result_table_name = "access_log"
user = "username"
password = "password"
jdbc.partitionColumn = "item_id"
jdbc.numPartitions = "10"
jdbc.lowerBound = 0
jdbc.upperBound = 100
}
Divide partitions based on specified fields
jdbc {
driver = "com.mysql.jdbc.Driver"
url = "jdbc:mysql://localhost:3306/info"
table = "access"
result_table_name = "access_log"
user = "username"
password = "password"
jdbc.connect_timeout = 10000
jdbc.socket_timeout = 10000
}
Timeout config