Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Spark] Implement Purge table for SparkCatalog #3015

Open
wants to merge 82 commits into
base: master
Choose a base branch
from
Open

[Spark] Implement Purge table for SparkCatalog #3015

wants to merge 82 commits into from

Conversation

zhu3pang
Copy link
Contributor

Purpose

Linked issue: close #1401

Implement Purge table for SparkCatalog

Tests

API and Format

Documentation

@zhu3pang zhu3pang changed the title [PAIMON-1401] [Feature] Implement Purge table for SparkCatalog [Spark] Implement Purge table for SparkCatalog Mar 15, 2024
@JingsongLi
Copy link
Contributor

CC @YannByron

yuzelin and others added 24 commits March 29, 2024 15:46
@yuzelin @zhu3pang
[hotfix] Fix unstable testRandomCdcEventsUnawareBucket (apache#3014)
db88321
@yuzelin @zhu3pang
[flink] Refactor: flink tests use common CatalogITCaseBase (apache#3016)
94a15a0
@xiangyuf @zhu3pang
[core] Support compiled class cache in CodeGenUtils
dd0ef63 
This closes apache#2976
@JingsongLi @zhu3pang
[core] Optimize compiled class cache in CodeGenUtils
dbdcffd
@leaves12138 @zhu3pang
[core] AvroBulkFormat should not get file-size by file-io repeatly (a…
3939cd3 
…pache#3019)
@wg1026688210 @zhu3pang
[hive] Cache TableSchema into Configuration to avoid loading read sch…
ae4ecc0 
…eme file in PaimonSerDe (apache#2946)
@FangYongs @zhu3pang
[core] Support configuring lock in paimon catalog (apache#2933)
4508589 
* [core] Support configuring lock in paimon catalog
@yuzelin @zhu3pang
[codegen] EqualiserCodeGenerator supports ARRAY<ROW> (apache#3023)
c647733
@sunxiaojian @zhu3pang
[core] Unified Naming of JdbcCatalog Fields (apache#3025)
a30275f
@zhuangchong @zhu3pang
[hotfix] upgrade json-path and avro version (apache#2987)
d0f684e
@Zouxxyy @zhu3pang
[core] Dv table supports value filter pushdown (apache#3024)
55304dc
@Alibaba-HZY @zhu3pang
[hive] Fix hive writer in tez-mr (apache#2954)
edbc555
@JingsongLi @zhu3pang
[hive] Add license for TezUtil
aea5509
@JingsongLi @zhu3pang
[core] Reduce schema read for readers (apache#3021)
c2b2cc8
@hzjhjjyy @zhu3pang
[core] Add more validation for sequence field during create table. (a…
b86a8a6 
…pache#3022)
@cxzl25 @zhu3pang
[doc] Updated Spark quickstart documentation for using --packages (ap…
5712563 
…ache#2897)
@JingsongLi @zhu3pang
[core] Use var length encoding for row position (apache#3031)
66eb936
@zhoulii @zhu3pang
[oss] use multi-release mechanism to ship jaxb (apache#2658)
a0817bd
@tsreaper @zhu3pang
[core] Data files with delete records should not be upgraded directly…
731f3ef 
… to max level (apache#2962)
@wg1026688210 @zhu3pang
[Flink] Introduce range strategy for sort compaction. (apache#2749)
b88ba74
@waywtdcc @zhu3pang
[spark] spark sql support get/list function (apache#2880)
e76d242
@leaves12138 @zhu3pang
[docs] Trino docs update for sharing file system. (apache#3035)
1e22661
@Zouxxyy @zhu3pang
[core] Fix dv table with partial-update and aggregate (apache#3036)
b8ebe59
@JingsongLi @zhu3pang
[core] Introduce BatchTableCommit.truncateTable (apache#3037)
5dacfb5
yuzelin and others added 28 commits March 29, 2024 15:46
@yuzelin @zhu3pang
[flink] Fix performance issue in ContinuousFileSplitEnumerator (apach…
de61a18 
…e#3071)
@yuzelin @zhu3pang
[Refactor] Minor refactor update the latest dynamic options using the…
0939a82 
… cdc action. (apache#3090)
@zhangjun0x01 @zhu3pang
[cdc] Extract common code (apache#3050)
8ce013c
@huyuanfeng2018 @zhu3pang
[cdc] Fix Mysql sync_table unable to synchronize newly added tables w…
7e33038 
…ith regular expression (apache#2851)
@yuzelin @zhu3pang
[hotfix] Apply spotless to fix checkstyle violation in flink-cdc modu…
effc57e 
…le (apache#3092)
@leaves12138 @zhu3pang
[test] Append only table can't test failing IO re-insert yet (apache#…
2e65cc5 
…3089)
@Zouxxyy @zhu3pang
[core] Introduce SplitGroup for SplitGenerator to optimize more rawFi…
8ce82a7 
…les (apache#3059)
@yuzelin @zhu3pang
[hotfix] Fix unstable test MySqlSyncTableActionITCase#testSyncShards (a…
f2a4e38 
…pache#3096)
@JingsongLi @zhu3pang
[core] Refactor FileStorePathFactory to clean test methods (apache#3093)
c928967
@JingsongLi @zhu3pang
[core] Add TableFormatBenchmark to test format only
096c433
@YannByron @zhu3pang
[spark][core] Support input_file_name UDF (apache#3094)
cbe121b
@JingsongLi @zhu3pang
[core] Unify FileRecordReader and reduce file access (apache#3098)
7f27f04
@sunxiaojian @zhu3pang
[core] Support common jdbc catalog lock for filesystem catalog. (apac…
59d9ec7 
…he#3076)
@wxplovecc @zhu3pang
[core] Fix OrphanFilesClean has a chance to delete normal file (apach…
78b1ace 
…e#3075)
@JingsongLi @zhu3pang
[catalog] Refactor Catalog Factory and revert jdbc lock in FileSystem…
4c17a3a 
…Catalog (apache#3099)
@Zouxxyy @zhu3pang
[core] Shade roaringbitmap dependency into paimon-common (apache#3100)
a0809a2
@zhuangchong @zhu3pang
[cdc] Fix cdc job mistakenly changes immutable options of existing ta…
ac57e65 
…ble (apache#3095)
@Zouxxyy @zhu3pang
[core] Support dv with avro format (apache#3105)
c265c6f
@leaves12138 @zhu3pang
[core] Introduce File index format for data skipping (apache#3068)
22832e9
@Zouxxyy @zhu3pang
[test] Fix flaky test in `KeyValueFileStoreScanTest.testWithValueFilt…
a5eff5f 
…er` (apache#3109)
@Zouxxyy @zhu3pang
[doc] Introduce deletion vector documentation page (apache#3107)
e110f57
@JingsongLi @zhu3pang
[core] Rename to compactDvIndexFiles in FileStoreCommitImpl
4b99a3f
@leaves12138 @zhu3pang
[test] Fix test in KeyValueFileStoreScanTest, make it more accurate (a…
5fa79b8 
…pache#3111)
@zyl891229 @zhu3pang
[format] Set orc.compression.zstd.level to 3 by default (apache#3114)
9a89253
@YannByron @zhu3pang
[core] add deletedFiles in NewFilesIncrement (apache#3117)
a9ff40a
@yuzelin @zhu3pang
[cdc] Fix that kafka message value might be null which causes NPE (ap…
fa9c6e7 
…ache#3113)
@zhu3pang
[PAIMON-1401] [Feature] Implement Purge table for SparkCatalog, add s…
ea3797d 
…ome UTs
@zhu3pang
Merge remote-tracking branch 'origin/master' into PAIMON-1401
9db339a
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@YannByron YannByron Awaiting requested review from YannByron

@JingsongLi JingsongLi Awaiting requested review from JingsongLi

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[Feature] Implement Purge table for SparkCatalog