Airbyte creating too many attempts and not terminating old ones #38187
Labels
area/platform
issues related to the platform
community
team/platform-move
type/bug
Something isn't working
Helm Chart Version
0.44.1
What step the error happened?
During the Sync
Relevant information
In a managed kubernetes deployment managed by another team, we deployed airbyte and created some pipelines (csv to a clickhouse database and postresql to clickhouse), the sync was running everyday for couples of days but started failing, after troubleshooting, one of the
![image](https://private-user-images.githubusercontent.com/19733876/330457232-6e2b997a-95b3-4a1a-b8f4-30c8dd5aeea9.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MTgwOTYxMTUsIm5iZiI6MTcxODA5NTgxNSwicGF0aCI6Ii8xOTczMzg3Ni8zMzA0NTcyMzItNmUyYjk5N2EtOTViMy00YTFhLWI4ZjQtMzBjOGRkNWFlZWE5LnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA2MTElMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwNjExVDA4NTAxNVomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPWYyZGU0ODJiYWYxMTZhM2E2MTFmNjI0YjExMDJlN2U0YTJhNmNhMGYyNWM3OWZlYTUxZGNlMDI0Y2I0YTkzZGEmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.dm1Ofgf2oog8_hrWpIV5Ko1T2cZ83Oqj0UCFEQy8qRQ)
orchestrator-repl-job-50-attempt-x
that is responsible for writing data to clickhouse has insufficient cpu:we could resolve it by adding more k8s nodes or free some resources, but we found out many pods starting with names (
![image](https://private-user-images.githubusercontent.com/19733876/330461336-985659e2-59d1-426d-8e15-ee423714b19b.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MTgwOTYxMTUsIm5iZiI6MTcxODA5NTgxNSwicGF0aCI6Ii8xOTczMzg3Ni8zMzA0NjEzMzYtOTg1NjU5ZTItNTlkMS00MjZkLThlMTUtZWU0MjM3MTRiMTliLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA2MTElMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwNjExVDA4NTAxNVomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTBjMDJkZTMzN2E2OThlODlkOGI3OWE4MTc5OTNhODg0NTBlMWQyZmM1OGIxMGMyZDMwYzc0YmU1OTkxOWJiOTYmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.K93_ptS_rXg4GIrNMH2a89vzJRGOI0vRo-LjTO3PTWA)
![image](https://private-user-images.githubusercontent.com/19733876/330461496-d3102857-6982-43fc-85d4-8ff559456576.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MTgwOTYxMTUsIm5iZiI6MTcxODA5NTgxNSwicGF0aCI6Ii8xOTczMzg3Ni8zMzA0NjE0OTYtZDMxMDI4NTctNjk4Mi00M2ZjLTg1ZDQtOGZmNTU5NDU2NTc2LnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA2MTElMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwNjExVDA4NTAxNVomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPWY0ODEzYzIwNzBjYjA2Yzc2ZGE4ZGFkZmNmODMwYjBlNDk5ZDA5YjJkYmI0YjkxODgwNzg1NjgyYjhmYjU3YmUmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.DfjdtiVGOVjp7GXoBLloFbfIDFHY5WnFDQoPERu6NvA)
orchestrator-repl-job-50-attempt-X
,destination-clickhouse-check-48-X-yzgdz
,n-clickhouse-check-1dd1ea2d-a22d-4b9a-bc6d-828
,rce-mysql-discover-09f290d4-c311-426e-bdc8-53f88f4059f1-0-eqymi
, etc.) are not being deleted by airbyte. Seems that Airbyte is forcing the attempts one after another:Checking the documentation about configuring jobs parameters, to force the number of attempts to be 2 for example like the
SYNC_JOB_MAX_ATTEMPTS
, but can't find where to configure it, is it by updating the configmapairbyte-env
? or in which section invalues.yml
? need a confirmation about it for experimentation reason.I'm new to airbyte and the main question why airbyte doesn't delete old pods when it is trying many attempts? is it a bug?
Thanks,
Marwane.
Relevant log output
The text was updated successfully, but these errors were encountered: