Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(conf): add concurrency_timeout option #13027

Closed
wants to merge 1 commit into from

Conversation

outsinre
Copy link
Contributor

@outsinre outsinre commented May 14, 2024

Summary

feat(conf): add concurrency_timeout option

Sets the timeout (in ms) for which a concurrency
job (e.g. declaractive reconfigure) should wait
for mutex before giving up. After the timeout,
Kong gateway would skip the job, and log an error
message.

If you happen to see "timeout acquiring ... lock"
log entries, try to adjust this value accordingly.
Please be careful that setting a large value might
result in a long list of concurreny jobs in queue,
which could overflow the queue.

It is fine to timeout if a job is safe to be
overriden by subsequent jobs, like declaractive
reconfigure. Do not touch this option unless you
clear about it.

Checklist

  • The Pull Request has tests
  • A changelog file has been created under changelog/unreleased/kong or skip-changelog label added on PR if changelog is unnecessary. README.md
  • There is a user-facing docs PR against https://github.com/Kong/docs.konghq.com - PUT DOCS PR HERE

Issue reference

Fix #FTI-5930 #[KAG-4480]

@github-actions github-actions bot added core/proxy core/templates core/configuration cherry-pick kong-ee schedule this PR for cherry-picking to kong/kong-ee labels May 14, 2024
@outsinre outsinre force-pushed the outsinre/FTI-5930-concurrency_timeout branch 2 times, most recently from 1c20977 to d517383 Compare May 14, 2024 03:39
@outsinre outsinre requested review from mheap and chronolaw May 14, 2024 03:40
@outsinre outsinre force-pushed the outsinre/FTI-5930-concurrency_timeout branch from d517383 to 462f6e3 Compare May 14, 2024 03:44
@outsinre outsinre marked this pull request as draft May 14, 2024 03:46
@outsinre outsinre requested a review from bungle May 14, 2024 03:46
@outsinre outsinre force-pushed the outsinre/FTI-5930-concurrency_timeout branch 3 times, most recently from 952559a to f7f4afe Compare May 14, 2024 07:04
@outsinre outsinre force-pushed the outsinre/FTI-5930-concurrency_timeout branch 2 times, most recently from c339c80 to e6347e6 Compare May 14, 2024 11:47
@chronolaw
Copy link
Contributor

Is this PR ready for review?

@outsinre outsinre marked this pull request as ready for review May 15, 2024 13:12
@outsinre
Copy link
Contributor Author

@chronolaw ready!

Sets the timeout (in ms) for which a concurrency
job (e.g. declaractive reconfigure) should wait
for mutex before giving up. After the timeout,
Kong gateway would skip the job, and log an error
message.

If you happen to see "timeout acquiring ... lock"
log entries, try to adjust this value accordingly.
Please be careful that setting a large value might
result in a long list of concurreny jobs in queue,
which could overflow the queue.

It is fine to timeout if a job is safe to be
overriden by subsequent jobs, like declaractive
reconfigure. Do not touch this option unless you
clear about it.
@outsinre outsinre force-pushed the outsinre/FTI-5930-concurrency_timeout branch from b73037f to 1cf75d3 Compare May 16, 2024 06:33
Copy link
Member

@bungle bungle left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don’t want this. Another setting that nobody knows how to use. And we have a better fix for the issue at hand already.

Better to fix root causes than the symptoms.

Copy link
Member

@bungle bungle left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let’s not continue this path.

-- Is it worth to have node level mutex instead?
-- If so, the RETRY_LRU also needs to be node level.
concurrency.with_coroutine_mutex({
name = name,
timeout = ROTATION_INTERVAL,
timeout = concurrency_timeout,
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changes timeout without changing rotation interval used elsewhere. Removes the link between then.

@bungle bungle marked this pull request as draft May 17, 2024 04:34
@outsinre outsinre closed this May 17, 2024
@outsinre
Copy link
Contributor Author

Let's leave this to customers as usually a job takes shorter than 60s.

@chronolaw chronolaw deleted the outsinre/FTI-5930-concurrency_timeout branch May 17, 2024 05:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants