Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Improvement][Master] Add master explicit overload control #16034

Open
wants to merge 1 commit into
base: dev
Choose a base branch
from

Conversation

pegasas
Copy link
Contributor

@pegasas pegasas commented May 20, 2024

Purpose of the pull request

resolve: #16032

Brief change log

Now master/worker are all based on cpu/memory control in protection. see


When deployed in k8s, it will get cpu/memory from pod node instead of pod JVM.
Add a explicit overload control like worker.

Verify this pull request

This pull request is code cleanup without any test coverage.

(or)

This pull request is already covered by existing tests, such as (please describe tests).

(or)

This change added tests and can be verified as follows:

(or)

If your pull request contain incompatible change, you should also add it to docs/docs/en/guide/upgrede/incompatible.md

Copy link
Member

@ruanwenjun ruanwenjun left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We shouldn't add control here, once a workflow is be loaded means it has get all resource.

@pegasas
Copy link
Contributor Author

pegasas commented May 20, 2024

We shouldn't add control here, once a workflow is be loaded means it has get all resource.

if (isOverload()) {
log.warn("WorkerTaskExecutorThreadPool is overload, cannot submit new WorkerTaskExecutor");
WorkerServerMetrics.incWorkerSubmitQueueIsFullCount();
return false;
}
WorkerTaskExecutorHolder.put(workerTaskExecutor);
threadPoolExecutor.execute(workerTaskExecutor);
return true;
}
}
public boolean isOverload() {

I saw worker control workload like this.

@pegasas
Copy link
Contributor Author

pegasas commented May 23, 2024

aligned with worker like https://github.com/apache/dolphinscheduler/pull/16027/files

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Improvement][Master] Add master explicit overload control like worker
2 participants