Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[CI] EsRejectedExecutionException trips TransportService assertion on shutdown #108642

Closed
DaveCTurner opened this issue May 14, 2024 · 1 comment · Fixed by #108643
Closed

[CI] EsRejectedExecutionException trips TransportService assertion on shutdown #108642

DaveCTurner opened this issue May 14, 2024 · 1 comment · Fixed by #108643
Labels
:Distributed/Cluster Coordination Cluster formation and cluster state publication, including cluster membership and fault detection. low-risk An open issue or test failure that is a low risk to future releases Team:Distributed Meta label for distributed team >test-failure Triaged test failures from CI

Comments

@DaveCTurner
Copy link
Contributor

Not directly related to this test I think.

Build scan:
https://gradle-enterprise.elastic.co/s/msnan5s2tjtxy/tests/:server:internalClusterTest/org.elasticsearch.action.admin.indices.refresh.RefreshBlocksIT/testRefreshWithBlocks

Reproduction line:

./gradlew ':server:internalClusterTest' --tests "org.elasticsearch.action.admin.indices.refresh.RefreshBlocksIT.testRefreshWithBlocks" -Dtests.seed=9FA63D5B374741D3 -Dtests.locale=ko -Dtests.timezone=America/Yakutat -Druntime.java=21

Applicable branches:
main

Reproduces locally?:
Didn't try

Failure history:
Failure dashboard for org.elasticsearch.action.admin.indices.refresh.RefreshBlocksIT#testRefreshWithBlocks

Failure excerpt:

java.lang.AssertionError: org.elasticsearch.common.util.concurrent.EsRejectedExecutionException: rejected execution of java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@7de16174[Not completed, task = java.util.concurrent.Executors$RunnableAdapter@25db480d[Wrapped task = scheduled check of leader {node_t2}{75sBpKQOQX23F0K31vZEPw}{ZW_aC5qJR-Cmmm45dw8Mpw}{node_t2}{127.0.0.1}{127.0.0.1:13812}{ilmrt}{8.15.0}{7000099-8506000}]] on org.elasticsearch.threadpool.Scheduler$SafeScheduledThreadPoolExecutor@558b42e4[Shutting down, pool size = 1, active threads = 1, queued tasks = 0, completed tasks = 5] (shutdown)

  at __randomizedtesting.SeedInfo.seed([9FA63D5B374741D3]:0)
  at org.elasticsearch.transport.TransportService$5.doRun(TransportService.java:988)
  at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
  at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:244)
  at org.elasticsearch.transport.TransportService.handleInternalSendException(TransportService.java:979)
  at org.elasticsearch.transport.TransportService.sendRequestInternal(TransportService.java:945)
  at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:849)
  at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:759)
  at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler.handleWakeUp(LeaderChecker.java:233)
  at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$3.run(LeaderChecker.java:387)
  at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:917)
  at org.elasticsearch.threadpool.ThreadPool$1.run(ThreadPool.java:589)
  at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572)
  at java.util.concurrent.FutureTask.run(FutureTask.java:317)
  at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
  at java.lang.Thread.run(Thread.java:1583)

  Caused by: org.elasticsearch.common.util.concurrent.EsRejectedExecutionException: rejected execution of java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask@7de16174[Not completed, task = java.util.concurrent.Executors$RunnableAdapter@25db480d[Wrapped task = scheduled check of leader {node_t2}{75sBpKQOQX23F0K31vZEPw}{ZW_aC5qJR-Cmmm45dw8Mpw}{node_t2}{127.0.0.1}{127.0.0.1:13812}{ilmrt}{8.15.0}{7000099-8506000}]] on org.elasticsearch.threadpool.Scheduler$SafeScheduledThreadPoolExecutor@558b42e4[Shutting down, pool size = 1, active threads = 1, queued tasks = 0, completed tasks = 5] (shutdown)

    at org.elasticsearch.common.util.concurrent.EsRejectedExecutionHandler.newRejectedException(EsRejectedExecutionHandler.java:51)
    at org.elasticsearch.common.util.concurrent.EsAbortPolicy.rejectedExecution(EsAbortPolicy.java:33)
    at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:841)
    at java.util.concurrent.ScheduledThreadPoolExecutor.delayedExecute(ScheduledThreadPoolExecutor.java:340)
    at java.util.concurrent.ScheduledThreadPoolExecutor.schedule(ScheduledThreadPoolExecutor.java:562)
    at org.elasticsearch.threadpool.ThreadPool.schedule(ThreadPool.java:611)
    at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler.scheduleNextWakeUp(LeaderChecker.java:384)
    at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$1.handleException(LeaderChecker.java:334)
    at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1476)
    at org.elasticsearch.transport.TransportService$5.doRun(TransportService.java:986)
    at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26)
    at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:244)
    at org.elasticsearch.transport.TransportService.handleInternalSendException(TransportService.java:979)
    at org.elasticsearch.transport.TransportService.sendRequestInternal(TransportService.java:945)
    at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:849)
    at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:759)
    at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler.handleWakeUp(LeaderChecker.java:233)
    at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$3.run(LeaderChecker.java:387)
    at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:917)
    at org.elasticsearch.threadpool.ThreadPool$1.run(ThreadPool.java:589)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572)
    at java.util.concurrent.FutureTask.run(FutureTask.java:317)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
    at java.lang.Thread.run(Thread.java:1583)
@DaveCTurner DaveCTurner added :Distributed/Network Http and internode communication implementations >test-failure Triaged test failures from CI medium-risk An open issue or test failure that is a medium risk to future releases labels May 14, 2024
@elasticsearchmachine
Copy link
Collaborator

Pinging @elastic/es-distributed (Team:Distributed)

@elasticsearchmachine elasticsearchmachine added the Team:Distributed Meta label for distributed team label May 14, 2024
@DaveCTurner DaveCTurner added :Distributed/Cluster Coordination Cluster formation and cluster state publication, including cluster membership and fault detection. low-risk An open issue or test failure that is a low risk to future releases and removed :Distributed/Network Http and internode communication implementations medium-risk An open issue or test failure that is a medium risk to future releases labels May 14, 2024
DaveCTurner added a commit to DaveCTurner/elasticsearch that referenced this issue May 14, 2024
@DaveCTurner
Use scheduleUnlessShuttingDown in LeaderChecker
5c707b4 
We mustn't throw an exception here, it trips an assertion. Simply giving
up on shutdown is fine.

Closes elastic#108642
@DaveCTurner DaveCTurner mentioned this issue May 14, 2024
Merged
DaveCTurner added a commit that referenced this issue May 15, 2024
@DaveCTurner
Use scheduleUnlessShuttingDown in LeaderChecker (#108643)
a037e8c 
We mustn't throw an exception here, it trips an assertion. Simply giving
up on shutdown is fine.

Closes #108642
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
:Distributed/Cluster Coordination Cluster formation and cluster state publication, including cluster membership and fault detection. low-risk An open issue or test failure that is a low risk to future releases Team:Distributed Meta label for distributed team >test-failure Triaged test failures from CI
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Use scheduleUnlessShuttingDown in LeaderChecker DaveCTurner/elasticsearch
2 participants
@DaveCTurner @elasticsearchmachine