Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[CI] ReactiveStorageDeciderDecisionTests testStoragePreventsAllocation failing #105957

Closed
fcofdez opened this issue Mar 5, 2024 · 1 comment · Fixed by #108676
Closed

[CI] ReactiveStorageDeciderDecisionTests testStoragePreventsAllocation failing #105957

fcofdez opened this issue Mar 5, 2024 · 1 comment · Fixed by #108676
Assignees
@idegtiarenko
Labels
:Distributed/Autoscaling medium-risk An open issue or test failure that is a medium risk to future releases Team:Distributed Meta label for distributed team >test-failure Triaged test failures from CI

Comments

@fcofdez
Copy link
Contributor

fcofdez commented Mar 5, 2024 •
edited by idegtiarenko

Build scan:
https://gradle-enterprise.elastic.co/s/nquynbhzzsdcw/tests/:x-pack:plugin:autoscaling:test/org.elasticsearch.xpack.autoscaling.storage.ReactiveStorageDeciderDecisionTests/testStoragePreventsAllocation

Reproduction line:

./gradlew ':x-pack:plugin:autoscaling:test' --tests "org.elasticsearch.xpack.autoscaling.storage.ReactiveStorageDeciderDecisionTests.testStoragePreventsAllocation" -Dtests.seed=2BCE202E19A4B5D1 -Dtests.locale=sr-ME -Dtests.timezone=Etc/GMT+2 -Druntime.java=17

Applicable branches:
main

Reproduces locally?:
Yes

Failure history:
Failure dashboard for org.elasticsearch.xpack.autoscaling.storage.ReactiveStorageDeciderDecisionTests#testStoragePreventsAllocation

Failure excerpt:

java.lang.AssertionError: 
Expected: sameInstance(<cluster uuid: _na_ [committed: false]
version: 0
state uuid: -4ZrEf4JTj2mQndGjOiwaQ
from_diff: false
meta data version: 0
   coordination_metadata:
      term: 0
      last_committed_config: VotingConfiguration{}
      last_accepted_config: VotingConfiguration{}
      voting tombstones: []
   [test-3]: v[9], mv[1], sv[1], av[1]
      0: p_term [1], isa_ids [ec5BqwZBT6mqOyI2FUMHRg, mMLUEpWpQKyPnfHXZxXQ8g]
      1: p_term [1], isa_ids [uvo_rzB0Rf6Kym-B3b-Mug, m-8s1dPtQUOABnEuWhboPQ]
      2: p_term [1], isa_ids [BBdn-HuwRTGEWQ6Xps-PjA, WUFjxQ7FRUmVyO_grRCvlw]
metadata customs:
   index-graveyard: IndexGraveyard[[]]
nodes (node-left generation: 0):
   {hot_2}{hot_2}{-C5v6i-kQaKXIdyHp7lvLQ}{8Oj_LkMHRHSeI94ly3SrgA}{0.0.0.0}{0.0.0.0:17}{h}{8.13.0}{7000099-8502001}
   {hot_1}{hot_1}{6DDHiw7mQni9TnpdunIsvA}{ln2XChMBReuj9Xq2UTlUrA}{0.0.0.0}{0.0.0.0:16}{h}{8.13.0}{7000099-8502001}
   {hot_0}{hot_0}{MNeDvWzdQWeGkAUl6b1vUw}{oyc-_bOQTcKLAVGP3oPGVg}{0.0.0.0}{0.0.0.0:15}{h}{8.13.0}{7000099-8502001}
   {warm_2}{warm_2}{-OMQMMfkSZePnORRIGipqw}{8SS8SE2CRfSVlzVMLGjkvQ}{0.0.0.0}{0.0.0.0:20}{w}{8.13.0}{7000099-8502001}
   {warm_1}{warm_1}{uE53YF6zRgqx9sdjPpAM9g}{j9D8Dv54TX-BlgTKfY0XmQ}{0.0.0.0}{0.0.0.0:19}{w}{8.13.0}{7000099-8502001}
   {warm_0}{warm_0}{Nx8BScO6QMCVfILKmGBPxQ}{1O0LsISXSh2_lNMq6DhnIw}{0.0.0.0}{0.0.0.0:18}{w}{8.13.0}{7000099-8502001}
cluster features:
   hot_2: []
   hot_1: []
   hot_0: []
   warm_2: []
   warm_1: []
   warm_0: []
routing_table (version 0):
-- index [[test-3]]
----shard_id [test-3][0]
--------[test-3][0], node[hot_0], [R], s[STARTED], a[id=mMLUEpWpQKyPnfHXZxXQ8g], failed_attempts[0]
--------[test-3][0], node[hot_2], [P], s[STARTED], a[id=ec5BqwZBT6mqOyI2FUMHRg], failed_attempts[0]
----shard_id [test-3][1]
--------[test-3][1], node[hot_1], relocating [hot_2], [R], s[RELOCATING], a[id=m-8s1dPtQUOABnEuWhboPQ, rId=ugIDHeyeTRi5WgN4oHhNkg], failed_attempts[0], expected_shard_size[0]
--------[test-3][1], node[hot_0], [P], s[STARTED], a[id=uvo_rzB0Rf6Kym-B3b-Mug], failed_attempts[0]
----shard_id [test-3][2]
--------[test-3][2], node[hot_0], relocating [hot_1], [R], s[RELOCATING], a[id=BBdn-HuwRTGEWQ6Xps-PjA, rId=9bdchFBJScW7elppTWRcGw], failed_attempts[0], expected_shard_size[1]
--------[test-3][2], node[hot_2], [P], s[STARTED], a[id=WUFjxQ7FRUmVyO_grRCvlw], failed_attempts[0]

routing_nodes:
-----node_id[hot_1][V]
--------[test-3][1], node[hot_1], relocating [hot_2], [R], s[RELOCATING], a[id=m-8s1dPtQUOABnEuWhboPQ, rId=ugIDHeyeTRi5WgN4oHhNkg], failed_attempts[0], expected_shard_size[0]
--------[test-3][2], node[hot_1], relocating [hot_0], [R], recovery_source[peer recovery], s[INITIALIZING], a[id=9bdchFBJScW7elppTWRcGw, rId=BBdn-HuwRTGEWQ6Xps-PjA], failed_attempts[0], expected_shard_size[1]
-----node_id[hot_0][V]
--------[test-3][0], node[hot_0], [R], s[STARTED], a[id=mMLUEpWpQKyPnfHXZxXQ8g], failed_attempts[0]
--------[test-3][1], node[hot_0], [P], s[STARTED], a[id=uvo_rzB0Rf6Kym-B3b-Mug], failed_attempts[0]
--------[test-3][2], node[hot_0], relocating [hot_1], [R], s[RELOCATING], a[id=BBdn-HuwRTGEWQ6Xps-PjA, rId=9bdchFBJScW7elppTWRcGw], failed_attempts[0], expected_shard_size[1]
-----node_id[warm_2][V]
-----node_id[warm_1][V]
-----node_id[warm_0][V]
-----node_id[hot_2][V]
--------[test-3][0], node[hot_2], [P], s[STARTED], a[id=ec5BqwZBT6mqOyI2FUMHRg], failed_attempts[0]
--------[test-3][1], node[hot_2], relocating [hot_1], [R], recovery_source[peer recovery], s[INITIALIZING], a[id=ugIDHeyeTRi5WgN4oHhNkg, rId=m-8s1dPtQUOABnEuWhboPQ], failed_attempts[0], expected_shard_size[0]
--------[test-3][2], node[hot_2], [P], s[STARTED], a[id=WUFjxQ7FRUmVyO_grRCvlw], failed_attempts[0]
---- unassigned
>)
     but: was <cluster uuid: _na_ [committed: false]
version: 0
state uuid: -4ZrEf4JTj2mQndGjOiwaQ
from_diff: false
meta data version: 0
   coordination_metadata:
      term: 0
      last_committed_config: VotingConfiguration{}
      last_accepted_config: VotingConfiguration{}
      voting tombstones: []
   [test-3]: v[10], mv[1], sv[1], av[1]
      0: p_term [1], isa_ids [ec5BqwZBT6mqOyI2FUMHRg, mMLUEpWpQKyPnfHXZxXQ8g]
      1: p_term [1], isa_ids [ugIDHeyeTRi5WgN4oHhNkg, uvo_rzB0Rf6Kym-B3b-Mug]
      2: p_term [1], isa_ids [WUFjxQ7FRUmVyO_grRCvlw, 9bdchFBJScW7elppTWRcGw]
metadata customs:
   index-graveyard: IndexGraveyard[[]]
nodes (node-left generation: 0):
   {hot_2}{hot_2}{-C5v6i-kQaKXIdyHp7lvLQ}{8Oj_LkMHRHSeI94ly3SrgA}{0.0.0.0}{0.0.0.0:17}{h}{8.13.0}{7000099-8502001}
   {hot_1}{hot_1}{6DDHiw7mQni9TnpdunIsvA}{ln2XChMBReuj9Xq2UTlUrA}{0.0.0.0}{0.0.0.0:16}{h}{8.13.0}{7000099-8502001}
   {hot_0}{hot_0}{MNeDvWzdQWeGkAUl6b1vUw}{oyc-_bOQTcKLAVGP3oPGVg}{0.0.0.0}{0.0.0.0:15}{h}{8.13.0}{7000099-8502001}
   {warm_2}{warm_2}{-OMQMMfkSZePnORRIGipqw}{8SS8SE2CRfSVlzVMLGjkvQ}{0.0.0.0}{0.0.0.0:20}{w}{8.13.0}{7000099-8502001}
   {warm_1}{warm_1}{uE53YF6zRgqx9sdjPpAM9g}{j9D8Dv54TX-BlgTKfY0XmQ}{0.0.0.0}{0.0.0.0:19}{w}{8.13.0}{7000099-8502001}
   {warm_0}{warm_0}{Nx8BScO6QMCVfILKmGBPxQ}{1O0LsISXSh2_lNMq6DhnIw}{0.0.0.0}{0.0.0.0:18}{w}{8.13.0}{7000099-8502001}
cluster features:
   hot_2: []
   hot_1: []
   hot_0: []
   warm_2: []
   warm_1: []
   warm_0: []
routing_table (version 0):
-- index [[test-3]]
----shard_id [test-3][0]
--------[test-3][0], node[hot_0], [R], s[STARTED], a[id=mMLUEpWpQKyPnfHXZxXQ8g], failed_attempts[0]
--------[test-3][0], node[hot_2], [P], s[STARTED], a[id=ec5BqwZBT6mqOyI2FUMHRg], failed_attempts[0]
----shard_id [test-3][1]
--------[test-3][1], node[hot_0], [P], s[STARTED], a[id=uvo_rzB0Rf6Kym-B3b-Mug], failed_attempts[0]
--------[test-3][1], node[hot_2], relocating [hot_1], [R], s[RELOCATING], a[id=ugIDHeyeTRi5WgN4oHhNkg, rId=Xfhb5zH0RdqN2tpmxTVuVg], failed_attempts[0], expected_shard_size[1]
----shard_id [test-3][2]
--------[test-3][2], node[hot_1], [R], s[STARTED], a[id=9bdchFBJScW7elppTWRcGw], failed_attempts[0]
--------[test-3][2], node[hot_2], [P], s[STARTED], a[id=WUFjxQ7FRUmVyO_grRCvlw], failed_attempts[0]

routing_nodes:
-----node_id[hot_1][V]
--------[test-3][1], node[hot_1], relocating [hot_2], [R], recovery_source[peer recovery], s[INITIALIZING], a[id=Xfhb5zH0RdqN2tpmxTVuVg, rId=ugIDHeyeTRi5WgN4oHhNkg], failed_attempts[0], expected_shard_size[1]
--------[test-3][2], node[hot_1], [R], s[STARTED], a[id=9bdchFBJScW7elppTWRcGw], failed_attempts[0]
-----node_id[hot_0][V]
--------[test-3][0], node[hot_0], [R], s[STARTED], a[id=mMLUEpWpQKyPnfHXZxXQ8g], failed_attempts[0]
--------[test-3][1], node[hot_0], [P], s[STARTED], a[id=uvo_rzB0Rf6Kym-B3b-Mug], failed_attempts[0]
-----node_id[warm_2][V]
-----node_id[warm_1][V]
-----node_id[warm_0][V]
-----node_id[hot_2][V]
--------[test-3][0], node[hot_2], [P], s[STARTED], a[id=ec5BqwZBT6mqOyI2FUMHRg], failed_attempts[0]
--------[test-3][1], node[hot_2], relocating [hot_1], [R], s[RELOCATING], a[id=ugIDHeyeTRi5WgN4oHhNkg, rId=Xfhb5zH0RdqN2tpmxTVuVg], failed_attempts[0], expected_shard_size[1]
--------[test-3][2], node[hot_2], [P], s[STARTED], a[id=WUFjxQ7FRUmVyO_grRCvlw], failed_attempts[0]
---- unassigned
>

  at __randomizedtesting.SeedInfo.seed([2BCE202E19A4B5D1:1386B774A1EB3848]:0)
  at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:18)
  at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:6)
  at org.elasticsearch.test.ESTestCase.assertThat(ESTestCase.java:2119)
  at org.elasticsearch.xpack.autoscaling.storage.ReactiveStorageDeciderDecisionTests.testStoragePreventsAllocation(ReactiveStorageDeciderDecisionTests.java:235)
  at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(NativeMethodAccessorImpl.java:-2)
  at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
  at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  at java.lang.reflect.Method.invoke(Method.java:568)
  at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:946)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:982)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at org.junit.rules.RunRules.evaluate(RunRules.java:20)
  at org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:48)
  at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  at org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
  at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
  at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
  at org.junit.rules.RunRules.evaluate(RunRules.java:20)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:843)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:490)
  at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
  at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
  at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
  at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
  at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
  at org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
  at org.junit.rules.RunRules.evaluate(RunRules.java:20)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850)
  at java.lang.Thread.run(Thread.java:833)
@fcofdez fcofdez added :Distributed/Autoscaling >test-failure Triaged test failures from CI Team:Distributed Meta label for distributed team labels Mar 5, 2024
@elasticsearchmachine
Copy link
Collaborator

Pinging @elastic/es-distributed (Team:Distributed)

@idegtiarenko idegtiarenko added medium-risk An open issue or test failure that is a medium risk to future releases and removed blocker labels Mar 12, 2024
This was referenced May 15, 2024
Merged
Closed
@idegtiarenko idegtiarenko self-assigned this May 22, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@idegtiarenko idegtiarenko

Labels
:Distributed/Autoscaling medium-risk An open issue or test failure that is a medium risk to future releases Team:Distributed Meta label for distributed team >test-failure Triaged test failures from CI
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fix testStoragePreventsAllocation idegtiarenko/elasticsearch
3 participants
@fcofdez @idegtiarenko @elasticsearchmachine