Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Failure in system test for io.pravega.test.system.LargeEventTest.largeEventSimpleTest #6924

Open
ShwethaSNayak opened this issue Oct 21, 2022 · 1 comment · May be fixed by #6955
Open

Failure in system test for io.pravega.test.system.LargeEventTest.largeEventSimpleTest #6924

ShwethaSNayak opened this issue Oct 21, 2022 · 1 comment · May be fixed by #6955
Assignees
@ShwethaSNayak
Labels
tag/flakeyTest Test is not reliable, and causes build errors

Comments

@ShwethaSNayak
Copy link
Contributor

ShwethaSNayak commented Oct 21, 2022 •
edited

Problem description

java.lang.AssertionError: largeEventSimpleTest test failed due to Error with message null
at io.pravega.test.system.framework.services.kubernetes.K8SequentialExecutor.lambda$startTestExecution$4(K8SequentialExecutor.java:94)
at java.base/java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:930)
at java.base/java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:907)
at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506)
at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2073)
at java.base/java.util.concurrent.CompletableFuture.uniAcceptNow(CompletableFuture.java:753)
at java.base/java.util.concurrent.CompletableFuture.uniAcceptStage(CompletableFuture.java:731)
at java.base/java.util.concurrent.CompletableFuture.thenAccept(CompletableFuture.java:2108)

Problem location

System test

Suggestions for an improvement
@ShwethaSNayak ShwethaSNayak added the tag/flakeyTest Test is not reliable, and causes build errors label Oct 21, 2022
@ShwethaSNayak ShwethaSNayak self-assigned this Oct 21, 2022
@RaulGracia
Copy link
Contributor

I have been debugging this issue and added logs to better understand what is going on. At the moment, it looks like the test fails because the LargeEventWriter used in the test gets blocked sometimes at this point:

pravega/client/src/main/java/io/pravega/client/stream/impl/LargeEventWriter.java

Line 187 in 9215909

transformDataAppended(getThrowingException(futures.get(i)), created.getSegment());

That is, the payload is split into the appropriate 8MB appends to the transient Segment, but one or more of these append futures seems to never complete. This leaves the LargeEventWriter blocked and therefore the test fails due to TimeoutException:

2022-11-16T16:28:56.56057617Z stdout F 2022-11-16 16:28:56,560 500468 [Time-limited test] INFO  i.pravega.client.stream.impl.Pinger - Closing Pinger periodic task
2022-11-16T16:28:56.561581462Z stdout F 2022-11-16 16:28:56,561 500469 [Time-limited test] INFO  i.p.c.c.impl.ConnectionPoolImpl - Shutting down connection pool
2022-11-16T16:28:56.561615609Z stdout F 2022-11-16 16:28:56,561 500469 [ClientSocketReaders-35] INFO  i.p.c.c.impl.TcpClientConnection - Closing TcpConnection.Reader because socket is closed.
2022-11-16T16:28:56.561708969Z stdout F 2022-11-16 16:28:56,561 500469 [Time-limited test] INFO  i.p.c.c.i.SocketConnectionFactoryImpl - Shutting down connection factory
2022-11-16T16:28:56.564049102Z stdout F 2022-11-16 16:28:56,563 500471 [main] WARN  i.p.t.s.f.TestFrameworkException - TestFramework Exception. Type: InternalError, Details: Exception while running test method: largeEventSimpleTest
2022-11-16T16:28:56.564081181Z stdout F org.junit.runners.model.TestTimedOutException: test timed out after 500 seconds

Next step is to understand why sometimes of these 8MB append futures never completes (and also does not timeout with an error).

@RaulGracia RaulGracia linked a pull request Nov 17, 2022 that will close this issue
Draft
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@ShwethaSNayak ShwethaSNayak

Labels
tag/flakeyTest Test is not reliable, and causes build errors
Projects
None yet
Milestone
No milestone
2 participants
@RaulGracia @ShwethaSNayak