Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Linux framework_tests_misc is 2.02% flaky #148393

Closed
fluttergithubbot opened this issue May 15, 2024 · 3 comments · Fixed by #148803
Closed

Linux framework_tests_misc is 2.02% flaky #148393

fluttergithubbot opened this issue May 15, 2024 · 3 comments · Fixed by #148803
Assignees
Labels
c: flake Tests that sometimes, but not always, incorrectly pass f: integration_test The flutter/packages/integration_test plugin P0 Critical issues such as a build break or regression team-framework Owned by Framework team

Comments

@fluttergithubbot
Copy link
Contributor

The post-submit test builder Linux framework_tests_misc had a flaky ratio 2.02% for the past (up to) 100 commits, which is above our 2.00% threshold.

One recent flaky example for a same commit: https://ci.chromium.org/ui/p/flutter/builders/prod/Linux%20framework_tests_misc/17221
Commit: 4c567a9

Flaky builds:
https://ci.chromium.org/ui/p/flutter/builders/prod/Linux%20framework_tests_misc/17221
https://ci.chromium.org/ui/p/flutter/builders/prod/Linux%20framework_tests_misc/17211

Recent test runs:
https://flutter-dashboard.appspot.com/#/build?taskFilter=Linux%20framework_tests_misc

Please follow https://github.com/flutter/flutter/wiki/Reducing-Test-Flakiness#fixing-flaky-tests to fix the flakiness and enable the test back after validating the fix (internal dashboard to validate: go/flutter_test_flakiness).

@fluttergithubbot fluttergithubbot added c: flake Tests that sometimes, but not always, incorrectly pass P0 Critical issues such as a build break or regression team-framework Owned by Framework team labels May 15, 2024
@goderbauer goderbauer added the f: integration_test The flutter/packages/integration_test plugin label May 21, 2024
@goderbauer
Copy link
Member

Looks like in both cases a native android test for package:integration_test is failing.

RUNNING: cd packages/integration_test/example/android; gradlew :integration_test:testDebugUnitTest --tests dev.flutter.plugins.integration_test.FlutterDeviceScreenshotTest
workingDirectory: /b/s/w/ir/x/w/flutter/packages/integration_test/example/android, executable: /b/s/w/ir/x/w/flutter/packages/integration_test/example/android/gradlew, arguments: [:integration_test:testDebugUnitTest, --tests, dev.flutter.plugins.integration_test.FlutterDeviceScreenshotTest]
Downloading https://services.gradle.org/distributions/gradle-8.3-all.zip
............................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................
Unzipping /b/s/w/ir/cache/gradle/wrapper/dists/gradle-8.3-all/6en3ugtfdg5xnpx44z4qbwgas/gradle-8.3-all.zip to /b/s/w/ir/cache/gradle/wrapper/dists/gradle-8.3-all/6en3ugtfdg5xnpx44z4qbwgas
Set executable permissions for: /b/s/w/ir/cache/gradle/wrapper/dists/gradle-8.3-all/6en3ugtfdg5xnpx44z4qbwgas/gradle-8.3/bin/gradle

Welcome to Gradle 8.3!

Here are the highlights of this release:
 - Faster Java compilation
 - Reduced memory usage
 - Support for running on Java 20

For more details see https://docs.gradle.org/8.3/release-notes.html

To honour the JVM settings for this build a single-use Daemon process will be forked. For more on this, please refer to https://docs.gradle.org/8.3/userguide/gradle_daemon.html#sec:disabling_the_daemon in the Gradle documentation.
Daemon will be stopped at the end of the build 
> Task :gradle:compileJava NO-SOURCE
> Task :gradle:compileGroovy
> Task :gradle:pluginDescriptors
> Task :gradle:processResources
> Task :gradle:classes
> Task :gradle:jar

> Configure project :app
w: file:///b/s/w/ir/x/w/flutter/packages/flutter_tools/gradle/src/main/kotlin/dependency_version_checker.gradle.kts:107:40: Variable 'agpVersion' initializer is redundant
w: file:///b/s/w/ir/x/w/flutter/packages/flutter_tools/gradle/src/main/kotlin/dependency_version_checker.gradle.kts:108:40: Variable 'kgpVersion' initializer is redundant
w: file:///b/s/w/ir/x/w/flutter/packages/flutter_tools/gradle/src/main/kotlin/dependency_version_checker.gradle.kts:143:28: Parameter 'project' is never used
w: file:///b/s/w/ir/x/w/flutter/packages/flutter_tools/gradle/src/main/kotlin/dependency_version_checker.gradle.kts:152:40: Variable 'agpVersion' initializer is redundant
w: file:///b/s/w/ir/x/w/flutter/packages/flutter_tools/gradle/src/main/kotlin/dependency_version_checker.gradle.kts:167:55: 'Version' is deprecated. Deprecated in Java
w: file:///b/s/w/ir/x/w/flutter/packages/flutter_tools/gradle/src/main/kotlin/dependency_version_checker.gradle.kts:195:56: Unnecessary non-null assertion (!!) on a non-null receiver of type Any
w: file:///b/s/w/ir/x/w/flutter/packages/flutter_tools/gradle/src/main/kotlin/dependency_version_checker.gradle.kts:344:28: The corresponding parameter in the supertype 'Comparable' is named 'other'. This may cause problems when calling this function with named arguments.
Warning: unable to detect project KGP version. Skipping version checking. 
This may be because you have applied KGP after the Flutter Gradle Plugin.
WARNING: We recommend using a newer Android Gradle plugin to use compileSdk = 34

This Android Gradle plugin (8.1.0) was tested up to compileSdk = 33 (and compileSdkPreview = "UpsideDownCakePrivacySandbox").

You are strongly encouraged to update your project to use a newer
Android Gradle plugin that has been tested with compileSdk = 34.

If you are already using the latest version of the Android Gradle plugin,
you may need to wait until a newer version with support for compileSdk = 34 is available.

To suppress this warning, add/update
    android.suppressUnsupportedCompileSdk=34
to this project's gradle.properties.
Errors during XML parse:
Additionally, the fallback loader failed to parse the XML.
Checking the license for package Android SDK Build-Tools 33.0.1 in /b/s/w/ir/cache/android/sdk/licenses
License for package Android SDK Build-Tools 33.0.1 accepted.
Preparing "Install Android SDK Build-Tools 33.0.1 v.33.0.1".
"Install Android SDK Build-Tools 33.0.1 v.33.0.1" ready.
Installing Android SDK Build-Tools 33.0.1 in /b/s/w/ir/cache/android/sdk/build-tools/33.0.1
"Install Android SDK Build-Tools 33.0.1 v.33.0.1" complete.
"Install Android SDK Build-Tools 33.0.1 v.33.0.1" finished.

> Task :integration_test:preBuild UP-TO-DATE
> Task :integration_test:preDebugBuild UP-TO-DATE
> Task :integration_test:generateDebugBuildConfig
> Task :integration_test:generateDebugResValues
> Task :integration_test:generateDebugResources
> Task :integration_test:packageDebugResources
> Task :integration_test:preDebugUnitTestBuild UP-TO-DATE
> Task :integration_test:javaPreCompileDebug
> Task :integration_test:javaPreCompileDebugUnitTest
> Task :integration_test:processDebugJavaRes NO-SOURCE
> Task :integration_test:processDebugUnitTestJavaRes NO-SOURCE
> Task :integration_test:parseDebugLocalResources

> Task :integration_test:processDebugManifest
package="dev.flutter.integration_test" found in source AndroidManifest.xml: /b/s/w/ir/x/w/flutter/packages/integration_test/android/src/main/AndroidManifest.xml.
Setting the namespace via the package attribute in the source AndroidManifest.xml is no longer supported, and the value is ignored.
Recommendation: remove package="dev.flutter.integration_test" from the source AndroidManifest.xml: /b/s/w/ir/x/w/flutter/packages/integration_test/android/src/main/AndroidManifest.xml.

> Task :integration_test:generateDebugRFile
> Task :integration_test:compileDebugJavaWithJavac
> Task :integration_test:bundleLibRuntimeToJarDebug
> Task :integration_test:bundleLibCompileToJarDebug
> Task :integration_test:generateDebugUnitTestStubRFile
> Task :integration_test:compileDebugUnitTestJavaWithJavac

> Task :integration_test:testDebugUnitTest

dev.flutter.plugins.integration_test.FlutterDeviceScreenshotTest > getFlutterView_returnsFlutterViewForFlutterActivity FAILED
    java.lang.AssertionError at MavenArtifactFetcher.java:129
        Caused by: java.util.concurrent.ExecutionException at AbstractFuture.java:588
            Caused by: java.net.SocketException at NioSocketImpl.java:323

OpenJDK 64-Bit Server VM warning: Sharing is only supported for boot loader classes because bootstrap classpath has been appended

> Task :integration_test:testDebugUnitTest FAILED

3 tests completed, 1 failed

FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':integration_test:testDebugUnitTest'.
> There were failing tests. See the report at: file:///b/s/w/ir/x/w/flutter/packages/integration_test/example/build/integration_test/reports/tests/testDebugUnitTest/index.html

* Try:
> Run with --scan to get full insights.

BUILD FAILED in 2m 41s
18 actionable tasks: 18 executed
╔═╡ERROR #1╞════════════════════════════════════════════════════════════════════
║ Command: gradlew :integration_test:testDebugUnitTest --tests dev.flutter.plugins.integration_test.FlutterDeviceScreenshotTest
║ Command exited with exit code 1 but expected zero exit code.
║ Working directory: /b/s/w/ir/x/w/flutter/packages/integration_test/example/android
╚═══════════════════════════════════════════════════════════════════════════════

@goderbauer
Copy link
Member

@gmackall I believe you added this test in #132406. Any chance you could take a look at its flakiness?

@gmackall
Copy link
Member

Yep, I'll take a look

@gmackall gmackall self-assigned this May 21, 2024
auto-submit bot pushed a commit that referenced this issue May 22, 2024
May fix #148393.

There are reports on the robolectric issue tracker of >`4.10.3` causing the error we are seeing
robolectric/robolectric#8158
The test doesn't actually need robolectric (and didn't originally use it, or flake before using it), but it was added in #144348. We can safely remove it by mocking the call to `View.generateViewId()`, so try removing it and see if flakes go away.
victorsanni pushed a commit to victorsanni/flutter that referenced this issue May 31, 2024
May fix flutter#148393.

There are reports on the robolectric issue tracker of >`4.10.3` causing the error we are seeing
robolectric/robolectric#8158
The test doesn't actually need robolectric (and didn't originally use it, or flake before using it), but it was added in flutter#144348. We can safely remove it by mocking the call to `View.generateViewId()`, so try removing it and see if flakes go away.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
c: flake Tests that sometimes, but not always, incorrectly pass f: integration_test The flutter/packages/integration_test plugin P0 Critical issues such as a build break or regression team-framework Owned by Framework team
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants