chore: add flaky retry in weekly CI #662

RomanBredehoft · 2024-04-30T15:34:57Z

Clean the weekly CI :

new workflow : scheduled + on demand , with dedicated status history
re-run flaky issues like in regular PRs (+ report them on slack notifications for weeklys)
remove slack reports for regular CIs (keep for weekly + push on main/release branches)
better slack report message (ex: https://zama-ai.slack.com/archives/C02RX6CF3FV/p1715090858656719)

refs https://github.com/zama-ai/concrete-ml-internal/issues/4030
refs https://github.com/zama-ai/concrete-ml-internal/issues/4428
closes https://github.com/zama-ai/concrete-ml-internal/issues/4390

RomanBredehoft · 2024-04-30T15:36:05Z

.github/workflows/continuous-integration.yaml

@@ -936,7 +952,7 @@ jobs:
    needs: [build-linux]
    runs-on: ubuntu-20.04
    timeout-minutes: 2
-    if: ${{ always() }}
+    if: success() || failure()


avoid running this if all previous jobs are skipped

RomanBredehoft · 2024-04-30T15:36:27Z

.github/workflows/continuous-integration.yaml

-      - name: PyTest (no flaky) with PyPI local wheel of Concrete ML (release)
+      # Run Pytest on all of our tests (except flaky ones) using PyPI's local wheel in the weekly 
+      # or during the release process
+      - name: PyTest (no flaky) with PyPI local wheel of Concrete ML (weekly, release)


avoid flaky tests in weekly tests with pypi as well (felt redundant)

.github/workflows/continuous-integration.yaml

github-actions · 2024-05-03T17:41:13Z

⚠️ Known flaky tests have been rerun ⚠️

One or several tests initially failed but were identified as known flaky. tests. Therefore, they have been rerun and passed. See below for more details.

Failed tests details

Known flaky tests that initially failed:

tests/torch/test_compile_torch.py::test_compile_torch_or_onnx_conv_networks[True-True-CNN_grouped-relu]

RomanBredehoft · 2024-05-07T14:25:03Z

.github/workflows/continuous-integration.yaml

@@ -60,12 +58,6 @@ on:
        type: boolean
        required: false
        default: false
-
-  schedule:


moved to the new weekly_tests.yaml

RomanBredehoft · 2024-05-07T14:26:36Z

.github/workflows/continuous-integration.yaml

-  IS_WEEKLY: ${{ github.event_name == 'schedule' || ((github.event_name == 'workflow_dispatch') && (inputs.event_name == 'weekly')) }}
+  # Run the weekly CI if it has been triggered manually by the weekly workflow, meaning 
+  # 'inputs.event_name' is set to "weekly"
+  IS_WEEKLY: ${{ inputs.event_name == 'weekly'}}


now weekly CIs are only triggerd by the new weekly_tests.yaml CI

RomanBredehoft · 2024-05-07T14:27:13Z

.github/workflows/continuous-integration.yaml

@@ -147,7 +144,7 @@ jobs:
          # Manage instance type
          INSTANCE_TYPE="c5.4xlarge"
          if [[ "${BUILD_TYPE}" == "weekly" ]]; then
-            INSTANCE_TYPE="m6i.metal"
+            INSTANCE_TYPE="m6i.16xlarge"


as for the release, m6i.metal started to be less available then before

RomanBredehoft · 2024-05-07T14:27:51Z

.github/workflows/continuous-integration.yaml

          )
          && steps.conformance.outcome == 'success' 
          && !cancelled()
        shell: bash +e {0}
        run: |
-          make pytest_and_report
+          if [[ "${{ env.IS_WEEKLY }}" == "true" ]]; then


run weekly tests in weekly CIs, but keep the same flaky re-run system

RomanBredehoft · 2024-05-07T14:29:19Z

.github/workflows/continuous-integration.yaml

@@ -951,20 +961,8 @@ jobs:
            exit 1
          fi

-      - name: Slack Notification


keep a single slack notif at the end of the whole CI

RomanBredehoft · 2024-05-07T14:30:44Z

.github/workflows/continuous-integration.yaml

      - name: PyTest Source Code
        run: |
          make pytest_macOS_for_GitHub

+  decide-slack-report:


only report on slack :

weekly CIs

push to main/release branch CIs

we can discuss about it but I feel it's a bit useless to report all regular CIs on slack as it makes things very verbose and not very easy to follow

RomanBredehoft · 2024-05-07T14:31:38Z

.github/workflows/continuous-integration.yaml

+          for linux_python_version in ${LINUX_PYTHON_VERSIONS}; do
+            file_name="failed_tests_slack_list_${linux_python_version}.txt"
+
+            if [ -f ${file_name} ]; then


print flaky tests that got re-run if they were some in the slack notification

RomanBredehoft · 2024-05-07T14:32:32Z

.github/workflows/weekly_tests.yaml

@@ -0,0 +1,22 @@
+name: Weekly Tests


new workflow that only handles weekly CIs, which run:

each sunday night

on demand using GH's interface

we will be able to find all weekly CI's status history much more easily (weeklys won't be mixed up with regular CIs)

RomanBredehoft · 2024-05-07T14:34:16Z

.github/workflows/continuous-integration.yaml

@@ -724,7 +735,8 @@ jobs:
          	poetry run python ./script/actions_utils/pytest_failed_test_report.py \
            --pytest-input-report "pytest_report.json" \
            --failed-tests-report "failed_tests_report.json" \
-            --failed-tests-comment "failed_tests_comment.txt"
+            --failed-tests-comment "failed_tests_comment_${{ matrix.python_version }}.txt"
+            --failed-tests-list "failed_tests_slack_list_${{ matrix.python_version }}.txt"


write the list of failed tests that will later be used for reporting in slack flaky tests that got re-run within a weekly CI

github-actions · 2024-05-07T15:45:24Z

Coverage passed ✅

Coverage details

---------- coverage: platform linux, python 3.8.18-final-0 -----------
Name    Stmts   Miss  Cover   Missing
-------------------------------------
TOTAL    7582      0   100%

59 files skipped due to complete coverage.

cla-bot bot added the cla-signed label Apr 30, 2024

RomanBredehoft force-pushed the chore/add_flaky_retry_in_weekly_ci_4390 branch from fbee1a7 to e25ceff Compare April 30, 2024 15:35

RomanBredehoft commented Apr 30, 2024

View reviewed changes

.github/workflows/continuous-integration.yaml Outdated Show resolved Hide resolved

RomanBredehoft force-pushed the chore/add_flaky_retry_in_weekly_ci_4390 branch 13 times, most recently from 2460879 to e5be845 Compare May 3, 2024 15:08

RomanBredehoft added 11 commits May 6, 2024 13:41

chore: add flaky retry in weekly CI

17962fd

chore: debug CI

c08bbb2

chore: fix CI

3a0437b

chore: debug CI

f899c60

chore: add better build names

740f824

chore: improve if condition for slack report

b316ab4

chore: add upload/dowload flaky tests list artiftacts

56abc68

chore: create new weekly workflow

4826125

chore: debug CI

0662dfb

chore: fix CI

7bd5ee5

chore: debug CI

5293469

RomanBredehoft added 6 commits May 6, 2024 13:41

chore: removemost debug comments

e7cb14f

chore: debug CI

741b1b3

chore: fix CI

34db22b

chore: skip macos

45f7ede

chore: fix skip macos

48fd2d8

chore: enable weekly tests

f234521

RomanBredehoft force-pushed the chore/add_flaky_retry_in_weekly_ci_4390 branch from 18601fb to f234521 Compare May 6, 2024 11:41

RomanBredehoft added 2 commits May 7, 2024 11:34

chore: remove unwanted changes in FHE training notebook

6fcdddc

chore: update AWS machine

be1a1b7

RomanBredehoft commented May 7, 2024

View reviewed changes

chore: clean CI

6e86840

RomanBredehoft force-pushed the chore/add_flaky_retry_in_weekly_ci_4390 branch from f4ec792 to 6e86840 Compare May 7, 2024 14:26

RomanBredehoft commented May 7, 2024

View reviewed changes

RomanBredehoft marked this pull request as ready for review May 7, 2024 14:36

RomanBredehoft requested a review from a team as a code owner May 7, 2024 14:36

RomanBredehoft marked this pull request as draft May 7, 2024 16:06

andrei-stoian-zama approved these changes May 21, 2024

View reviewed changes

RomanBredehoft marked this pull request as ready for review May 23, 2024 09:49

RomanBredehoft merged commit bf47498 into main May 23, 2024
12 checks passed

RomanBredehoft deleted the chore/add_flaky_retry_in_weekly_ci_4390 branch May 23, 2024 09:49

This was referenced May 27, 2024

chore: fix flaky re-run step in CI #698

Merged

chore: fix multiline slack report for weekly CI #706

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

chore: add flaky retry in weekly CI #662

chore: add flaky retry in weekly CI #662

RomanBredehoft commented Apr 30, 2024 •

edited

RomanBredehoft Apr 30, 2024

RomanBredehoft Apr 30, 2024 •

edited

github-actions bot commented May 3, 2024

Known flaky tests that initially failed:

RomanBredehoft May 7, 2024

RomanBredehoft May 7, 2024

RomanBredehoft May 7, 2024

RomanBredehoft May 7, 2024

RomanBredehoft May 7, 2024

RomanBredehoft May 7, 2024 •

edited

RomanBredehoft May 7, 2024

RomanBredehoft May 7, 2024 •

edited

RomanBredehoft May 7, 2024

github-actions bot commented May 7, 2024

chore: add flaky retry in weekly CI #662

chore: add flaky retry in weekly CI #662

Conversation

RomanBredehoft commented Apr 30, 2024 • edited

Choose a reason for hiding this comment

RomanBredehoft Apr 30, 2024 • edited

Choose a reason for hiding this comment

github-actions bot commented May 3, 2024

⚠️ Known flaky tests have been rerun ⚠️

Known flaky tests that initially failed:

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

RomanBredehoft May 7, 2024 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

RomanBredehoft May 7, 2024 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

github-actions bot commented May 7, 2024

Coverage passed ✅

RomanBredehoft commented Apr 30, 2024 •

edited

RomanBredehoft Apr 30, 2024 •

edited

RomanBredehoft May 7, 2024 •

edited

RomanBredehoft May 7, 2024 •

edited