-
Notifications
You must be signed in to change notification settings - Fork 3.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat(test runner): Use timing file to balance shards #30388
feat(test runner): Use timing file to balance shards #30388
Conversation
@microsoft-github-policy-service agree company="Artlist" |
This comment has been minimized.
This comment has been minimized.
Test results for "tests 1"1 flaky26922 passed, 628 skipped Merge workflow run. |
Hi @ofirpardo-artlist |
Archiving this |
Was this closed due to inactivity? Or due to other reasons? |
These usually start with filing an issue and figuring out the scope of the feature, what's feasible, edge cases like sharding, merging, CI, etc. This is a big feature and it is unlikely that the contribution will be in a form of a PR. |
I understand that, but there is an open issue that's completely inactive and was created 2 years, so lets either make it active somehow by trying to answer the unknowns you mentioned, or find some other solution perhaps to see how we could follow the steps necessary. Hope to see it continues, thanks :) |
For what it's worth, it seems pretty disrespectful of author's time and effort to just ignore the work and then close the PR a few years later with no context or explanation. I understand that there's probably procedures that need to be gone through to get a fairly big piece of work like this shipped (and equally, I know managing an open source project is not free or easy!), but it'd be good to set that expectation at the start of the process. |
This feature came from this issue: #17969
The idea is to use the JSON reporter and extract the duration of each test, then map the test groups based on the duration so the shards are all as equal as possible, the amount of tests per shard is not relevant here, only the duration of the tests.
If there are tests that are not recorded in the JSON report they will be distributed like they currently do based on the amount.
This is still in a POC state to get some feedback, but the functionality works well.
Some open questions:
At the end of the day unless you come up with some cloud solution, the 'infra' for this feature would need to be done mostly manually, a static JSON report is optional but might get irrelevant very fast.
On my project I've managed it by averaging the durations of the last 10 successful runs, which is being updated to a CDN from CI.
Some more details on current POC can be seen here:
#17969 (comment)
#17969 (comment)