Add more unittest #304
Add more unittest #304
Conversation
pan-x-c
commented
Apr 24, 2024
pan-x-c
commented
- setup local actions runner on a GPU machine
- use docker-compose to setup a cluster
- add more unit tests
- single process
- multi process
- multi process with GPU
- ray with CPU
- ray with GPU
|
This PR is marked as stale because there has been no activity for 21 days. Remove stale label or add new comments or this PR will be closed in 3 day. |
| capabilities: [gpu] | ||
|
|
||
| ray-worker: | ||
| image: data-juicer-unittest:0.2.0 |
There was a problem hiding this comment.
In the future, we may change it into data-juicer-unittest:latest and maintain a latest image
| from data_juicer.utils.registry import Registry | ||
|
|
||
| SKIPPED_TESTS = Registry('SkippedTests') | ||
|
|
||
|
|
||
| def TEST_TAG(*tags): |
There was a problem hiding this comment.
There are two type of tags, while the PR's description mentioned 5 types of test scenarios. So what is the mapping relationship?
single process
multi process
multi process with GPU
ray with CPU
ray with GPU
There was a problem hiding this comment.
For now, there is no distinction between CPU and GPU testing, and the corresponding configuration will be added in the future when needed.
|
|
||
| jobs: | ||
| unittest-single: | ||
| runs-on: [self-hosted, linux] |
There was a problem hiding this comment.
By now, does our self-hosted machine satisfy the requirements from security colleagues?
