Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Run benchmarks in CI #6539

Open
Darksonn opened this issue May 5, 2024 · 1 comment
Open

Run benchmarks in CI #6539

Darksonn opened this issue May 5, 2024 · 1 comment
Labels
A-benches Area: Benchmarks A-ci Area: The continuous integration setup E-help-wanted Call for participation: Help is requested to fix this issue. E-medium Call for participation: Experience needed to fix: Medium / intermediate

Comments

@Darksonn
Copy link
Contributor

Darksonn commented May 5, 2024

We don't currently run benchmarks in CI, because doing that without it being really flaky is hard. However, if we used a tool such as iai-callgrind to get consistent measurements using something other than time, then this would become feasible.

Doing this would let us more easily catch performance regressions. Inspired by Towards Impeccable Rust by Jon Gjengset.

@Darksonn Darksonn added E-help-wanted Call for participation: Help is requested to fix this issue. E-medium Call for participation: Experience needed to fix: Medium / intermediate A-ci Area: The continuous integration setup A-benches Area: Benchmarks labels May 5, 2024
@wathenjiang
Copy link
Contributor

Unfortunately, Valgrind's Callgrind tool does not support parallel applications like in production environments. Please refer to: https://valgrind.org/docs/manual/manual-core.html#manual-core.pthreads

This means that once the benchmark of the application is related to multithreading, using the Callgrind tool may not provide valuable benchmark results. Unfortunately, currently most benchmark tests in the benches/ are under multi threads.

I think before considering whether iai-callgrind should be introduced into CI, we should first identify which code blocks are suitable for Callgrind based benchmark.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-benches Area: Benchmarks A-ci Area: The continuous integration setup E-help-wanted Call for participation: Help is requested to fix this issue. E-medium Call for participation: Experience needed to fix: Medium / intermediate
Projects
None yet
Development

No branches or pull requests

2 participants