[CP 2.6] Enabled SPOT Instance deployments on CI benchmarks

[filipecosta90]

CP of #4279 into v2.6 branch.
This PR enables spot instance deployments as a 1st deployment option. If it fails it uses the default non spot deployment in an agnostic manner to the developer/CI runner.

[filipecosta90]

Automated performance analysis summary

This comment was automatically generated given there is performance data available.

In summary:

• Detected a total of 16 stable tests between versions.
• Detected a total of 3 highly unstable benchmarks.
• Detected a total of 12 improvements above the improvement water line.

You can check a comparison in detail via the grafana link

Comparison between master and cp.spot.2.6.

Time Period from 30 days ago. (environment used: oss-standalone)

Test Case	Baseline master (median obs. +- std.dev)	Comparison cp.spot.2.6 (median obs. +- std.dev)	% change (higher-better)	Note
ftsb-10K-enwiki_abstract-hashes-fulltext-sortby	68 +- 1.2% (7 datapoints)	75	9.8%	IMPROVEMENT
ftsb-10K-enwiki_abstract-hashes-term-prefix	7414 +- 1.3% (7 datapoints)	8362	12.8%	IMPROVEMENT
ftsb-10K-enwiki_abstract-hashes-term-suffix	2065 +- 2.7% (7 datapoints)	2004	-3.0%	No Change
ftsb-10K-enwiki_abstract-hashes-term-suffix-withsuffixtrie	65742 +- 4.8% (7 datapoints)	75415	14.7%	IMPROVEMENT
ftsb-10K-enwiki_abstract-hashes-term-wildcard	12128 +- 1.5% (7 datapoints)	13320	9.8%	IMPROVEMENT
ftsb-10K-enwiki_pages-hashes-fulltext-mixed_simple-1word-query_write_1_to_read_20.yml	1240 +- 3.5% (7 datapoints)	1269	2.4%	No Change
ftsb-10K-enwiki_pages-hashes-load	52301 +- 3.2% (7 datapoints)	55260	5.7%	IMPROVEMENT
ftsb-1K-enwiki_abstract-hashes-term-contains	1744 +- 2.5% (7 datapoints)	1733	-0.6%	No Change
ftsb-1M-enwiki_abstract-hashes-fulltext-2word-intersection-query	355 +- 3.3% (7 datapoints)	394	11.1%	IMPROVEMENT
ftsb-1M-enwiki_abstract-hashes-fulltext-2word-intersection-query-non-sortable	43 +- 14.6% UNSTABLE (7 datapoints)	30	-29.9%	UNSTABLE (very high variance)
ftsb-1M-enwiki_abstract-hashes-fulltext-2word-intersection-query@100_ops_sec	100 +- 0.0% (7 datapoints)	100	0.0%	No Change
ftsb-1M-enwiki_abstract-hashes-fulltext-2word-union-query	2633 +- 3.1% (7 datapoints)	2891	9.8%	IMPROVEMENT
ftsb-1M-enwiki_abstract-hashes-fulltext-2word-union-query-non-sortable	1045 +- 7.8% (7 datapoints)	992	-5.0%	waterline=7.8%. potential REGRESSION
ftsb-1M-enwiki_abstract-hashes-fulltext-2word-union-query@100_ops_sec	100 +- 0.0% (7 datapoints)	100	0.0%	No Change
ftsb-1M-enwiki_abstract-hashes-fulltext-simple-1word-query	983 +- 4.8% (7 datapoints)	1096	11.5%	IMPROVEMENT
ftsb-1M-enwiki_abstract-hashes-fulltext-simple-1word-query@100_ops_sec	100 +- 0.0% (7 datapoints)	100	0.0%	No Change
ftsb-1M-enwiki_abstract-hashes-load	23097 +- 2.7% (7 datapoints)	24143	4.5%	potential IMPROVEMENT
ftsb-1M-nyc_taxis-ftadd-load	22062 +- 1.5% (7 datapoints)	22012	-0.2%	No Change
ftsb-1M-nyc_taxis-hashes-load	23148 +- 1.8% (7 datapoints)	23991	3.6%	potential IMPROVEMENT
search-ftsb-10K-enwiki_abstract-hashes-term-withoutsuffix-trie	40127 +- 2.8% (7 datapoints)	46396	15.6%	IMPROVEMENT
search-ftsb-10K-enwiki_abstract-hashes-term-withsuffix-trie	39548 +- 3.0% (7 datapoints)	46467	17.5%	IMPROVEMENT
search-ftsb-1700K-docs-union-iterators-q3	6 +- 1.4% (7 datapoints)	6	3.5%	potential IMPROVEMENT
search-ftsb-1M-enwiki_abstract-hashes-fulltext-2word-intersection-query-non-sortable@50_ops_sec	50 +- 4.5% (7 datapoints)	50	0.5%	No Change
search-ftsb-1M-enwiki_abstract-hashes-fulltext-2word-union-query-non-sortable@100_ops_sec	100 +- 0.0% (7 datapoints)	100	0.0%	No Change
search-ftsb-1M-enwiki_abstract-hashes-fulltext-simple-1word-query-non-sortable	151 +- 2.4% (7 datapoints)	159	4.8%	potential IMPROVEMENT
search-ftsb-370K-docs-union-iterators-q4	7 +- 0.9% (7 datapoints)	7	2.8%	No Change
search-ftsb-5500K-docs-union-iterators-q2	1 +- 1.1% (7 datapoints)	1	1.1%	No Change
search-geo	183 +- 5.1% (7 datapoints)	212	16.1%	waterline=5.1%. IMPROVEMENT
search-numeric	2967 +- 25.1% UNSTABLE (7 datapoints)	7034	137.1%	UNSTABLE (very high variance)
search-numeric-sortby	3217 +- 28.5% UNSTABLE (7 datapoints)	3935	22.3%	UNSTABLE (very high variance)
search-numeric-sortby-desc	2919 +- 2.6% (7 datapoints)	3245	11.2%	IMPROVEMENT

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 82.69%. Comparing base (44bd47e) to head (954233c).
Report is 36 commits behind head on 2.6.

Additional details and impacted files

@@            Coverage Diff             @@
##              2.6    #4281      +/-   ##
==========================================
- Coverage   82.74%   82.69%   -0.06%     
==========================================
  Files         176      176              
  Lines       31017    31017              
==========================================
- Hits        25666    25649      -17     
- Misses       5351     5368      +17     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[GuyAv46]

Here we keep the vecsim cluster benchmarks?

[filipecosta90]

Here we keep the vecsim cluster benchmarks?

@GuyAv46 i've removed the cluster envs from those benchmarks (the ones loading from rdb) given if you load from rdb the cluster benchmark wont run. This means we have 4 benchmarks for vecsim cluster (based on ann) but this PR is not reducing/increasing it.
I'll work on adding more cluster benchmarks that are vecsim related

[github-actions]

This pull request is stale because it has been open for 60 days with no activity.