Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[compiler v2] Emulating some unexpected v1 borrow behavior #13314

Merged
merged 1 commit into from
May 22, 2024
Merged

Conversation

wrwg
Copy link
Contributor

@wrwg wrwg commented May 17, 2024

Description

Fixes #12781

This also refactors and simplifies the releasing of references before write operations. I initially thought this was the problem but it wasn't, anyway the refactoring simplifies the code a bit.

Type of Change

  • New feature
  • Bug fix
  • Breaking change
  • Performance improvement
  • Refactoring
  • Dependency update
  • Documentation update
  • Tests

Which Components or Systems Does This Change Impact?

  • Validator Node
  • Full Node (API, Indexer, etc.)
  • Move/Aptos Virtual Machine
  • Aptos Framework
  • Aptos CLI/SDK
  • Developer Infrastructure
  • Other (specify)

How Has This Been Tested?

New baseline test

Copy link

trunk-io bot commented May 17, 2024

⏱️ 10h 52m total CI duration on this PR
Job Cumulative Duration Recent Runs
rust-targeted-unit-tests 1h 32m 🟩🟩🟩🟩
windows-build 1h 18m 🟩🟩
rust-smoke-tests 1h 5m 🟥🟩
rust-move-unit-coverage 59m 🟩🟩🟩🟩
rust-move-tests 52m 🟩🟩🟩 (+1 more)
execution-performance / single-node-performance 47m 🟩🟩
forge-framework-upgrade-test / forge 32m 🟩🟩
rust-lints 32m 🟩🟩🟩🟩
rust-images / rust-all 30m 🟩🟩
cli-e2e-tests / run-cli-tests 30m 🟥🟥🟩
forge-e2e-test / forge 28m 🟩🟩
forge-compat-test / forge 27m 🟩🟩
run-tests-main-branch 22m 🟩🟩🟩🟩🟩
rust-build-cached-packages 12m 🟩🟩
check-dynamic-deps 9m 🟩🟩🟩🟩🟩 (+2 more)
check 8m 🟩🟩
general-lints 8m 🟩🟩🟩🟩
test-target-determinator 6m 🟩🟩
execution-performance / test-target-determinator 6m 🟩🟩
semgrep/ci 3m 🟩🟩🟩🟩🟩 (+2 more)
node-api-compatibility-tests / node-api-compatibility-tests 2m 🟩🟩
file_change_determinator 58s 🟩🟩🟩🟩 (+1 more)
file_change_determinator 51s 🟩🟩🟩🟩 (+1 more)
permission-check 27s 🟩🟩🟩🟩🟩 (+2 more)
permission-check 26s 🟩🟩🟩🟩🟩 (+2 more)
file_change_determinator 23s 🟩🟩
permission-check 18s 🟩🟩🟩🟩🟩 (+1 more)
permission-check 18s 🟩🟩🟩🟩🟩 (+1 more)
permission-check 4s 🟩🟩
determine-docker-build-metadata 4s 🟩🟩

🚨 5 jobs on the last run were significantly faster/slower than expected

Job Duration vs 7d avg Delta
rust-move-tests 16m 9m +82%
cli-e2e-tests / run-cli-tests 12m 7m +78%
rust-build-cached-packages 8m 5m +57%
rust-lints 10m 7m +51%
rust-move-unit-coverage 13m 18m -27%

settingsfeedbackdocs ⋅ learn more about trunk.io

Copy link

codecov bot commented May 17, 2024

Codecov Report

Attention: Patch coverage is 0% with 42 lines in your changes are missing coverage. Please review.

Project coverage is 32.9%. Comparing base (4da21b9) to head (083bd58).
Report is 1 commits behind head on main.

Files Patch % Lines
...iler-v2/src/pipeline/reference_safety_processor.rs 0.0% 42 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff            @@
##             main   #13314     +/-   ##
=========================================
- Coverage    33.0%    32.9%   -0.1%     
=========================================
  Files        1768     1768             
  Lines      339266   339263      -3     
=========================================
- Hits       111995   111840    -155     
- Misses     227271   227423    +152     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Copy link
Contributor

@rahxephon89 rahxephon89 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thanks!

// conditions and produce an error ad-hoc.
if is_mut
&& !self.alive.after.contains_key(&dest)
//&& self.state.children(&label).any(|e| e.target != child)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe remove this commented code?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done


// Preprocessing: check borrow safety of the currently active borrow graph for read ref,
// write ref, and function calls.
// Preprocessing: check borrow safety of the currently active borrow graph for
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since release_refs_not_alive_after is called after handling this instr, this is not needed anymore?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It was always like a defensive meaure 'if something has been not release by now, do it now', but it should not be required as releasing happens (before and after) at the end of each instruction, with or without this new function. I removed it and verified everything works like before.

The difference is really that the cleanup at the end and what you did with your 'release before write' is now unified in release_refs_not_alive_after. The special check whether the write destination is not a ref is also not longer needed, by doing the releasing after the output graph has been created.

@brmataptos brmataptos changed the title [compiler v2] Simulating some unexpected v1 borrow behavior [compiler v2] Emulating some unexpected v1 borrow behavior May 21, 2024
Copy link
Contributor

@brmataptos brmataptos left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I suggest adding an "idempotent" comment to the function. Also, I renamed your PR, as discussed in slack.

}
}

// Some instructions may not have release inputs, do so now. The operation
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

release --> released

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

And if this is operation is idempotent, you should document that on the function, not so necessary here.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done both

@wrwg wrwg force-pushed the wrwg/borrow branch 2 times, most recently from 8439986 to cc88ce9 Compare May 22, 2024 05:03
@wrwg wrwg enabled auto-merge (squash) May 22, 2024 05:03

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

Fixes #12781

This also refactors and simplifies the releasing of references before write operations. I initially thought this was the problem but it wasn't, anyway the refactoring simplifies the code a bit.

This comment has been minimized.

This comment has been minimized.

This comment has been minimized.

Copy link
Contributor

✅ Forge suite compat success on 3ffe0986b5fe4acb76544ae7ae85d73b91a6a411 ==> 083bd58443479c89796c74555f3681c24c3723d3

Compatibility test results for 3ffe0986b5fe4acb76544ae7ae85d73b91a6a411 ==> 083bd58443479c89796c74555f3681c24c3723d3 (PR)
1. Check liveness of validators at old version: 3ffe0986b5fe4acb76544ae7ae85d73b91a6a411
compatibility::simple-validator-upgrade::liveness-check : committed: 6801.522066260085 txn/s, latency: 4836.229302859016 ms, (p50: 4800 ms, p90: 8000 ms, p99: 8500 ms), latency samples: 244140
2. Upgrading first Validator to new version: 083bd58443479c89796c74555f3681c24c3723d3
compatibility::simple-validator-upgrade::single-validator-upgrade : committed: 1487.831750092359 txn/s, latency: 20783.535725885027 ms, (p50: 27500 ms, p90: 29600 ms, p99: 30200 ms), latency samples: 61580
3. Upgrading rest of first batch to new version: 083bd58443479c89796c74555f3681c24c3723d3
compatibility::simple-validator-upgrade::half-validator-upgrade : committed: 2923.3145600531766 txn/s, latency: 10609.058182853425 ms, (p50: 9800 ms, p90: 14200 ms, p99: 14500 ms), latency samples: 122940
4. upgrading second batch to new version: 083bd58443479c89796c74555f3681c24c3723d3
compatibility::simple-validator-upgrade::rest-validator-upgrade : committed: 6465.42605961615 txn/s, latency: 5080.684569450455 ms, (p50: 4800 ms, p90: 8200 ms, p99: 9300 ms), latency samples: 231100
5. check swarm health
Compatibility test for 3ffe0986b5fe4acb76544ae7ae85d73b91a6a411 ==> 083bd58443479c89796c74555f3681c24c3723d3 passed
Test Ok

Copy link
Contributor

✅ Forge suite realistic_env_max_load success on 083bd58443479c89796c74555f3681c24c3723d3

two traffics test: inner traffic : committed: 7864.801220546377 txn/s, latency: 4968.654083386116 ms, (p50: 4800 ms, p90: 6900 ms, p99: 11100 ms), latency samples: 3410160
two traffics test : committed: 99.98810701091804 txn/s, latency: 1876.5604395604396 ms, (p50: 1800 ms, p90: 2100 ms, p99: 4800 ms), latency samples: 1820
Latency breakdown for phase 0: ["QsBatchToPos: max: 0.216, avg: 0.204", "QsPosToProposal: max: 0.272, avg: 0.258", "ConsensusProposalToOrdered: max: 0.449, avg: 0.420", "ConsensusOrderedToCommit: max: 0.386, avg: 0.373", "ConsensusProposalToCommit: max: 0.802, avg: 0.793"]
Max round gap was 1 [limit 4] at version 1636635. Max no progress secs was 4.66593 [limit 15] at version 1636635.
Test Ok

Copy link
Contributor

✅ Forge suite framework_upgrade success on 3ffe0986b5fe4acb76544ae7ae85d73b91a6a411 ==> 083bd58443479c89796c74555f3681c24c3723d3

Compatibility test results for 3ffe0986b5fe4acb76544ae7ae85d73b91a6a411 ==> 083bd58443479c89796c74555f3681c24c3723d3 (PR)
Upgrade the nodes to version: 083bd58443479c89796c74555f3681c24c3723d3
framework_upgrade::framework-upgrade::full-framework-upgrade : committed: 1063.5901398338751 txn/s, submitted: 1066.0034012897688 txn/s, failed submission: 2.413261455893694 txn/s, expired: 2.413261455893694 txn/s, latency: 2969.9663056930694 ms, (p50: 2100 ms, p90: 5100 ms, p99: 11600 ms), latency samples: 96960
framework_upgrade::framework-upgrade::full-framework-upgrade : committed: 1100.5844179163958 txn/s, submitted: 1102.9512661269687 txn/s, failed submission: 2.3668482105728943 txn/s, expired: 2.3668482105728943 txn/s, latency: 2898.681677419355 ms, (p50: 2100 ms, p90: 5200 ms, p99: 9000 ms), latency samples: 93000
5. check swarm health
Compatibility test for 3ffe0986b5fe4acb76544ae7ae85d73b91a6a411 ==> 083bd58443479c89796c74555f3681c24c3723d3 passed
Upgrade the remaining nodes to version: 083bd58443479c89796c74555f3681c24c3723d3
framework_upgrade::framework-upgrade::full-framework-upgrade : committed: 1152.3768098203454 txn/s, submitted: 1153.313130608337 txn/s, failed submission: 0.936320787991343 txn/s, expired: 0.936320787991343 txn/s, latency: 2828.4813426772293 ms, (p50: 2100 ms, p90: 5100 ms, p99: 9000 ms), latency samples: 98460
Test Ok

@wrwg wrwg merged commit e93cd69 into main May 22, 2024
54 checks passed
@wrwg wrwg deleted the wrwg/borrow branch May 22, 2024 15:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Bug][Compiler-v2] BORROWFIELD_EXISTS_MUTABLE_BORROW_ERROR raised in factor_invalid_2
3 participants