Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[New Engine] Add flow run pause handling #13264

Merged
merged 10 commits into from
May 15, 2024
Merged

Conversation

desertaxle
Copy link
Member

@desertaxle desertaxle commented May 6, 2024

Adds handling for paused flow run to the new engine. Changes to the new engine are relatively small, and this PR mostly contains tests for the new verifying that pausing flow runs works as expected.

Example

Checklist

  • This pull request references any related issue by including "closes <link to issue>"
    • If no issue exists and your change is not a small fix, please create an issue first.
  • If this pull request adds new functionality, it includes unit tests that cover the changes
  • This pull request includes a label categorizing the change e.g. maintenance, fix, feature, enhancement, docs.

For documentation changes:

  • This pull request includes redirect settings in netlify.toml for files that are removed or renamed.

For new functions or classes in the Python SDK:

  • This pull request includes helpful docstrings.
  • If a new Python file was added, this pull request contains a stub page in the Python SDK docs and an entry in mkdocs.yml navigation.

Copy link

netlify bot commented May 6, 2024

Deploy Preview for prefect-docs-preview ready!

Name Link
🔨 Latest commit a1d0490
🔍 Latest deploy log https://app.netlify.com/sites/prefect-docs-preview/deploys/6642b445cce6bf000873ad8a
😎 Deploy Preview https://deploy-preview-13264--prefect-docs-preview.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.

To edit notification comments on pull requests, go to your Netlify site configuration.

@desertaxle desertaxle added the experimental Related to an experimental feature label May 6, 2024
@desertaxle desertaxle marked this pull request as ready for review May 6, 2024 21:13
@desertaxle desertaxle requested a review from a team as a code owner May 6, 2024 21:13
@desertaxle desertaxle requested review from cicdw and jlowin May 6, 2024 21:13
Copy link
Member

@cicdw cicdw left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think a (more?) common pattern is pausing from within the body of a flow - that seems like it would require analogous changes to the new flow run engine yea?

@desertaxle
Copy link
Member Author

I think a (more?) common pattern is pausing from within the body of a flow - that seems like it would require analogous changes to the new flow run engine yea?

AFAICT, pause_flow_run is blocking and doesn't require any changes to the flow engine to keep the flow from executing. suspend_flow_run raises a Pause signal, which bubbles up through the flow run engine and is caught by the entry point we use for flow runs from a deployment.

The changes in the task engine prevent a task from starting if the parent flow is paused, which can happen if the task is submitted to remote infrastructure. That hasn't been needed for flow runs because they have historically been blocking, but I could see that changing with the new engine. We would also need an orchestration rule to prevent starting flow runs from a paused parent run, so I think that might be best tackled in another PR.

@cicdw
Copy link
Member

cicdw commented May 7, 2024

@desertaxle ah wow, that's nice! Thanks for explaining

@desertaxle desertaxle merged commit c626d1b into main May 15, 2024
44 checks passed
@desertaxle desertaxle deleted the new-engine-pause-resume branch May 15, 2024 15:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
experimental Related to an experimental feature
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants