Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FEATURE] Define flag allowing to execute light steps in main process #562

Closed
gabrielmbmb opened this issue Apr 23, 2024 · 0 comments
Closed
Assignees
Labels
enhancement New feature or request

Comments

@gabrielmbmb
Copy link
Member

Description

The current Pipeline implementation creates a subprocess to execute each Step. There are some Steps that only execute simpler transformations and that are very light to execute, therefore, the overhead added of creating a process and then sending the batches via queues is not worth it.

Add a flag for the Step and GlobalStep that allows executing the step in the main process without the need of creating an additional process.

@gabrielmbmb gabrielmbmb added the enhancement New feature or request label Apr 23, 2024
@gabrielmbmb gabrielmbmb added this to the 1.1.0 milestone Apr 23, 2024
@gabrielmbmb gabrielmbmb changed the title [FEATURE] Define LightStep (or similar) to execute those in the main process instead [FEATURE] Define flag allowing to execute light steps in main process Apr 23, 2024
@alvarobartt alvarobartt modified the milestones: 1.1.0, 1.2.0 May 7, 2024
@alvarobartt alvarobartt removed this from the 1.2.0 milestone May 21, 2024
@alvarobartt alvarobartt closed this as not planned Won't fix, can't repro, duplicate, stale May 21, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
Status: Todo
Development

No branches or pull requests

2 participants