You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The current Pipeline implementation creates a subprocess to execute each Step. There are some Steps that only execute simpler transformations and that are very light to execute, therefore, the overhead added of creating a process and then sending the batches via queues is not worth it.
Add a flag for the Step and GlobalStep that allows executing the step in the main process without the need of creating an additional process.
The text was updated successfully, but these errors were encountered:
gabrielmbmb
changed the title
[FEATURE] Define LightStep (or similar) to execute those in the main process instead
[FEATURE] Define flag allowing to execute light steps in main process
Apr 23, 2024
Description
The current
Pipeline
implementation creates a subprocess to execute eachStep
. There are someStep
s that only execute simpler transformations and that are very light to execute, therefore, the overhead added of creating a process and then sending the batches via queues is not worth it.Add a flag for the
Step
andGlobalStep
that allows executing the step in the main process without the need of creating an additional process.The text was updated successfully, but these errors were encountered: