You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Background
Each resource has a single execution pipe that yields items when extracted. We want to support resources with many pipes that can yield items against the same resource schema. This allows to create resources that extract several files or several partitions of the same table easily.
Currently if source tries to create two resources with the same name, we get an error. The intended behavior was to create a resource with additional pipes.
Requirements
allow to add secondary pipes to a resource
update the source that if a resource with the same name is returned/added we add the pipe to the existing resource and apply hints from new resource (we can add table_variant if the new resource affects different table_name)
update DltResourceDict to handle additional pipes. there's plenty of code that is accessing _pipe property of a resource
Tests
We need to test several low level operations on the resources.
cloning and calling
iterating (as part of resources and source)
async generators and parallel extraction should work (pipes should be parallelized)
DltResourceDict tests must be extended to resources with multiple pipes. adding new resources, generating dag etc. are all to be checked
standard end to end tests
The text was updated successfully, but these errors were encountered:
Background
Each resource has a single execution pipe that yields items when extracted. We want to support resources with many pipes that can yield items against the same resource schema. This allows to create resources that extract several files or several partitions of the same table easily.
Currently if source tries to create two resources with the same name, we get an error. The intended behavior was to create a resource with additional pipes.
Requirements
DltResourceDict
to handle additional pipes. there's plenty of code that is accessing_pipe
property of a resourceTests
We need to test several low level operations on the resources.
DltResourceDict
tests must be extended to resources with multiple pipes. adding new resources, generating dag etc. are all to be checkedThe text was updated successfully, but these errors were encountered: