Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

allow resources with many pipes (ie. partitioned resources) #1319

Open
rudolfix opened this issue May 3, 2024 · 0 comments
Open

allow resources with many pipes (ie. partitioned resources) #1319

rudolfix opened this issue May 3, 2024 · 0 comments

Comments

@rudolfix
Copy link
Collaborator

rudolfix commented May 3, 2024

Background
Each resource has a single execution pipe that yields items when extracted. We want to support resources with many pipes that can yield items against the same resource schema. This allows to create resources that extract several files or several partitions of the same table easily.

Currently if source tries to create two resources with the same name, we get an error. The intended behavior was to create a resource with additional pipes.

Requirements

    • allow to add secondary pipes to a resource
    • update the source that if a resource with the same name is returned/added we add the pipe to the existing resource and apply hints from new resource (we can add table_variant if the new resource affects different table_name)
    • update DltResourceDict to handle additional pipes. there's plenty of code that is accessing _pipe property of a resource

Tests
We need to test several low level operations on the resources.

  1. cloning and calling
  2. iterating (as part of resources and source)
  3. async generators and parallel extraction should work (pipes should be parallelized)
  4. DltResourceDict tests must be extended to resources with multiple pipes. adding new resources, generating dag etc. are all to be checked
  5. standard end to end tests
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Status: Todo
Development

No branches or pull requests

1 participant