Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

prefect deploy command fails if not all dependencies are present at runtime #9512

Open
3 tasks done
jamiezieziula opened this issue May 10, 2023 · 15 comments · May be fixed by #13315
Open
3 tasks done

prefect deploy command fails if not all dependencies are present at runtime #9512

jamiezieziula opened this issue May 10, 2023 · 15 comments · May be fixed by #13315
Assignees
Labels
enhancement An improvement of an existing feature needs:design Blocked by a need for an implementation outline

Comments

@jamiezieziula
Copy link
Contributor

First check

  • I added a descriptive title to this issue.
  • I used the GitHub search to find a similar request and didn't find it.
  • I searched the Prefect documentation for this feature.

Prefect Version

2.x

Describe the current behavior

Currently, running prefect deploy... will fail if not all dependencies required by the deployments are present:

jamiedick in ~/flows on platform-refactor ● λ prefect deploy --all   
Deploying all deployments for current project...
╭─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ Deploying Deployment 1                                                                                                              │
╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
Script at 'deployments/deployment1/flow.py' encountered an exception: ModuleNotFoundError("No module named 
'slack_sdk'")
jamiedick in ~/flows on main

This can be problematic if deploying multiple flows that have different requirements / when using virtual environments to isolate dependencies. This also makes deploying via cicd cumbersome because, again, all dependencies need to be present (which is duplicative of something like a docker build step)

Describe the proposed behavior

The prefect deploy command would not require all dependencies to be present at runtime (either optionally or by default)

Example Use

No response

Additional context

No response

@jamiezieziula jamiezieziula added enhancement An improvement of an existing feature status:triage labels May 10, 2023
@zanieb
Copy link
Contributor

zanieb commented May 10, 2023

We think we need to load the flow to generate the OpenAPI schema for its parameters. We could do this at an earlier time and store the schema JSON somewhere?

@zanieb zanieb added the needs:triage Needs feedback from the Prefect product team label May 10, 2023
@github-actions
Copy link
Contributor

github-actions bot commented Jun 9, 2023

This issue is stale because it has been open 30 days with no activity. To keep this issue open remove stale label or comment.

@github-actions github-actions bot added the status:stale This may not be relevant anymore label Jun 9, 2023
@desertaxle desertaxle removed the status:stale This may not be relevant anymore label Jun 9, 2023
@github-actions
Copy link
Contributor

github-actions bot commented Jul 9, 2023

This issue is stale because it has been open 30 days with no activity. To keep this issue open remove stale label or comment.

@github-actions github-actions bot added the status:stale This may not be relevant anymore label Jul 9, 2023
@jamiezieziula jamiezieziula removed the status:stale This may not be relevant anymore label Jul 19, 2023
@prefectcboyd
Copy link
Contributor

+1 on this - Using both the previous deployment.apply() and currently prefect deploy require me to have all dependencies installed. This has frustrating implications when I'm running this in any CI/CD fashion as I have to install packages into container, and then install the packages again in the builder to deploy.

@zhen0 zhen0 removed the needs:triage Needs feedback from the Prefect product team label Aug 18, 2023
@ne-warwick
Copy link

ne-warwick commented Dec 11, 2023

+1 on this.
I'm surprised this seems like quite a stale issue. How are others solving for this? We have this problem for deploying flows from a centralised CICD but each flow runs using a different docker image and we don't want the CICD to have ALL the requirements at runtime, nor build docker images for the simple task of creating a deployment config.

@ialejandro
Copy link

+1 on this.

On CI/CD waste a lot of time to install dependencies which already installed on Docker image (for example)

@speedyturkey
Copy link

Would love to see this behavior changed. My initial understanding of the prefect.yaml / deployment.yaml file was that all necessary metadata was present, rendering it unnecessary to manipulate the Python representation of the flow. It certainly complicates CI/CD.

My workaround will be to install flows in my CI/CD environment, and to have a separate build action for each flow, but this not ideal from a code hygiene and performance perspective.

@furkanrollic
Copy link

Any updates on this issue? Having to install all dependencies to create a simple deployment object is kind of cumbersome and conflicts with the purpose of separating flow's dependencies using different images.

@ialejandro
Copy link

My workaround will be to install flows in my CI/CD environment, and to have a separate build action for each flow, but this not ideal from a code hygiene and performance perspective.

Here too. I use PIpfile with pipenv to ensure all works properly.

Dockerfile (image where run flow)

...
RUN pip install --no-cache-dir pipenv==${PIPENV_RELEASE} && \
    pipenv sync --clear --system                         && \
    pip cache purge

CI/CD (GitHub Actions)

      - name: Setup Python env
        uses: actions/setup-python@v4
        with:
          python-version: ${{ vars.PYTHON_VERSION }}

      - name: Setup Prefect CLI
        run: |
          pip install --no-cache-dir prefect==${{ vars.PREFECT_CLI_VERSION }}

      - name: Setup pipenv and Install dependencies
        working-directory: ${{ env.PREFECT_FLOW_NAME }}
        run: |
          pip install --no-cache-dir pipenv==${{ vars.PIPENV_RELEASE }}
          pipenv sync --clear --system

     ...
     # step to prefect deploy

So, I ensure that CI/CD needs the requirements like Dockerfile where build a flow image.

@pierreloicq
Copy link

+1 on this. Currently I deal with that by calling the external packages only inside functions.

@netanelm-upstream
Copy link

+1 on this.

@serinamarie
Copy link
Contributor

Thank you for all the +1. We'll look more into this.

@baxen
Copy link

baxen commented Mar 21, 2024

adding another +1

@akfmdl
Copy link

akfmdl commented Apr 18, 2024

it is so frustrating issue..

@serinamarie serinamarie added the needs:design Blocked by a need for an implementation outline label Apr 18, 2024
@syakesaba
Copy link

+1

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement An improvement of an existing feature needs:design Blocked by a need for an implementation outline
Projects
None yet
Development

Successfully merging a pull request may close this issue.