feat: Just-in-time plugin installation

[ReubenFrankel]

Implements just-in-time install for the following commands:

• invoke
• run
• el/elt
• test
• config <plugin> test
• select --list

[netlify]

👷 Deploy request for meltano pending review.

Visit the deploys page to approve it

Name	Link
🔨 Latest commit	46707f0

[codecov]

Codecov Report

Attention: Patch coverage is 99.28058% with 1 lines in your changes are missing coverage. Please review.

Project coverage is 91.86%. Comparing base (7129516) to head (46707f0).
Report is 3 commits behind head on main.

Files	Patch %	Lines
src/meltano/cli/select.py	80.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #8482      +/-   ##
==========================================
+ Coverage   91.81%   91.86%   +0.05%     
==========================================
  Files         245      245              
  Lines       19296    19357      +61     
  Branches     2152     2156       +4     
==========================================
+ Hits        17717    17783      +66     
+ Misses       1306     1302       -4     
+ Partials      273      272       -1     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[ReubenFrankel]

We probably want to suppress install progress to stdout from meltano.cli.utils.install_plugins here, or emit as logs to stderr... For invoke, this currently breaks the common tap/target debugging technique of

meltano invoke <tap> > tap.out
cat tap.out | meltano invoke <target>

[edgarrmondragon]

We probably want to suppress install progress to stdout from meltano.cli.utils.install_plugins here, or emit as logs to stderr... For invoke, this currently breaks the common tap/target debugging technique of

meltano invoke <tap> > tap.out
cat tap.out | meltano invoke <target>

Yeah, I'd be more than OK with starting to unify our output away from click.echo and into the logging framework.

[ReubenFrankel]

Think I have covered all relevant commands here. Wondering if we want a new --no-install option for these (same convention as for meltano add) to opt-out of auto-install behaviour (e.g. for CI), or stick to the original proposal in #2089:

Interactive (TTY) experience

If a tty is available, we can simply ask the user if they'd like to auto-install the missing plugins.

Updated error message with hints

If a TTY is not available and --install is not specified, then the error message would be changed to something like:

ERROR: Plugin not installed. Please install with meltano install ... or rerun with meltano invoke --install ....

Why require confirmation / opt-in

Install operations are not always safe to run, and they can hide other issues in the invocation or installation, such as using the wrong plugin name or the wrong base image in a scheduled job. They could also trigger unnecessarily for a container-backed spec that doesn't require installation.

I haven't yet addressed install progress/status as logs, but the output is suppressed for meltano invoke for now.

[ReubenFrankel]

Should this be added for select too?

[edgarrmondragon]

Should this be added for select too?

If it's opt-in, probably yes 👍

[edgarrmondragon]

I'm happy to ignore missing coverage for this for now. We have an incoming test that'll cover this in #8518.

[ReubenFrankel]

Great! I was about to ask if you wanted a test 😅