Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve lwe startup time #341

Open
johnd0e opened this issue Feb 9, 2024 · 2 comments
Open

Improve lwe startup time #341

johnd0e opened this issue Feb 9, 2024 · 2 comments

Comments

@johnd0e
Copy link

johnd0e commented Feb 9, 2024

Is your feature request related to a problem? Please describe.
Currently at my system it takes about 5s to start.
It's not an issue when start it once, and then use it for long time.
But additional 5s is too long if I want just run some non-interactive one-time request in command line.

Describe the solution you'd like
lwe is pretty complex, as my venv takes ~780MB.
And I'm sure that only small subset of all features are used for simple cases.
May be it's possible not to load all that huge megabytes at once, but only when really needed.

Describe alternatives you've considered
I am understand that this type of request may require too much changes, so...
I just want you to know that for some use cases startup time may be critical.

@thehunmonkgroup
Copy link
Collaborator

I just want you to know that for some use cases startup time may be critical.

Then use a library that doesn't leverage the slowest scripting language on earth ;)

I've actually found a great deal of the startup time is taken by the provider plugins. Some of the underlying libraries are probably issuing HTTP requests and such to gather info. There's not a whole lot I can do about that w/o rolling my own provider code, which is not worth it to save a little startup time.

If I disable all plugins, my startup time is about 1.5 seconds.

The design goals of LWE do not include blazing fast startup times. The two main use cases are:

  1. Start it and leave it available for quick CLI interaction with LLMs
  2. Run more complicated workflows in some kind of automation framework

@thehunmonkgroup
Copy link
Collaborator

I'm going to open this back up, because although I personally have no interest in doing the work on this feature, I would review a well-considered PR that improves startup time.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants