Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ollama preload keep alive #1100

Open
wants to merge 4 commits into
base: main
Choose a base branch
from

Conversation

frangin2003
Copy link

Issue

N/A

Change

In order to ensure that Ollama model is ready to go at startup, I have introduced an optional preload boolean keyword argument (default set to False) to the Ollama LLM, allowing the model to be preloaded into memory during the initialization phase. This feature includes a preload_model method to trigger preloading at a later time if desired. The preloading process leverages the Ollama FAQ's recommended approach of calling generate with an empty prompt (see how-can-i-pre-load-a-model-to-get-faster-response-times).
Additionally, the keep_alive parameter is now usable to be passed and tell for how long the model stays in memory.

General checklist

  • There are no breaking changes
  • I have added unit and integration tests for my change
  • I have manually run all the unit and integration tests in the module I have added/changed, and they are all green
  • I have manually run all the unit and integration tests in the core and main modules, and they are all green

@langchain4j langchain4j added the P3 Medium priority label May 14, 2024
@frangin2003 frangin2003 marked this pull request as ready for review May 14, 2024 13:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
P3 Medium priority
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants