Nano Jarvis

Nano Jarvis is compatible with either a cloud-based (managed) LLM service (e.g. OpenAI GPT model, Grog, OpenRouter, etc) or with a locally hosted LLM server (e.g. llama.cpp, LocalAI, Ollama, etc). Please continue reading for detailed instructions.

Requirement: Node.js v18 or later.

Launch with:

./nano-jarvis.js

then open localhost:5000 with your favorite web browser.

Using Local LLM Servers

Supported local LLM servers include llama.cpp, Nitro, Ollama, and LocalAI.

To utilize llama.cpp locally with its inference engine, ensure to load a quantized model such as Phi-3 Mini, LLama-3 8B, or OpenHermes 2.5. Adjust the environment variable LLM_API_BASE_URL accordingly:

/path/to/llama.cpp/server -m Phi-3-mini-4k-instruct-q4.gguf
export LLM_API_BASE_URL=http://127.0.0.1:8080/v1

To utilize Nitro locally, refer to its Quickstart guide for loading a model like Phi-3 Mini, LLama-3 8B, or OpenHermes 2.5 and set the environment variable LLM_API_BASE_URL:

export LLM_API_BASE_URL=http://localhost:3928/v1

To use Ollama locally, load a model and configure the environment variable LLM_API_BASE_URL:

ollama pull phi3
export LLM_API_BASE_URL=http://127.0.0.1:11434/v1
export LLM_CHAT_MODEL='phi3'

For LocalAI, initiate its container and adjust the environment variable LLM_API_BASE_URL:

docker run -ti -p 8080:8080 localai/localai tinyllama-chat
export LLM_API_BASE_URL=http://localhost:3928/v1

Using Managed LLM Services

To use OpenAI GPT model, configure the environment variable OPENAI_API_KEY with your API key:

export LLM_API_BASE_URL=https://api.openai.com/v1
export OPENAI_API_KEY="sk-yourownapikey"

To utilize other LLM services, populate the relevant environment variables as demonstrated in the following examples:

• Anyscale

export LLM_API_BASE_URL=https://api.endpoints.anyscale.com/v1
export LLM_API_KEY="yourownapikey"
export LLM_CHAT_MODEL="meta-llama/Llama-3-8b-chat-hf"

• Deep Infra

export LLM_API_BASE_URL=https://api.deepinfra.com/v1/openai
export LLM_API_KEY="yourownapikey"
export LLM_CHAT_MODEL="mistralai/Mistral-7B-Instruct-v0.1"

• Fireworks

export LLM_API_BASE_URL=https://api.fireworks.ai/inference/v1
export LLM_API_KEY="yourownapikey"
export LLM_CHAT_MODEL="accounts/fireworks/models/llama-v3-8b-instruct"

• Grog

export LLM_API_BASE_URL=https://api.groq.com/openai/v1
export LLM_API_KEY="yourownapikey"
export LLM_CHAT_MODEL="gemma-7b-it"

• Lepton

export LLM_API_BASE_URL=https://mixtral-8x7b.lepton.run/api/v1/
export LLM_API_KEY="yourownapikey"

• OpenRouter

export LLM_API_BASE_URL=https://openrouter.ai/api/v1
export LLM_API_KEY="yourownapikey"
export LLM_CHAT_MODEL="mistralai/mistral-7b-instruct:free"

• Together

export LLM_API_BASE_URL=https://api.together.xyz/v1
export LLM_API_KEY="yourownapikey"
export LLM_CHAT_MODEL="meta-llama/Llama-3-8b-chat-hf"