Possibility to connect a local LLM to smart connections

[automataIA]

Possibility to connect a local LLM to smart connections, like mistral, orca etc...

[RealRavens]

Brian mentioned this is planned, gotta keep waiting along side the rest of us. 😆

See #137 (comment)

[brianpetro]

Hi @automataIA

Local models will be introduced in the next release, starting a local embedding model which SCv2 will ship with as the default model.

I'm also working on a universal-ish adapter so that additional APIs may be added for use with the Smart Chat, which would allow using models like Mistral but through an HTTP API. If you are savvy enough to setup a local model, and have the necessary hardware, then adding the necessary localhost API configuration should be trivial to someone with that amount of technical experience.

Regarding SC shipping with local chat models, given the currently significant resource requirements, I have no estimated timeline. I do think the smaller and quantized models are getting closer to being useful, and an application like Smart Connections, ones that can provide sufficient & individualized k-shots, will be among the first to make those models useful.

🌴

[pascalandy]

Ollama is openai API compataible since 2 weeks :)

[pascalandy]

I would prefer to NOT have another dedicated LLM server than Ollama or LM Studio. Why ? Because I already have about 200GB in managed using Ollama. Every apps that wants to manage the server adds XX GB to my machine and it's not sustainable.

[brianpetro]

@pascalandy plan is currently to connect to your existing Ollama (no extra model server)

[Spartanmans]

Hello, I just tried it. Isn't it possible to connect to LM Studio now?

[christianngarcia]

That should work. If you're having trouble, please screenshot the developer console logs so we can see any errors that might be preventing it from working 🌼

[tprcpa]

Hi, I'm a newbie with very little modern programming expertise. I changed majors from IT to Accounting 25 years ago and have only stayed on the fringes of the tech space. I am gathering all my notes into Obsidian and would like to write several books/novels based on the notes that I have been compiling for years. I have checked Fiverr, Freelance, Upwork, etc. with no success. Where can I go to hire someone to assist in setting up Obsidian Vault with Smart Connections and Co-Pilot plug-ins? I have LM Studio (Llama3) installed locally and get Co-Pilot to respond but it seems to hallucinate frequently. I am not able to get Smart Connections to function properly with several API embed errors, etc. Any ideas where I can pay for help?

[jwhco]

I've taken this journey before, consolidating 23 years of notes into Obsidian. I own a company that publishes books and provides agency services. My advice is to start small and local with Smart Connections. Run a local model before connecting to LM Studio or Ollama.

Most AI models will hallucinate more than half the time when doing serious work. What I mean by this is that because you understand the underlying data well, your brain can outpace what AI can compile. Focus instead on your editorial process and deliverables; use AI as a query tool.

Here's a recent training I did on productive editorial and writing environments, AI must work for you before you start querying it. Writing books based on your notes is something I do every day. AI can help you cluster topics, understand your audience, and bullet point concepts.

But remember, AI can only reply with what is known, not the new ideas and concepts you can create to make the book of value to a specific audience. Your approach to writing is what gets the books done, AI can be a huge distraction. Start with a local model like BGE-micro, which has been useful to me. If you'd like to speak directly, contact me via https://www.justinhitt.com/contact/