Langchain integration

[batrasrishti]

Hello,

1. Can we use ggml models with spacy_llm using Llamacpp from langchain, if yes then can you please guide how that can be done.
2. Please can you tell me what are the things with '@' prefix called, for example @llm_models, @llm_queries?
3. In the documentation it is written, @llm_models = "langchain.OpenAI.v1" I don't know what I should be putting at the right side for a model on HuggingFaceHub.

[rmitsch]

Hi @batrasrishti!

1. You can pass on all langchain config parameter to the langchain model via the spacy-llm config. I've never tried .ggml models via langchain, but I don't see a reason why this shouldn't work. If you encounter a particular problem, let us know.
2. The @ is used for registries. See the docs on the spaCy config system.
3. Any of the handles for the supported HuggingFace models, e. g. spacy.Lllama2.v1. See the docs.

[batrasrishti]

1. Suppose I want to use a model like this, llm = Llama(model_path="llama-2-7b-chat.Q4_K_S.gguf", n_ctx=2048, n_batch=512)
   I am confused about how to put in these type of settings for the LLM in the config file?
2. Understood, thank you.
3. Point 3 was for HuggingFace models for example: google/flan-t5-xxl which are still not supported by spacy.

[rmitsch]

1. You pass these arguments in your config attribute in your spacy-llm config. All these arguments are passed on to langchain.
2. 👍
3. If you want us to support a particular HF model, feel free to open an issue with a feature request! We'll look into it 🙂

[pacoBOeT]

Hi, it would be fantastic to have Falcon 40b (not the instruct version) as a supported HF model in spacy_llm. Could it be possible? Is there any other way to implement in spacy a non supported HF model with langchain? I may need to use other models from HF and wouldn't want to be annoying you each time. Thank you so much!

[rmitsch]

You can always use the langchain integration to run any model supported by langchain - if that's the case for Falcon 40b, you can use it in spacy-llm.

[OzielLeos]

Hello batrasrishti,

Yes, you can use models with spacy_llm by leveraging Llamacpp from Langchain! Below, I'll show you how to set this up and provide an example of using it for Named Entity Recognition (NER).

Here’s the langchain_llm_ner.cfg configuration file that outlines how to load the model and set up NER:

langchain_llm_ner.cfg

[nlp]
lang = "en"
pipeline = ["llm"]
[components]
[components.llm]
factory = "llm"
[components.llm.task]
@llm_tasks = "spacy.NER.v3"
labels = ["PERSON", "ORGANISATION", "LOCATION"] 
[components.llm.model]
@llm_models = "langchain.LlamaCpp.v1"
name =  "mistralai_mistral-7b-instruct-v0.1"
config = {
    "model_path": "models/mistral-7b-instruct-v0.1.Q4_0.gguf",
    "temperature": 0.0 
    }

Python code

from spacy_llm.util import assemble
nlp = assemble("langchain_llm_ner.cfg")
content = "Jack and Jill rode up the hill in Les Deux Alpes"
doc = nlp(content)
print([(ent.text, ent.label_) for ent in doc.ents])