New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[FEAT]: Integration of Vllm as model server #1153
Comments
I were able to use vllm selecting Local AI in AnythingLLM LLM Settings. |
Thanks for your advice.
I have tested but i failed.
I do confirm vllm instance is working fine since
https://vllm-mixtral.myserver.fr/v1/models
{"object":"list","data":[{"id":"mistralai/Mixtral-8x7B-Instruct-v0.1","object":"model","created":1714112327,"owned_by":"vllm","root":"mistralai/Mixtral-8x7B-Instruct-v0.1","parent":null,"permission":[{"id":"modelperm-76d249bf4f0e44698e3bb82a41424183","object":"model_permission","created":1714112327,"allow_create_engine":false,"allow_sampling":true,"allow_logprobs":true,"allow_search_indices":false,"allow_view":true,"allow_fine_tuning":false,"organization":"*","group":null,"is_blocking":false}]}]}
i have putted in the config: LocalAI with http://vllm-mistral:5002/v1
Anything is able to retrieve the model.
But when I tried to engage chat, i got an error:vCould not respond to message.
Request failed with status code 400
Looking to the log of Anythingllm , I ahev the following trace:
I would appreciate your help.
Thanks in advance.
Francois, from France
```
_events: [Object: null prototype],
349
_eventsCount: 1,
350
_maxListeners: undefined,
351
socket: [Socket],
352
httpVersionMajor: 1,
353
httpVersionMinor: 1,
354
httpVersion: '1.1',
355
complete: true,
356
rawHeaders: [Array],
357
rawTrailers: [],
358
joinDuplicateHeaders: undefined,
359
aborted: false,
360
upgrade: false,
361
url: '',
362
method: null,
363
statusCode: 400,
364
statusMessage: 'Bad Request',
365
client: [Socket],
366
_consuming: false,
367
_dumped: false,
368
req: [ClientRequest],
369
responseUrl: 'http://vllm-mixtral:5002/v1/chat/completions',
370
redirects: [],
371
[Symbol(kCapture)]: false,
372
[Symbol(kHeaders)]: [Object],
373
[Symbol(kHeadersCount)]: 10,
374
[Symbol(kTrailers)]: null,
375
[Symbol(kTrailersCount)]: 0
376
}
377
},
378
isAxiosError: true,
379
toJSON: [Function: toJSON]
380
}
```
…On Wed, Apr 24, 2024 at 2:52 PM Mikhail Khludnev ***@***.***> wrote:
I were able to use vllm selecting Local AI in AnythingLLM LLM Settings.
Enjoy.
—
Reply to this email directly, view it on GitHub
<#1153 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ABKZRFGNVGLCW5HBLY7LT2DY66TI7AVCNFSM6AAAAABGQZ5LSOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDANZUHA3TONJRGQ>
.
You are receiving this because you authored the thread.Message ID:
***@***.***>
--
|
If it could help you in your analysis: but when i go this url i got: it is a problem of api between vllm and anythingllm ? |
That endpoint is POST only, not GET - which is part of the reason you got method not allowed when going to the URL directly. |
I have the same issue, no mather the integration (Local AI or Generic OpenAI). |
What was the solution? Did you manage to integrate vLLM? |
Dear all, |
@flefevre once again, I did it twice it works. Maybe it's a container connectivity issue? I remember that I use curl to check connectivity between containers. Can you tell how your containers,hosts and processes are aligned? |
It seems these are two different issues, one related to connectivity, and the other one on format. |
Dear all, Docker configuration
Docker validation When I connect to the anythingllm container, i am able to retrieve model of vllm through the command:
Anythingllm Webui configuration
Anythingllm Webui Test When i look at the Anythingllm logs, i have the following trace
Analysis I have created the following Feature proposal here #1154 , I do think it is the good solution. If yes, my ticket should perhaps invalidated since Anythingllm is compatible with vllm but not with all models served by vllm. Thanks for for your expertise. |
What would you like to see?
it would be great to be able to configure AnythingLLM with a Vllm model
https://github.com/vllm-project/vllm
The text was updated successfully, but these errors were encountered: