Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We鈥檒l occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

馃 feat: OpenAI Assistants v2 (initial support) #2781

Merged
merged 10 commits into from
May 19, 2024
Merged

馃 feat: OpenAI Assistants v2 (initial support) #2781

merged 10 commits into from
May 19, 2024

Conversation

danny-avila
Copy link
Owner

@danny-avila danny-avila commented May 18, 2024 •
edited

Summary

This PR introduces initial OpenAI Assistant v2 support, a separate endpoint for Azure Assistants and includes several bug fixes and improvements. The major changes are summarized below:

Changes

New Features

  • GPT-4o Support for Assistants (v2)
  • Native image vision support for Assistants (v2)
  • Images with Code Interpreter now supported by LibreChat (v2)
    • Note: local copies are also made so the image can be displayed
  • Azure Assistants Endpoint (v1)
    • Added a separate endpoint for Azure Assistants, which still uses v1

Notes

Attaching vector stores for File Search is not yet supported. You can attach them from the Provider Playground or attach files to messages for file search on a thread basis.

Bug Fixes

  • Resolved the issue where the UI stays in an "analyzing" state if the run ends without a response for Code Interpreter.
  • Fixed an issue with annotations not being applied correctly with multiple Assistant file outputs
  • Fixed an issue with Assistant avatars becoming "obtuse" for images without squared dimensions
  • Fixed issue with new function name validation regex, which was affecting the naming strategy for Actions
  • Fixed issue where database records of files would be prematurely deleted if one of the file deletions failed
    • Now checks for each individual file deletion resolution to determine deletion from file records

Testing

Routes tested:

API Versions 1 & 2

  • GET / - List assistants
  • POST /avatar/:assistant_id - Upload/update avatar
  • GET /documents - User's assistant docs
  • DELETE /:id - Delete assistant
  • POST / - Create assistant
  • GET /:id - Fetch assistant (Not used)
  • PATCH /:id - Update assistant
  • All v1/actions
  • GET /tools

Additional Improvements

  • Added "source" icons for OpenAI storage files to indicate whether they are from Azure or OpenAI.
  • Adds "unknown" documents to Assistants Panel if uploaded elsewhere
  • Improved query invalidation logic for assistant-related queries.
  • Localized text across the application.
  • Fixed layout shift issues related to Clipboard Icon app-wide
  • Removed emboldened text from message results, closes [Bug]: Markdown Syntax Included in Copied Search Results聽#2780

Known Issues

  • Assistant Deletion Issue
    • Issue where deleting the last assistant leaves it selected in the UI form.
  • Azure Assistants Rate limiting errors.
    • Azure is strict with rate limits on polling, so the default rate was increased
    • Currently, there will be no message displayed when this happens, needs an error message
  • When switching between Assistants and Azure Assistants for the first time, an assistant from the other endpoint may incorrectly still be selected in the builder form (but not the conversation state)

Planned Updates

@danny-avila
馃 Assistants V2 Support: Part 1
2bdbff5 
- Separated Azure Assistants to its own endpoint
- File Search / Vector Store integration is incomplete, but can toggle and use storage from playground
- Code Interpreter resource files can be added but not deleted
- GPT-4o is supported
- Many improvements to the Assistants Endpoint overall

data-provider v2 changes

copy existing route as v1

chore: rename new endpoint to reduce comparison operations and add new azure filesource

api: add azureAssistants part 1

force use of version for assistants/assistantsAzure

chore: switch name back to azureAssistants

refactor type version: string | number

Ensure assistants endpoints have version set

fix: isArchived type issue in ConversationListParams

refactor: update assistants mutations/queries with endpoint/version definitions, update Assistants Map structure

chore:  FilePreview component ExtendedFile type assertion

feat: isAssistantsEndpoint helper

chore: remove unused useGenerations

chore(buildTree): type issue

chore(Advanced): type issue (unused component, maybe in future)

first pass for multi-assistant endpoint rewrite

fix(listAssistants): pass params correctly

feat: list separate assistants by endpoint

fix(useTextarea): access assistantMap correctly

fix: assistant endpoint switching, resetting ID

fix: broken during rewrite, selecting assistant mention

fix: set/invalidate assistants endpoint query data correctly

feat: Fix issue with assistant ID not being reset correctly

getOpenAIClient helper function

feat: add toast for assistant deletion

fix: assistants delete right after create issue for azure

fix: assistant patching

refactor: actions to use getOpenAIClient

refactor: consolidate logic into helpers file

fix: issue where conversation data was not initially available

v1 chat support

refactor(spendTokens): only early return if completionTokens isNaN

fix(OpenAIClient): ensure spendTokens has all necessary params

refactor: route/controller logic

fix(assistants/initializeClient): use defaultHeaders field

fix: sanitize default operation id

chore: bump openai package

first pass v2 action service

feat: retroactive domain parsing for actions added via v1

feat: delete db records of actions/assistants on openai assistant deletion

chore: remove vision tools from v2 assistants

feat: v2 upload and delete assistant vision images

WIP first pass, thread attachments

fix: show assistant vision files (save local/firebase copy)

v2 image continue

fix: annotations

fix: refine annotations

show analyze as error if is no longer submitting before progress reaches 1 and show file_search as retrieval tool

fix: abort run, undefined endpoint issue

refactor: consolidate capabilities logic and anticipate versioning

frontend version 2 changes

fix: query selection and filter

add endpoint to unknown filepath

add file ids to resource, deleting in progress

enable/disable file search

remove version log
@danny-avila
馃 Assistants V2 Support: Part 2
bc46ccd 
馃幑 fix: Autocompletion Chrome Bug on Action API Key Input

chore: remove `useOriginNavigate`

chore: set correct OpenAI Storage Source

fix: azure file deletions, instantiate clients by source for deletion

update code interpret files info

feat: deleteResourceFileId

chore: increase poll interval as azure easily rate limits

fix: openai file deletions, TODO: evaluate rejected deletion settled promises to determine which to delete from db records

file source icons

update table file filters

chore: file search info and versioning

fix: retrieval update with necessary tool_resources if specified

fix(useMentions): add optional chaining in case listMap value is undefined

fix: force assistant avatar roundedness

fix: azure assistants, check correct flag

chore: bump data-provider
@danny-avila
fix: merge conflict
6082e26
@danny-avila danny-avila mentioned this pull request May 19, 2024
Open
@danny-avila danny-avila merged commit 1a45212 into main May 19, 2024
4 checks passed
@danny-avila danny-avila deleted the v2-assistants branch May 19, 2024 16:56
techwithanirudh pushed a commit to techwithanirudh/LibreChat-dev that referenced this pull request May 20, 2024
@danny-avila @techwithanirudh
馃 feat: OpenAI Assistants v2 (initial support) (danny-avila#2781)
9f91ac6 
* 馃 Assistants V2 Support: Part 1

- Separated Azure Assistants to its own endpoint
- File Search / Vector Store integration is incomplete, but can toggle and use storage from playground
- Code Interpreter resource files can be added but not deleted
- GPT-4o is supported
- Many improvements to the Assistants Endpoint overall

data-provider v2 changes

copy existing route as v1

chore: rename new endpoint to reduce comparison operations and add new azure filesource

api: add azureAssistants part 1

force use of version for assistants/assistantsAzure

chore: switch name back to azureAssistants

refactor type version: string | number

Ensure assistants endpoints have version set

fix: isArchived type issue in ConversationListParams

refactor: update assistants mutations/queries with endpoint/version definitions, update Assistants Map structure

chore:  FilePreview component ExtendedFile type assertion

feat: isAssistantsEndpoint helper

chore: remove unused useGenerations

chore(buildTree): type issue

chore(Advanced): type issue (unused component, maybe in future)

first pass for multi-assistant endpoint rewrite

fix(listAssistants): pass params correctly

feat: list separate assistants by endpoint

fix(useTextarea): access assistantMap correctly

fix: assistant endpoint switching, resetting ID

fix: broken during rewrite, selecting assistant mention

fix: set/invalidate assistants endpoint query data correctly

feat: Fix issue with assistant ID not being reset correctly

getOpenAIClient helper function

feat: add toast for assistant deletion

fix: assistants delete right after create issue for azure

fix: assistant patching

refactor: actions to use getOpenAIClient

refactor: consolidate logic into helpers file

fix: issue where conversation data was not initially available

v1 chat support

refactor(spendTokens): only early return if completionTokens isNaN

fix(OpenAIClient): ensure spendTokens has all necessary params

refactor: route/controller logic

fix(assistants/initializeClient): use defaultHeaders field

fix: sanitize default operation id

chore: bump openai package

first pass v2 action service

feat: retroactive domain parsing for actions added via v1

feat: delete db records of actions/assistants on openai assistant deletion

chore: remove vision tools from v2 assistants

feat: v2 upload and delete assistant vision images

WIP first pass, thread attachments

fix: show assistant vision files (save local/firebase copy)

v2 image continue

fix: annotations

fix: refine annotations

show analyze as error if is no longer submitting before progress reaches 1 and show file_search as retrieval tool

fix: abort run, undefined endpoint issue

refactor: consolidate capabilities logic and anticipate versioning

frontend version 2 changes

fix: query selection and filter

add endpoint to unknown filepath

add file ids to resource, deleting in progress

enable/disable file search

remove version log

* 馃 Assistants V2 Support: Part 2

馃幑 fix: Autocompletion Chrome Bug on Action API Key Input

chore: remove `useOriginNavigate`

chore: set correct OpenAI Storage Source

fix: azure file deletions, instantiate clients by source for deletion

update code interpret files info

feat: deleteResourceFileId

chore: increase poll interval as azure easily rate limits

fix: openai file deletions, TODO: evaluate rejected deletion settled promises to determine which to delete from db records

file source icons

update table file filters

chore: file search info and versioning

fix: retrieval update with necessary tool_resources if specified

fix(useMentions): add optional chaining in case listMap value is undefined

fix: force assistant avatar roundedness

fix: azure assistants, check correct flag

chore: bump data-provider

* fix: merge conflict

* ci: fix backend tests due to new updates

* chore: update .env.example

* meilisearch improvements

* localization updates

* chore: update comparisons

* feat: add additional metadata: endpoint, author ID

* chore: azureAssistants ENDPOINTS exclusion warning
@danny-avila danny-avila restored the v2-assistants branch May 21, 2024 14:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[Bug]: Markdown Syntax Included in Copied Search Results
1 participant
@danny-avila