Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

BOUNTY: Convert the endpoints to a smart router like litellm does in python #286

Open
sandys opened this issue Dec 2, 2023 · 27 comments
Open

BOUNTY: Convert the endpoints to a smart router like litellm does in python #286

sandys opened this issue Dec 2, 2023 · 27 comments
Labels
💎 Bounty

Comments

@sandys
Copy link
Contributor

sandys commented Dec 2, 2023 •
edited

this bounty refers to the JS SDK only http://github.com/arakoodev/EdgeChains/tree/ts/JS/edgechains

we are super inspired by Litellm in python (and have a lot of respect for them) - https://github.com/BerriAI/litellm and we want to create a similar infrastructure in Edgechain.js using Typescript. We are hoping libraries like axios makes this easier.

the four features we want are these:

  1. load balance between multiple deployments. It picks the deployment which is below rate-limit and has the least amount of tokens used. (only openai, google palm and cohere needed). this includes basic reliability features (https://docs.litellm.ai/docs/routing#basic-reliability) like timeouts/retries/etc. you can use axios.interceptors.response.use to achieve these.
  2. streaming capability
  3. token usage - https://docs.litellm.ai/docs/completion/token_usage
  4. logging - https://docs.litellm.ai/docs/observability/callbacks (only sentry and posthog needed)

Once created, you will replace the Endpoint class with this new infrastructure (https://github.com/arakoodev/EdgeChains/blob/ts/JS/edgechains/lib/src/lib/endpoints/OpenAiEndpoint.ts)

project will be deemed complete when:

  1. you add testcases for all the features above.
  2. mock the openai, palm and cohere endpoints for testing. (you can take inspiration from here https://github.com/polly3d/mockai)
  3. you have deleted the current endpoint classes and replaced it with your new classes in all examples and testcases.

NOTE: the core of edgechains is the usage of jsonnet (https://jsonnet.org/) configuration to manage all kinds of config - like prompts, etc. we will prefer that as far as possible, we use jsonnet to configure this routing.
@sandys
Copy link
Contributor Author

sandys commented Dec 2, 2023

/bounty 200

@algora-pbc Algora PBC
Copy link

algora-pbc bot commented Dec 2, 2023 •
edited

💎 $200 bounty • Arakoo.ai

Steps to solve:

  1. Start working: Comment /attempt #286 with your implementation plan
  2. Submit work: Create a pull request including /claim #286 in the PR body to claim the bounty
  3. Receive payment: 100% of the bounty is received 2-5 days post-reward. Make sure you are eligible for payouts

Thank you for contributing to arakoodev/EdgeChains!

Add a bounty • Share on socials

Attempt Started (GMT+0) Solution
🔴 @rajdip-b Dec 2, 2023, 12:50:05 PM WIP

@rajdip-b
Copy link

rajdip-b commented Dec 2, 2023 •
edited by algora-pbc bot

/attempt #286

Options

@rajdip-b
Copy link

rajdip-b commented Dec 2, 2023

@sandys can i be assigned this issue?

@sandys
Copy link
Contributor Author

sandys commented Dec 2, 2023

@rajdip-b happy to.
but first want to check if you understand what has to be achieved ? any doubts, etc ?

@guhitb
Copy link

guhitb commented Dec 2, 2023

@rajdip-b Any interest in collaborating on this?

@rajdip-b
Copy link

rajdip-b commented Dec 2, 2023 •
edited

@sandys As per my understanding, the main challenge is creating the load balancer in the endpoints of OpenAiEndpoint.ts. I was going through the references you mentioned and have a clarity over what needs to be implemented. I would have to do some further digging about the how's of the implementation, but yes, I feel I can definitely give it a shot!

And not to mention, as and when issues come up, I will ping you about them.

@sandys
Copy link
Contributor Author

sandys commented Dec 2, 2023

@rajdip-b there are 4 capabilities needed (as mentioned in my original post). it is important that you look at all four, because the architecture will need to be inclusive.
also, this architecture will REPLACE the current endpoint structure

@rajdip-b
Copy link

rajdip-b commented Dec 2, 2023

@guhitb, are you familiar with the project structure/implementation details? I would be happy to collaborate!

@rajdip-b
Copy link

rajdip-b commented Dec 2, 2023

@rajdip-b there are 4 capabilities needed (as mentioned in my original post). it is important that you look at all four, because the architecture will need to be inclusive. also, this architecture will REPLACE the current endpoint structure

Yeah, of course, I understand that. For me, I feel the challenge would be implementing load balancing, hence mentioned that. As for the rest, I feel they can be done with lesser efforts than the former.

@sandys
Copy link
Contributor Author

sandys commented Dec 2, 2023

very well. assigned to @rajdip-b best wishes. please feel free to drop in to discord if u want to discuss anything quickly. https://discord.gg/Avsbume2

@rajdip-b
Copy link

rajdip-b commented Dec 2, 2023

very well. assigned to @rajdip-b best wishes. please feel free to drop in to discord if u want to discuss anything quickly. https://discord.gg/Avsbume2

Discord is my go-to for discussion! Hopping in right now!

@guhitb
Copy link

guhitb commented Dec 2, 2023

@rajdip-b Not yet, aside from browsing through the repo today. I don't have much experience with open source, but I'm sure I can help with mocks, logging, tests, etc. The load balancing part feels a little daunting though.

@rajdip-b
Copy link

rajdip-b commented Dec 2, 2023 •
edited

@rajdip-b Not yet, aside from browsing through the repo today. I don't have much experience with open source, but I'm sure I can help with mocks, logging, tests, etc. The load balancing part feels a little daunting though.

Hey sure thing! We can pair up!

Drop me a text on discord! _the_heisenberg

@rajdip-b
Copy link

rajdip-b commented Dec 2, 2023

/split @guhitb

@sandys
Copy link
Contributor Author

sandys commented Dec 2, 2023 via email

@rajdip-b
Copy link

rajdip-b commented Dec 2, 2023

@sandys Ah, that's bad. Sure, I'll take this issue op alone. @guhitb You can still connect with me in case you want to collab on any other issue! Hope that's fine by everyone :D

@rajdip-b
Copy link

rajdip-b commented Dec 3, 2023

Hey @sandys, I have started with the docs today. I have went through the code base of litellm and how they implement their stuff. I also went though openai docs. I have some questions and suggestions that I would like to share and discuss with you. I couldn't find you in discord in you channel. mind pinging you handle in here? ill open a thread in discord and tag you in it!

@sandys
Copy link
Contributor Author

sandys commented Dec 4, 2023 via email

@rajdip-b
Copy link

rajdip-b commented Dec 4, 2023

Created! @sandys

@Sajiyah-Salat
Copy link
Contributor

Hello @rajdip-b are you still working on this. Let me know your discord handle. I would like to connect with you there

@rajdip-b
Copy link

Hey @Sajiyah-Salat, there has been a blocker, and hence I have paused my work currently.

@Sajiyah-Salat
Copy link
Contributor

Okay. Thanks for the reply

@Sambit003
Copy link

Sambit003 commented Mar 25, 2024 •
edited

@sandys @Sajiyah-Salat can I work on this?

@rajdip-b
Copy link

@sandys I'm dropping my attempt on this issue. For anyone who attempts next, feel free to pick up from my PR.

@rajdip-b rajdip-b removed their assignment Mar 26, 2024
@snigdha510
Copy link

can I work on this?

@sandys
Copy link
Contributor Author

sandys commented May 17, 2024 via email

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
💎 Bounty
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
6 participants
@sandys @Sambit003 @rajdip-b @snigdha510 @Sajiyah-Salat @guhitb