-
-
Notifications
You must be signed in to change notification settings - Fork 1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Feature]: Cloudflare AI Gateway support for Bedrock #1040
Comments
Lower priority since we aren't using Amazon Bedrock as much after discovering that our AWS Activate credits don't apply to Anthropic or Cohere models on Bedrock. =P |
Small note: I'm not even 100% sure how to use Cloudflare AI Gateway with Bedrock (ignoring LiteLLM), so don't worry about this ticket at all in the meantime. Not a high priority at all imo. |
This is doable using boto3 hooks. I helped Cloudflare figure this one out, they will eventually be documenting it at https://developers.cloudflare.com/ai-gateway/providers/bedrock/ In the meantime, gonna save my notes here just in case. https://stackoverflow.com/a/66097613/2079814 event_system.register_first('after-sign.*.*', _add_header) or: https://boto3.amazonaws.com/v1/documentation/api/latest/guide/events.html#before-send
Unlike my example, you should only only boto3 requests going to bedrock invoke and invoke stream, since hooking others is just gonna break things if eventually the same boto3 client is used to do other API calls in the future. |
JavaScript/TypeScript solution here: https://gist.github.com/Manouchehri/d1fa65b6d842a76d836cb6b3f9e1bbff |
The Feature
The docs aren't added yet, but Cloudflare has added AI Gateway support for Amazon Bedrock.
https://developers.cloudflare.com/ai-gateway/providers/
The example request looks like:
Motivation, pitch
Free caching :D
Twitter / LinkedIn details
https://www.linkedin.com/in/davidmanouchehri/
The text was updated successfully, but these errors were encountered: