Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

prompt sharing for faster page attn #28

Open
mmoskal opened this issue Dec 22, 2023 · 0 comments
Open

prompt sharing for faster page attn #28

mmoskal opened this issue Dec 22, 2023 · 0 comments
Assignees
Labels

Comments

@mmoskal
Copy link
Member

mmoskal commented Dec 22, 2023

Right now (validate this!) the paged attn kernel doesn't take advantage of the fact that a significant part of the prompt may be shared between many queries - probably the kernel could be modified to only read these prompt entries once.

@mmoskal mmoskal added the rLLM label Dec 22, 2023
@mmoskal mmoskal self-assigned this Dec 22, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

1 participant