-
Notifications
You must be signed in to change notification settings - Fork 914
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Pali gemma modeling #1895
Pali gemma modeling #1895
Conversation
if config.model_type == "paligemma": | ||
full_text += "<bos>" + chunk["content"] + "\n" | ||
else: | ||
full_text += chunk["content"] |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can you revert this ? This is already taken care of by the PaliGemmaBatch.
Also we should probably raise an error when the query is not {image}, {text}
. (single text, single image, image before text)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We also need to add the causal flag to all flash attention places.
This PR adds paligemma modeling code
Blog post: https://huggingface.co/blog/paligemma
Transformers PR: huggingface/transformers#30814
install the latest changes and run with
basic example sending various requests