feat: Integration of VLM embedding model #446
Conversation
|
@FUYICC Hi, is the pr still in progress? Let me know if you have any difficulties. |
Thank you for your kind help! Sorry I've been mostly working on my dissertation for the past 3 weeks so I haven't had time to move forward, I'll be up and running starting next week, we'll discuss any questions anytime! |
FUYICC
changed the title
camel/embeddings/vlm_embedding.py
Outdated
| images=obj, return_tensors="pt", padding=True, **kwargs | ||
| ) | ||
| image_feature = ( | ||
| self.model.get_image_features(**input, **kwargs) |
There was a problem hiding this comment.
redundant kwargs could lead to unexpected behaviors if kwargs contains overlapping keys
There was a problem hiding this comment.
how about separate kwargs into 2 dict?
def embed_list(
self,
objs: List[Union[Image.Image, str]],
processor_kwargs: dict ={},
model_kwargs: dict = {},
) -> List[List[float]]:
text_input = self.processor(
text=obj,
return_tensors="pt",
padding=True,
**processor_kwargs,
)
text_feature = (
self.model.get_text_features(**text_input, **model_kwargs)
.squeeze(dim=0)
.tolist()
)
camel/embeddings/vlm_embedding.py
Outdated
| text=obj, return_tensors="pt", padding=True, **kwargs | ||
| ) | ||
| text_feature = ( | ||
| self.model.get_text_features(**input, **kwargs) |
| @@ -59,7 +59,7 @@ pyowm = { version = "^3.3.0", optional = true } | |||
| googlemaps = { version = "^4.10.0", optional = true } | |||
| requests_oauthlib = { version = "^1.3.1", optional = true } | |||
| unstructured = { extras = ["all-docs"], version = "^0.10.30", optional = true } | |||
|
|
|||
| pillow = { version = "^10.2.0", optional = true } | |||
There was a problem hiding this comment.
Because we need to determine if the input is an image or not in vlm embedding class.
There was a problem hiding this comment.
also add it under [tool.poetry.extras] tools and all part, as well as [[tool.mypy.overrides]]
Co-authored-by: Wendong-Fan <133094783+Wendong-Fan@users.noreply.github.com>
Co-authored-by: Wendong-Fan <133094783+Wendong-Fan@users.noreply.github.com>
Co-authored-by: Wendong-Fan <133094783+Wendong-Fan@users.noreply.github.com>
Co-authored-by: Wendong-Fan <133094783+Wendong-Fan@users.noreply.github.com>
Co-authored-by: Wendong-Fan <133094783+Wendong-Fan@users.noreply.github.com>
Wendong-Fan
changed the title
Description
Issue #445
Summary by CodeRabbit
CLIPEmbeddingfor image and text embedding functionalities.CLIPEmbeddingfunctionality, covering initialization, embedding processes, and output dimension retrieval.