Skip to content
/ M2PT Public

[CVPR'24] Multimodal Pathway: Improve Transformers with Irrelevant Data from Other Modalities

License

Notifications You must be signed in to change notification settings

AILab-CVC/M2PT

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

13 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Multimodal Pathway: Improve Transformers with Irrelevant Data from Other Modalities [CVPR 2024]


arXiv website

Inspiration of Multimodal Pathway

This diagram's composition is inspired by a figure in Jeff Dean's blog post, where he envisions "pathways" as a high-level concept for general AI models. Our proposed Multimodal Pathway Transformer is a novel approach, and we are delighted to discover that some of its effects align with Jeff Dean's high-level vision, such as training a single model to do many things, enabling multiple senses, and making models sparse and efficient. Multimodal Pathway Transformer can be seen as an initial exploration of this "pathways" concept in the context of basic Transformer models and multimodal learning. Read more about Jeff Dean's concept in his blog post.

Abstract

We propose to improve transformers of a specific modality with irrelevant data from other modalities, e.g, improve an ImageNet model with audio or point cloud datasets. We would like to highlight that the data samples of the target modality are irrelevant to the other modalities, which distinguishes our method from other works utilizing paired (e.g. CLIP) or interleaved data of different modalities. We propose a methodology named Multimodal Pathway - given a target modality and a transformer designed for it, we use an auxiliary transformer trained with data of another modality and construct pathways to connect components of the two models so that data of the target modality can be processed by both models. In this way, we utilize the universal sequence-to-sequence modeling abilities of transformers obtained from two modalities. As a concrete implementation, we use a modality-specific tokenizer and task-specific head as usual but utilize the transformer blocks of the auxiliary model via a proposed method named Cross-Modal Re-parameterization, which exploits the auxiliary weights without any inference costs. On the image, point cloud, video, and audio recognition tasks, we observe significant and consistent performance improvements with irrelevant data from other modalities.

Model Zoo

Model Modality Pretraining #Param Google Drive Tencent Cloud
ViT-B16 Image MAE 86M ckpt ckpt
Audio ViT-B16 Audio Audio MAE 86M ckpt ckpt
Point ViT-B16 Point Point MAE 86M ckpt ckpt
Video ViT-B16 Video Video MAE 86M ckpt ckpt

Usage

  • Demo of use:

    #  Create Model
    import torch,timm
    model = timm.create_model("vit_base_patch16_224",pretrained = False)
    aux_model = timm.create_model("vit_base_patch16_224",pretrained = False)
    #  Load Pretrained Models
    pretrained_state_dict = torch.load("Image_ViT_B16.pth")
    aux_state_dict = torch.load("Audio_ViT_B16.pth")
    model.load_state_dict(pretrained_state_dict, strict=True)
    aux_model.load_state_dict(aux_state_dict, strict=True)
    #  Construct Multimodal Pathway
    from multimodal_pathway import reparameterize_aux_into_target_model
    reparameterize_aux_into_target_model(model, aux_model, layer_names=('attn.qkv', 'attn.proj', 'mlp.fc1','mlp.fc2'))
  • For image recognition, please refer to image doc.

  • For video recognition, please refer to video doc.

  • For point cloud analysis, please refer to pcd doc.

  • For audio recognition, please refer to audio doc.

BibTeX

If you find our work useful, please kindly cite:
@article{zhang2024multimodal,
      title={Multimodal Pathway: Improve Transformers with Irrelevant Data from Other Modalities},
      author={Zhang, Yiyuan and Ding, Xiaohan and Gong, Kaixiong and Ge, Yixiao and Shan, Ying and Yue, Xiangyu},
      journal={arXiv preprint arXiv:2401.14405},
      year={2024}
    }

About

[CVPR'24] Multimodal Pathway: Improve Transformers with Irrelevant Data from Other Modalities

Topics

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published