Skip to content

Explore different way to mix speech model(wav2vec2, hubert) and nlp model(BART,T5,GPT) together

Notifications You must be signed in to change notification settings

voidful/SpeechMix

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

53 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

SpeechMix

Explore different way to mix speech model(wav2vec2, hubert) and nlp model(BART,T5,GPT) together.

Implementation of:
Large-Scale Self- and Semi-Supervised Learning for Speech Translation - ACL2021
Multilingual Speech Translation with Efficient Finetuning of Pretrained Models - ACL2021
Lightweight Adapter Tuning for Multilingual Speech Translation - Interspeech 2021
Improving Speech Translation by Understanding and Learning from the Auxiliary Text Translation Task - ACL2021
A General Multi-Task Learning Framework to Leverage Text Data for Speech to Text Tasks - ICASSP 2021

Installation

pip install

pip install speechmix

Build from source

git clone and cd into this project.

pip install -e .

Name the project(!important)

WANDB_PROJECT=amazing

base

python train.py --speech_model_config wav2vec2 \
--nlp_model_config facebook/bart-base \
--SpeechMixEED \
--dataset librispeech_asr \
--field clean \
--train_split train.100 \
--test_split validation \
--batch 3 \
--grad_accum 20 \
--epoch 30 \
--worker 15 \
--share_layer_ratio 0 \
--down_scale 2 \
--lr 4e-5 \
--warmup_steps 500 \
--wandb \
--notes base

About

Explore different way to mix speech model(wav2vec2, hubert) and nlp model(BART,T5,GPT) together

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published