param_efficient_continual.md

Parameter Efficient Learning and Continual Learning

Adapter-based Parameter Efficient Learning

• [2019 ICML] Parameter-Efficient Transfer Learning for NLP, [paper], [bibtex], [slides], sources: [google-research/adapter-bert].
• [2019 ICML] BERT and PALs: Projected Attention Layers for Efficient Adaptation in Multi-Task Learning, [paper], [bibtex], [supplementary], sources: [AsaCooperStickland/Bert-n-Pals].
• [2020 ArXiv] Continual Learning in Task-Oriented Dialogue Systems, [paper], [bibtex].
• [2020 ArXiv] Orthogonal Language and Task Adapters in Zero-Shot Cross-Lingual Transfer, [paper], [bibtex].
• [2020 EMNLP] AdapterHub: A Framework for Adapting Transformers, [paper], [bibtex], [homepage], sources: [adapter-hub].
• [2020 EMNLP] Exploring Versatile Generative Language Model Via Parameter-Efficient Transfer Learning, [paper], [bibtex], sources: [zlinao/VGLM].
• [2020 EMNLP] MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfer, [paper], [bibtex], [homepage], sources: [adapter-hub].
• [2020 EMNLP] UDapter: Language Adaptation for Truly Universal Dependency Parsing, [paper], [bibtex], sources: [ahmetustun/udapter].
• [2020 DeeLIO] Common Sense or World Knowledge? Investigating Adapter-Based Knowledge Injection into Pretrained Transformers, [paper], [bibtex], sources: [wluper/retrograph].
• [2021 EACL] AdapterFusion: Non-Destructive Task Composition for Transfer Learning, [paper], [bibtex], [homepage], sources: [adapter-hub].
• [2021 ArXiv] COMPACTER: Efficient Low-Rank Hypercomplex Adapter Layers, [paper], [bibtex], sources: [rabeehk/compacter].
• [2021 ACL] K-ADAPTER: Infusing Knowledge into Pre-Trained Models with Adapters, [paper], [bibtex], sources: [microsoft/K-Adapter].
• [2021 ACL] On the Effectiveness of Adapter-based Tuning for Pretrained Language Model Adaptation, [paper], [bibtex].
• [2021 ACL] Parameter-efficient Multi-task Fine-tuning for Transformers via Shared Hypernetworks, [paper], [bibtex], sources: [rabeehk/hyperformer].
• [2021 ACL] Robust Transfer Learning with Pretrained Language Models through Adapters, [paper], [bibtex], sources: [WinnieHAN/Adapter-Robustness].
• [2021 EMNLP] Learn Continually, Generalize Rapidly: Lifelong Knowledge Accumulation for Few-shot Learning, [paper], [bibtex], sources: [INK-USC/CLIF].
• [2021 AAAI] The Adapter-Bot: All-In-One Controllable Conversational Model, [paper], [bibtex], sources: [HLTCHKUST/adapterbot].

Prompt-based Parameter Efficient Learning

• [2021 NAACL] How Many Data Points is a Prompt Worth?, [paper], [bibtex], sources: [TevenLeScao/pet].
• [2021 ACL] Prefix-Tuning: Optimizing Continuous Prompts for Generation, [paper], [bibtex], sources: [XiangLi1999/PrefixTuning].
• [2021 ArXiv] GPT Understands, Too, [paper], [bibtex], sources: [THUDM/P-tuning].
• [2021 ArXiv] Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing, [paper], [bibtex], [homepage], sources: [pfliu-nlp/NLPedia-Pretrain].

Continual Learning (Life-long Learning)

• Contniual Learning resources: [xialeiliu/Awesome-Incremental-Learning].
• [2016 PNAS] Overcoming Catastrophic Forgetting in Neural Networks, [paper], [bibtex], [supplementary], sources: [ariseff/overcoming-catastrophic], [AntiAegis/Overcoming-Catastrophic-Forgetting].
• [2017 NIPS] Overcoming Catastrophic Forgetting by Incremental Moment Matching, [paper], [bibtex], sources: [btjhjeon/IMM_tensorflow].
• [2018 ArXiv] Overcoming Catastrophic Forgetting by Soft Parameter Pruning, [paper], [bibtex], sources: [lehaifeng/Learning_by_memory].
• [2018 ICML] Overcoming Catastrophic Forgetting with Hard Attention to the Task, [paper], [bibtex], [supplementary], sources: [joansj/hat].
• [2019 ICLR] Overcoming Catastrophic Forgetting for Continual Learning via Model Adaptation, [paper], [bibtex], sources: [morning-dews/PGMA_tensorflow].
• [2019 NAACL] Overcoming Catastrophic Forgetting During Domain Adaptation of Neural Machine Translation, [paper], [bibtex].
• [2019 ICML] Learn to Grow: A Continual Structure Learning Framework for Overcoming Catastrophic Forgetting, [paper], [bibtex], [supplementary], sources: [xilaili/LearnToGrow].
• [2020 ICLR] BatchEnsemble: An Alternative Approach to Efficient Ensemble and Lifelong Learning, [paper], [bibtex], sources: [google/edward2].
• [2020 ICLR] Continual Learning with Hypernetworks, [paper], [bibtex], sources: [chrhenning/hypercl].
• [2020 ICLR] LAMOL: LAnguage MOdeling for Lifelong Language Learning, [paper], [bibtex], sources: [chho33/LAMOL].
• [2020 EMNLP] Continual Learning for Natural Language Generation in Task-oriented Dialog Systems, [paper], [bibtex], sources: [MiFei/Continual-Learning-for-NLG].
• [2020 ArXiv] Continual Learning in Task-Oriented Dialogue Systems, [paper], [bibtex].
• [2021 NAACL] Continual Learning for Text Classification with Information Disentanglement Based Regularization, [paper], [bibtex], sources: [GT-SALT/IDBR].
• [2021 NeurIPS] Towards a robust experimental framework and benchmark for lifelong language learning, [paper], [bibtex], [dataset], sources: [AmanDaVinci/lifelong-learning-baselines], [AmanDaVinci/lifelong-learning-limitations].
• [2021 ICML] Not All Memories are Created Equal: Learning to Forget by Expiring, [paper], [bibtex], [Supplementary], [slides], sources: [lucidrains/learning-to-expire-pytorch].

Others

• [2021 EMNLP] UNKs Everywhere: Adapting Multilingual Language Models to New Scripts, [paper], [bibtex], sources: [Adapter-Hub/UNKs_everywhere].
• [2021 ICLR] Beyond Fully-Connected Layers with Quaternions: Parameterization of Hypercomplex Multiplications with 1/n Parameters, [paper], [bibtex], sources: [demegire/Parameterization-of-Hypercomplex-Multiplications].
• [2021 ACL] Parameter-Efficient Transfer Learning with Diff Pruning, [paper], [bibtex], sources: [dguo98/DiffPruning].
• [2021 ArXiv] BitFit: Simple Parameter-efficient Fine-tuningfor Transformer-based Masked Language-models, [paper], [bibtex], sources: [benzakenelad/BitFit].