This repository includes a new method for privately fine-tuning large language models (LLM) using Edgeworth Accountant.
-
The code is based on the implementation provided in https://github.com/dayu11/Differentially-Private-Deep-Learning/tree/main/language.
-
Edgeworth Accountant is based on an implementation from https://github.com/HuaWang-wharton/EdgeworthAccountant.
-
roBERTa LLM can be downloaded from https://github.com/pytorch/fairseq/tree/master/examples/roberta