How to use Huggingface Datasets?

[ms337]

How can we train on custom Huggingface datasets with this?

I could not find the code that takes the HF dataset outputted from pretokenize.py and converts to the sharded dataset.

But maybe Im missing something!

Thanks for this project btw!

[zanussbaum]

hey sorry for the delay @ms337 ! The pretokenize.py script is only used for MLM pretraining. If you have a dataset you want to train with a MLM objective, you can sub it here

If you are trying to do contrastive training, the dataset needs to be in a slightly different format which is roughly a gzipped jsonl file where the lines look something like

{"query": "what is the capital of france?", "document": "the capital of france is paris", "negatives": ["list of hard negatives"], 'metadata': {'objective': {'self': [], 'paired': [['query', 'document']], 'triplet': [['query', 'document', 'negatives']]}}}

The metadata field defines what columns to look for for each objective that eventually gets defined in the data config (e.g. https://github.com/nomic-ai/contrastors/blob/main/src/contrastors/configs/data/finetune_triplets.yaml#L15)