Why LLaMA3-8B after GPTQ test in wikitext2 so bad?

[lyx-111111]

In Paper《How Good Are Low-bit Quantized LLAMA3 Models?An Empirical Study》,LLAMA3-8B after GPTQ ppl in wikitext2 should be 6.5，but I use AutoGPTQ the result is 17.5, who can tell me the difference or answer?

[Qubitium]

Pass at least 128 examples of sufficient length to quantize()

[wenhuach21]

In my opinion, using perplexity (ppl) as a metric might not be optimal due to its sensitivity to outliers, and a high perplexity score doesn't necessarily indicate poor model performance.

After thoroughly analyzing extensive datasets using various weight-only algorithms and assessing perplexities on datasets like Wikitext-2, PTB-new, and C4-new with GPTQ code and wiki ppl using LM-eval, we discovered that many algorithms, including floating models (though the probability is low), encounter this issue. We hypothesize that this issue stems from the mathematical calculation of perplexity (https://huggingface.co/docs/transformers/en/perplexity). For instance, even a low probability of a single token could significantly inflate the perplexity score.

[davidgxue]

#657
this maybe related

[lyx-111111]

I change wikiText2 as calibration dataset, then the ppl is 6.62,which use c4 as calibration dataset is 17.5. Now the ppl can basically met expectations. I didn't realize before that the calibration dataset has such a significant impact on the quantization results. Do we need to quantifybased on the actual dataset every time we use GPTQ?

[Qubitium]

@lyx-111111 Wikitext2 is just a generic dataset and not guaranteed to work the best for all models. For best result, use a dataset that the model was pretrained on. Unfortunately dataset is often private/proprietary secret and thus lots of people use "wikitext2" as alternative. So yes, you need to use/find closest dataset for each model to get best calibration result.

[wenhuach21]

I change wikiText2 as calibration dataset, then the ppl is 6.62,which use c4 as calibration dataset is 17.5. Now the ppl can basically met expectations. I didn't realize before that the calibration dataset has such a significant impact on the quantization results. Do we need to quantifybased on the actual dataset every time we use GPTQ?

Hi, have you evaluated the models with the accuracy like mmlu, I suspect there won't be significant differences. Using the same dataset for both calibration and evaluation could lead to overfitting, resulting in poor generalization when applied to other tasks. We have already observed this issue with an 'SOTA' paper.

[Qubitium]

Using the same dataset for both calibration and evaluation could lead to overfitting, resulting in poor generalization when applied to other tasks. We have already observed this issue with an 'SOTA' paper.

@wenhuach21 Do you have a link to the SOTA paper? I really want to check it out. Also would a larger nsamples help with the overfittiing on quantization?

[wenhuach21]

Using the same dataset for both calibration and evaluation could lead to overfitting, resulting in poor generalization when applied to other tasks. We have already observed this issue with an 'SOTA' paper.

@wenhuach21 Do you have a link to the SOTA paper? I really want to check it out. Also would a larger nsamples help with the overfittiing on quantization?

I apologize for any confusion. What I meant is that some papers utilize WikiText for calibration and solely report WikiText perplexity, which may appear satisfactory but doesn't generalize effectively. Based on our data on llama2, increasing the sample size(128->512) could enhance average accuracy by an absolute 0.1-0.2 sometimes at W4G128, but this comes with about 1x tuning cost for GPTQ. And we do find that increase the samples size could have a big help occasionally in some other scenarios.

In this context, what I want to emphasize is that perplexity is highly sensitive to outliers. A large perplexity value may be caused by a low probability of a single token and might not necessarily indicate significant issues in real-world scenarios. We've observed this phenomenon across multiple quantization algorithms, where perplexity appears to be high on certain datasets, despite the accuracy remaining consistently good.