attn impl to sdpa...

[saa1028]

new version of transfomer, no need to use BetterTransformer, try setting attn impl to sdpa...
attn imp: <class 'transformers.models.llama.modeling_llama.LlamaSdpaAttention'>

[profintegra]

I have the same issue

[profintegra]

Solution:
in your python code,
insert line:
model.tokenizer.pad_token = model.tokenizer.eos_token
before this line:
input_tokens = model.tokenizer(input_text, ......

[ahmedbr]

I have same problem. Any updates on this?

[leedahae340]

这个不是问题，和这里有关系max_new_tokens=20，如果是20，就要跑20次，如果是200，就要跑200次。。。
有点慢