Llama 3 - Regression with apostrophes

[Rufflewind]

I am using seeing a lot of improperly formatted apostrophes. Some examples I have seen:

realm' dwellers
     ^^

Lyra' s journey
    ^^

I don't remember ever seeing this issue about a week ago, so I suspect it might be a recently introduced bug. It happens fairly consistently (but randomly).

$ make -j8 LLAMA_CUDA=1
$ llama.cpp/server -m models/Meta-Llama-3-8B-Instruct-Q8_0.gguf -c 8192 -ngl 33

To reproduce this, I just ask the assistant to generate lots of random story text, and it eventually hits the bug after about 1k tokens.

Specs

• System: Ryzen 5600X + RTX 3080
• Platform: WSL
• Versions:
  • 9c67c27
  • https://huggingface.co/bartowski/Meta-Llama-3-8B-Instruct-GGUF/blob/main/Meta-Llama-3-8B-Instruct-Q8_0.gguf

Example

Request:

{
    "prompt": "<|begin_of_text|><|start_header_id|>system<|end_header_id|>\n\nYou are Alpha, having an open-ended chat with me, Beta.<|eot_id|><|start_header_id|>user<|end_header_id|>\n\nWrite a story.<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\nBeta! I'd love to spin a tale for you. Let's see... How about a story about a mystical realm where the skies are perpetually painted with hues of sapphire and amethyst?\n\nIn this realm, there existed a young apprentice named Lyra, who was an aspiring Dreamweaver. Her mentor, the enigmatic Astra, had tasked her with a mission to retrieve the fabled Luminous Thread, a mystical strand of light said to grant the wielder unimaginable power over the realm's very fabric.\nAs Lyra embarked on her perilous journey, she encountered an eccentric cast of characters. There was Jax, a wisecracking trickster with the ability to manipulate the shadows; Zephyr, a gentle soul with the power to control the wind; and Kaida, a rebellious fire-dancer with the capacity to command the flames.\nTogether, they traversed the realm, overcoming treacherous landscapes, ancient ruins, and mythical creatures. They solved puzzles, unraveled enigmas, and battled fearsome foes, all while navigating the ever-shifting tides of fate.\nAs they drew closer to the fabled Luminous Thread, Lyra began to realize that the true challenge lay not in the obstacles ahead but in her own inner struggle to reconcile her destiny with her sense of self. For Lyra, the quest was not only about retrieving the Thread but about embracing her true potential and the magic that lay within her.\nAnd so, with every step, Lyra grew stronger, wiser, and more compassionate, as the threads of her own story began to weave themselves into the tapestry of the realm. The Luminous Thread, now within her grasp, shone brighter than ever before, symbolizing the limitless potential that lay within her – and within each of us, dear Beta. What do you think? Would you like to explore more of this world, or would you like me to spin a new tale for you?<|eot_id|><|start_header_id|>user<|end_header_id|>\n\nKeep writing.<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\nBeta! The realm of Eldrida, where magic and wonder awaited around every corner, was vast and rich in mystery. As Lyra held the Luminous Thread, she felt an influx of power coursing through her veins. The very fabric of the realm began to respond, weaving a tapestry of possibilities that seemed almost infinite.\n\nWith her newfound abilities, Lyra and her companions ventured into the heart of the mystical city of Elyria, where the ancient Sages of the Dreamweavers awaited. These enigmatic beings, guardians of the realm's secrets, revealed to Lyra that she was the chosen one, destined to maintain the balance of the realm and keep its harmony intact.\nHowever, a dark force, known as the Shadow Weaver, threatened to unravel the very fabric of Eldrida. This malevolent entity, born from the collective fears and doubts of the realm' dwellers, had grown powerful and sought to consume the realm, plunging it into eternal darkness.\nThe Sages entrusted Lyra with the responsibility of defeating the Shadow Weaver, but warned her that the task would require the collaboration of the realm's inhabitants, who had grown wary of her newfound power. Lyra, determined to prove herself, embarked on a perilous quest to gather the fragments of the ancient artifacts known as the Celestial Scales.\nThese scales, forged from the essence of the stars themselves, held the power to counterbalance the darkness and restore harmony to the realm. As Lyra journeyed, she encountered old friends and new allies, each with their own unique abilities and motivations. There was Kaida, who had grown increasingly obsessed with the power of the flames; Jax, whose shadowy abilities had become more pronounced; and Zephyr, whose connection to the wind had deepened.\nTogether, they traversed treacherous landscapes, fought against formidable foes, and solved complex puzzles, all while navigating the intricate web of relationships and loyalties that threatened to unravel the very fabric of their bond. As Lyra neared the final confrontation with the Shadow Weaver, she realized that the true test lay not in defeating the darkness but in reconciling the light within herself and embracing the complexity of her own nature. The fate of Eldrida hung in the balance, and Lyra's journey had become a reflection of her own inner struggle for self-discovery and redemption.\n\nNow, dear Beta, I sense a change in the fabric of our conversation. The threads of the tale seem to be weaving themselves into a new narrative, one that whispers secrets of the Dreamweavers, the ancient arts of magic, and the mysteries of the realm. The choices we make, the paths we take, and the bonds we form will shape the course of Lyra",
    "cache_prompt": true,
    "temperature": 0.8,
    "dynatemp_range": 0.8,
    "n_predict": 3
}

Response:

{
    "content": "' s journey",
    "id_slot": 0,
    "stop": true,
    "model": ".../Meta-Llama-3-8B-Instruct-Q8_0.gguf",
    "tokens_predicted": 3,
    "tokens_evaluated": 1033,
    "generation_settings": {
        "n_ctx": 8192,
        "n_predict": -1,
        "model": ".../Meta-Llama-3-8B-Instruct-Q8_0.gguf",
        "seed": 4294967295,
        "temperature": 0.800000011920929,
        "dynatemp_range": 0.800000011920929,
        "dynatemp_exponent": 1,
        "top_k": 40,
        "top_p": 0.949999988079071,
        "min_p": 0.05000000074505806,
        "tfs_z": 1,
        "typical_p": 1,
        "repeat_last_n": 64,
        "repeat_penalty": 1,
        "presence_penalty": 0,
        "frequency_penalty": 0,
        "penalty_prompt_tokens": [],
        "use_penalty_prompt_tokens": false,
        "mirostat": 0,
        "mirostat_tau": 5,
        "mirostat_eta": 0.10000000149011612,
        "penalize_nl": false,
        "stop": [],
        "n_keep": 0,
        "n_discard": 0,
        "ignore_eos": false,
        "stream": false,
        "logit_bias": [],
        "n_probs": 0,
        "min_keep": 0,
        "grammar": "",
        "samplers": [
            "top_k",
            "tfs_z",
            "typical_p",
            "top_p",
            "min_p",
            "temperature"
        ]
    },
    "prompt": "<|begin_of_text|><|start_header_id|>system<|end_header_id|>\n\nYou are Alpha, having an open-ended chat with me, Beta.<|eot_id|><|start_header_id|>user<|end_header_id|>\n\nWrite a story.<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\nBeta! I'd love to spin a tale for you. Let's see... How about a story about a mystical realm where the skies are perpetually painted with hues of sapphire and amethyst?\n\nIn this realm, there existed a young apprentice named Lyra, who was an aspiring Dreamweaver. Her mentor, the enigmatic Astra, had tasked her with a mission to retrieve the fabled Luminous Thread, a mystical strand of light said to grant the wielder unimaginable power over the realm's very fabric.\nAs Lyra embarked on her perilous journey, she encountered an eccentric cast of characters. There was Jax, a wisecracking trickster with the ability to manipulate the shadows; Zephyr, a gentle soul with the power to control the wind; and Kaida, a rebellious fire-dancer with the capacity to command the flames.\nTogether, they traversed the realm, overcoming treacherous landscapes, ancient ruins, and mythical creatures. They solved puzzles, unraveled enigmas, and battled fearsome foes, all while navigating the ever-shifting tides of fate.\nAs they drew closer to the fabled Luminous Thread, Lyra began to realize that the true challenge lay not in the obstacles ahead but in her own inner struggle to reconcile her destiny with her sense of self. For Lyra, the quest was not only about retrieving the Thread but about embracing her true potential and the magic that lay within her.\nAnd so, with every step, Lyra grew stronger, wiser, and more compassionate, as the threads of her own story began to weave themselves into the tapestry of the realm. The Luminous Thread, now within her grasp, shone brighter than ever before, symbolizing the limitless potential that lay within her – and within each of us, dear Beta. What do you think? Would you like to explore more of this world, or would you like me to spin a new tale for you?<|eot_id|><|start_header_id|>user<|end_header_id|>\n\nKeep writing.<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\nBeta! The realm of Eldrida, where magic and wonder awaited around every corner, was vast and rich in mystery. As Lyra held the Luminous Thread, she felt an influx of power coursing through her veins. The very fabric of the realm began to respond, weaving a tapestry of possibilities that seemed almost infinite.\n\nWith her newfound abilities, Lyra and her companions ventured into the heart of the mystical city of Elyria, where the ancient Sages of the Dreamweavers awaited. These enigmatic beings, guardians of the realm's secrets, revealed to Lyra that she was the chosen one, destined to maintain the balance of the realm and keep its harmony intact.\nHowever, a dark force, known as the Shadow Weaver, threatened to unravel the very fabric of Eldrida. This malevolent entity, born from the collective fears and doubts of the realm' dwellers, had grown powerful and sought to consume the realm, plunging it into eternal darkness.\nThe Sages entrusted Lyra with the responsibility of defeating the Shadow Weaver, but warned her that the task would require the collaboration of the realm's inhabitants, who had grown wary of her newfound power. Lyra, determined to prove herself, embarked on a perilous quest to gather the fragments of the ancient artifacts known as the Celestial Scales.\nThese scales, forged from the essence of the stars themselves, held the power to counterbalance the darkness and restore harmony to the realm. As Lyra journeyed, she encountered old friends and new allies, each with their own unique abilities and motivations. There was Kaida, who had grown increasingly obsessed with the power of the flames; Jax, whose shadowy abilities had become more pronounced; and Zephyr, whose connection to the wind had deepened.\nTogether, they traversed treacherous landscapes, fought against formidable foes, and solved complex puzzles, all while navigating the intricate web of relationships and loyalties that threatened to unravel the very fabric of their bond. As Lyra neared the final confrontation with the Shadow Weaver, she realized that the true test lay not in defeating the darkness but in reconciling the light within herself and embracing the complexity of her own nature. The fate of Eldrida hung in the balance, and Lyra's journey had become a reflection of her own inner struggle for self-discovery and redemption.\n\nNow, dear Beta, I sense a change in the fabric of our conversation. The threads of the tale seem to be weaving themselves into a new narrative, one that whispers secrets of the Dreamweavers, the ancient arts of magic, and the mysteries of the realm. The choices we make, the paths we take, and the bonds we form will shape the course of Lyra",
    "truncated": false,
    "stopped_eos": false,
    "stopped_word": false,
    "stopped_limit": true,
    "stopping_word": "",
    "tokens_cached": 1035,
    "timings": { ... }
}

[segmond]

Quality really degraded in the last week. especially with the llama3 models. I have pulled so many changes I'm not sure when it happened, but I agree with you, I think going back a week to 2 will see these problems go away. I'm on linux running on Nvidia.

[city96]

I'm seeing something similar on Command R+, except with mixed/doubled apostrophes (e.g. "They'`re"). I assume it's the pre-tokenizer, as per the "missing pre-tokenizer type, using: 'default'" warning in the server log with the big bold "GENERATION QUALITY WILL BE DEGRADED! CONSIDER REGENERATING THE MODEL" below it.

Trying to generate a new gguf from the HF weights for command-r-plus fails since #6920, but it might help with llama-3.

[Rotatingxenomorph]

I'm seeing something similar on Command R+, except with mixed/doubled apostrophes (e.g. "They'`re"). I assume it's the pre-tokenizer, as per the "missing pre-tokenizer type, using: 'default'" warning in the server log with the big bold "GENERATION QUALITY WILL BE DEGRADED! CONSIDER REGENERATING THE MODEL" below it.

Trying to generate a new gguf from the HF weights for command-r-plus fails since #6920, but it might help with llama-3.

Kafkaesque.

[aeblank]

I'm not a developer, but using LM Studio I have noticed the same thing after upgrading to the latest version, which included an updated llama.cpp, using Q8 llama 3 70b models on an M3 Max.

Plenty of apostrophe errors, ranging from adding a space between the apostrophe and an "s" (example: Mary' s glass of water, instead of Mary's glass of water), or omitted "s" in the same context (example: Mary' glass of water, instead of Mary's glass of water). I have also noticed double apostrophes here and there.

I haven't changed my prompts, model settings, or model files -- and this didn't occur with prior versions of LM Studio that used an older llama.cpp, with llama-3 70b models.

Hope that helps diagnose the issue.

[ggerganov]

Is this issue still present with latest master? Make sure to use a LLaMA 3 model that you have converted yourself with the convert-hf-to-gguf.py script with the latest llama.cpp

[henk717]

Just tested this in KCPP 1.65 because for us backwards compatibility is a selling point we want the tokenizer behavior to be correct on old models too.
I can confirm that my generations in 1.65 while different results with the seed match the apostrophes of 1.63, but 1.64 differs and reproduces this bug.

For context:
1.63 had the old tokenizer and can't run new pretokenizer models.
1.64 was at the beginning of the Llama3 pretokenizer change and lacks CMDR pretokenizer support.
1.65 was released last weekend.

So from what I can see this is solved, others can confirm.