Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

BPE pretokenizer - add support for command-r-plus and command-r models #7041

Closed
wants to merge 0 commits into from
Closed

BPE pretokenizer - add support for command-r-plus and command-r models #7041

wants to merge 0 commits into from

Conversation

sealad886
Copy link

Fixes #7030
Fixes #7040

Added both command-r and command-r-plus pretokenizer regex's to the model, updated the files as recommended in (120cf37).

Tests compiled at ./tests/test-tokenizer-o passed for both models.

@dranger003
Copy link
Contributor

There is already a PR for this #7033.

@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented May 2, 2024

📈 llama.cpp server for bench-server-baseline on Standard_NC4as_T4_v3 for phi-2-q4_0: 550 iterations 🚀

Expand details for performance related PR only
  • Concurrent users: 8, duration: 10m
  • HTTP request : avg=8495.23ms p(95)=20849.85ms fails=, finish reason: stop=479 truncated=71
  • Prompt processing (pp): avg=96.13tk/s p(95)=378.82tk/s
  • Token generation (tg): avg=32.63tk/s p(95)=47.83tk/s
  • ggml-org/models/phi-2/ggml-model-q4_0.gguf parallel=8 ctx-size=16384 ngl=33 batch-size=2048 ubatch-size=256 pp=1024 pp+tg=2048 branch=master commit=0d1a0164e7a86b63053230ae962abcc3cdaf6df7

prompt_tokens_seconds

More 
---
config:
    xyChart:
        titleFontSize: 12
        width: 900
        height: 600
    themeVariables:
        xyChart:
            titleColor: "#000000"
---
xychart-beta
    title "llama.cpp bench-server-baseline on Standard_NC4as_T4_v3
 duration=10m 550 iterations"
    y-axis "llamacpp:prompt_tokens_seconds"
    x-axis "llamacpp:prompt_tokens_seconds" 1714648467 --> 1714649097
    line [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 619.38, 619.38, 619.38, 619.38, 619.38, 503.23, 503.23, 503.23, 503.23, 503.23, 533.6, 533.6, 533.6, 533.6, 533.6, 596.15, 596.15, 596.15, 596.15, 596.15, 633.36, 633.36, 633.36, 633.36, 633.36, 640.06, 640.06, 640.06, 640.06, 640.06, 657.46, 657.46, 657.46, 657.46, 657.46, 676.7, 676.7, 676.7, 676.7, 676.7, 696.37, 696.37, 696.37, 696.37, 696.37, 698.87, 698.87, 698.87, 698.87, 698.87, 736.94, 736.94, 736.94, 736.94, 736.94, 771.44, 771.44, 771.44, 771.44, 771.44, 823.53, 823.53, 823.53, 823.53, 823.53, 830.21, 830.21, 830.21, 830.21, 830.21, 834.12, 834.12, 834.12, 834.12, 834.12, 831.56, 831.56, 831.56, 831.56, 831.56, 847.0, 847.0, 847.0, 847.0, 847.0, 848.17, 848.17, 848.17, 848.17, 848.17, 853.2, 853.2, 853.2, 853.2, 853.2, 853.26, 853.26, 853.26, 853.26, 853.26, 852.58, 852.58, 852.58, 852.58, 852.58, 855.25, 855.25, 855.25, 855.25, 855.25, 849.78, 849.78, 849.78, 849.78, 849.78, 851.58, 851.58, 851.58, 851.58, 851.58, 852.04, 852.04, 852.04, 852.04, 852.04, 861.19, 861.19, 861.19, 861.19, 861.19, 859.48, 859.48, 859.48, 859.48, 859.48, 859.47, 859.47, 859.47, 859.47, 859.47, 863.12, 863.12, 863.12, 863.12, 863.12, 863.69, 863.69, 863.69, 863.69, 863.69, 861.9, 861.9, 861.9, 861.9, 861.9, 857.38, 857.38, 857.38, 857.38, 857.38, 852.8, 852.8, 852.8, 852.8, 852.8, 860.08, 860.08, 860.08, 860.08, 860.08, 858.72, 858.72, 858.72, 858.72, 858.72, 857.89, 857.89, 857.89, 857.89, 857.89, 855.89, 855.89, 855.89, 855.89, 855.89, 857.93, 857.93, 857.93, 857.93, 857.93, 858.22, 858.22, 858.22, 858.22, 858.22, 858.07, 858.07, 858.07, 858.07, 858.07, 830.46, 830.46, 830.46, 830.46, 830.46, 828.38, 828.38, 828.38, 828.38, 828.38, 808.55, 808.55, 808.55, 808.55, 808.55, 807.84, 807.84, 807.84, 807.84, 807.84, 806.15, 806.15, 806.15, 806.15, 806.15, 810.81, 810.81, 810.81, 810.81, 810.81, 810.44, 810.44, 810.44, 810.44, 810.44, 813.46, 813.46, 813.46, 813.46, 813.46, 815.81, 815.81, 815.81, 815.81, 815.81, 819.77, 819.77, 819.77, 819.77, 819.77, 822.75, 822.75, 822.75, 822.75, 822.75, 823.54, 823.54, 823.54, 823.54, 823.54, 825.21, 825.21, 825.21, 825.21, 825.21, 829.8, 829.8, 829.8, 829.8, 829.8, 830.9, 830.9, 830.9, 830.9, 830.9, 829.77, 829.77, 829.77, 829.77, 829.77, 831.07, 831.07, 831.07, 831.07, 831.07, 832.6, 832.6, 832.6, 832.6, 832.6, 833.32, 833.32, 833.32, 833.32, 833.32, 835.55, 835.55, 835.55, 835.55, 835.55, 835.87]
predicted_tokens_seconds
More 
---
config:
    xyChart:
        titleFontSize: 12
        width: 900
        height: 600
    themeVariables:
        xyChart:
            titleColor: "#000000"
---
xychart-beta
    title "llama.cpp bench-server-baseline on Standard_NC4as_T4_v3
 duration=10m 550 iterations"
    y-axis "llamacpp:predicted_tokens_seconds"
    x-axis "llamacpp:predicted_tokens_seconds" 1714648467 --> 1714649097
    line [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 42.47, 42.47, 42.47, 42.47, 42.47, 33.89, 33.89, 33.89, 33.89, 33.89, 32.43, 32.43, 32.43, 32.43, 32.43, 35.81, 35.81, 35.81, 35.81, 35.81, 35.12, 35.12, 35.12, 35.12, 35.12, 35.75, 35.75, 35.75, 35.75, 35.75, 36.7, 36.7, 36.7, 36.7, 36.7, 36.83, 36.83, 36.83, 36.83, 36.83, 36.66, 36.66, 36.66, 36.66, 36.66, 36.19, 36.19, 36.19, 36.19, 36.19, 35.94, 35.94, 35.94, 35.94, 35.94, 35.0, 35.0, 35.0, 35.0, 35.0, 34.13, 34.13, 34.13, 34.13, 34.13, 33.47, 33.47, 33.47, 33.47, 33.47, 33.69, 33.69, 33.69, 33.69, 33.69, 34.01, 34.01, 34.01, 34.01, 34.01, 33.5, 33.5, 33.5, 33.5, 33.5, 33.49, 33.49, 33.49, 33.49, 33.49, 33.16, 33.16, 33.16, 33.16, 33.16, 32.77, 32.77, 32.77, 32.77, 32.77, 32.65, 32.65, 32.65, 32.65, 32.65, 32.72, 32.72, 32.72, 32.72, 32.72, 32.61, 32.61, 32.61, 32.61, 32.61, 32.62, 32.62, 32.62, 32.62, 32.62, 32.65, 32.65, 32.65, 32.65, 32.65, 32.59, 32.59, 32.59, 32.59, 32.59, 32.25, 32.25, 32.25, 32.25, 32.25, 32.27, 32.27, 32.27, 32.27, 32.27, 32.46, 32.46, 32.46, 32.46, 32.46, 32.58, 32.58, 32.58, 32.58, 32.58, 32.66, 32.66, 32.66, 32.66, 32.66, 32.78, 32.78, 32.78, 32.78, 32.78, 32.59, 32.59, 32.59, 32.59, 32.59, 32.49, 32.49, 32.49, 32.49, 32.49, 32.32, 32.32, 32.32, 32.32, 32.32, 32.1, 32.1, 32.1, 32.1, 32.1, 32.09, 32.09, 32.09, 32.09, 32.09, 32.14, 32.14, 32.14, 32.14, 32.14, 32.29, 32.29, 32.29, 32.29, 32.29, 32.3, 32.3, 32.3, 32.3, 32.3, 32.43, 32.43, 32.43, 32.43, 32.43, 32.12, 32.12, 32.12, 32.12, 32.12, 31.81, 31.81, 31.81, 31.81, 31.81, 31.69, 31.69, 31.69, 31.69, 31.69, 30.39, 30.39, 30.39, 30.39, 30.39, 30.37, 30.37, 30.37, 30.37, 30.37, 30.37, 30.37, 30.37, 30.37, 30.37, 30.51, 30.51, 30.51, 30.51, 30.51, 30.54, 30.54, 30.54, 30.54, 30.54, 30.64, 30.64, 30.64, 30.64, 30.64, 30.65, 30.65, 30.65, 30.65, 30.65, 30.62, 30.62, 30.62, 30.62, 30.62, 30.36, 30.36, 30.36, 30.36, 30.36, 30.35, 30.35, 30.35, 30.35, 30.35, 30.42, 30.42, 30.42, 30.42, 30.42, 30.52, 30.52, 30.52, 30.52, 30.52, 30.63, 30.63, 30.63, 30.63, 30.63, 30.74, 30.74, 30.74, 30.74, 30.74, 30.75, 30.75, 30.75, 30.75, 30.75, 30.81, 30.81, 30.81, 30.81, 30.81, 30.81]

Details

kv_cache_usage_ratio

More 
---
config:
    xyChart:
        titleFontSize: 12
        width: 900
        height: 600
    themeVariables:
        xyChart:
            titleColor: "#000000"
---
xychart-beta
    title "llama.cpp bench-server-baseline on Standard_NC4as_T4_v3
 duration=10m 550 iterations"
    y-axis "llamacpp:kv_cache_usage_ratio"
    x-axis "llamacpp:kv_cache_usage_ratio" 1714648467 --> 1714649097
    line [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.17, 0.17, 0.17, 0.17, 0.17, 0.23, 0.23, 0.23, 0.23, 0.23, 0.11, 0.11, 0.11, 0.11, 0.11, 0.2, 0.2, 0.2, 0.2, 0.2, 0.13, 0.13, 0.13, 0.13, 0.13, 0.11, 0.11, 0.11, 0.11, 0.11, 0.12, 0.12, 0.12, 0.12, 0.12, 0.18, 0.18, 0.18, 0.18, 0.18, 0.19, 0.19, 0.19, 0.19, 0.19, 0.14, 0.14, 0.14, 0.14, 0.14, 0.19, 0.19, 0.19, 0.19, 0.19, 0.22, 0.22, 0.22, 0.22, 0.22, 0.31, 0.31, 0.31, 0.31, 0.31, 0.15, 0.15, 0.15, 0.15, 0.15, 0.15, 0.15, 0.15, 0.15, 0.15, 0.17, 0.17, 0.17, 0.17, 0.17, 0.25, 0.25, 0.25, 0.25, 0.25, 0.26, 0.26, 0.26, 0.26, 0.26, 0.23, 0.23, 0.23, 0.23, 0.23, 0.17, 0.17, 0.17, 0.17, 0.17, 0.13, 0.13, 0.13, 0.13, 0.13, 0.3, 0.3, 0.3, 0.3, 0.3, 0.1, 0.1, 0.1, 0.1, 0.1, 0.18, 0.18, 0.18, 0.18, 0.18, 0.16, 0.16, 0.16, 0.16, 0.16, 0.25, 0.25, 0.25, 0.25, 0.25, 0.18, 0.18, 0.18, 0.18, 0.18, 0.15, 0.15, 0.15, 0.15, 0.15, 0.14, 0.14, 0.14, 0.14, 0.14, 0.13, 0.13, 0.13, 0.13, 0.13, 0.12, 0.12, 0.12, 0.12, 0.12, 0.11, 0.11, 0.11, 0.11, 0.11, 0.17, 0.17, 0.17, 0.17, 0.17, 0.19, 0.19, 0.19, 0.19, 0.19, 0.32, 0.32, 0.32, 0.32, 0.32, 0.16, 0.16, 0.16, 0.16, 0.16, 0.18, 0.18, 0.18, 0.18, 0.18, 0.15, 0.15, 0.15, 0.15, 0.15, 0.12, 0.12, 0.12, 0.12, 0.12, 0.12, 0.12, 0.12, 0.12, 0.12, 0.36, 0.36, 0.36, 0.36, 0.36, 0.52, 0.52, 0.52, 0.52, 0.52, 0.49, 0.49, 0.49, 0.49, 0.49, 0.55, 0.55, 0.55, 0.55, 0.55, 0.21, 0.21, 0.21, 0.21, 0.21, 0.22, 0.22, 0.22, 0.22, 0.22, 0.14, 0.14, 0.14, 0.14, 0.14, 0.16, 0.16, 0.16, 0.16, 0.16, 0.13, 0.13, 0.13, 0.13, 0.13, 0.21, 0.21, 0.21, 0.21, 0.21, 0.25, 0.25, 0.25, 0.25, 0.25, 0.29, 0.29, 0.29, 0.29, 0.29, 0.17, 0.17, 0.17, 0.17, 0.17, 0.2, 0.2, 0.2, 0.2, 0.2, 0.12, 0.12, 0.12, 0.12, 0.12, 0.09, 0.09, 0.09, 0.09, 0.09, 0.14, 0.14, 0.14, 0.14, 0.14, 0.12, 0.12, 0.12, 0.12, 0.12, 0.13, 0.13, 0.13, 0.13, 0.13, 0.13, 0.13, 0.13, 0.13, 0.13, 0.23]
requests_processing
More 
---
config:
    xyChart:
        titleFontSize: 12
        width: 900
        height: 600
    themeVariables:
        xyChart:
            titleColor: "#000000"
---
xychart-beta
    title "llama.cpp bench-server-baseline on Standard_NC4as_T4_v3
 duration=10m 550 iterations"
    y-axis "llamacpp:requests_processing"
    x-axis "llamacpp:requests_processing" 1714648467 --> 1714649097
    line [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 8.0, 8.0, 8.0, 8.0, 8.0, 3.0, 3.0, 3.0, 3.0, 3.0, 7.0, 7.0, 7.0, 7.0, 7.0, 4.0, 4.0, 4.0, 4.0, 4.0, 7.0, 7.0, 7.0, 7.0, 7.0, 6.0, 6.0, 6.0, 6.0, 6.0, 4.0, 4.0, 4.0, 4.0, 4.0, 5.0, 5.0, 5.0, 5.0, 5.0, 8.0, 8.0, 8.0, 8.0, 8.0, 4.0, 4.0, 4.0, 4.0, 4.0, 7.0, 7.0, 7.0, 7.0, 7.0, 4.0, 4.0, 4.0, 4.0, 4.0, 4.0, 4.0, 4.0, 4.0, 4.0, 2.0, 2.0, 2.0, 2.0, 2.0, 5.0, 5.0, 5.0, 5.0, 5.0, 4.0, 4.0, 4.0, 4.0, 4.0, 2.0, 2.0, 2.0, 2.0, 2.0, 8.0, 8.0, 8.0, 8.0, 8.0, 7.0, 7.0, 7.0, 7.0, 7.0, 8.0, 8.0, 8.0, 8.0, 8.0, 2.0, 2.0, 2.0, 2.0, 2.0, 6.0, 6.0, 6.0, 6.0, 6.0, 4.0, 4.0, 4.0, 4.0, 4.0, 5.0, 5.0, 5.0, 5.0, 5.0, 6.0, 6.0, 6.0, 6.0, 6.0, 4.0, 4.0, 4.0, 4.0, 4.0, 4.0, 4.0, 4.0, 4.0, 4.0, 3.0, 3.0, 3.0, 3.0, 3.0, 3.0, 3.0, 3.0, 3.0, 3.0, 5.0, 5.0, 5.0, 5.0, 5.0, 8.0, 8.0, 8.0, 8.0, 8.0, 3.0, 3.0, 3.0, 3.0, 3.0, 3.0, 3.0, 3.0, 3.0, 3.0, 4.0, 4.0, 4.0, 4.0, 4.0, 4.0, 4.0, 4.0, 4.0, 4.0, 5.0, 5.0, 5.0, 5.0, 5.0, 6.0, 6.0, 6.0, 6.0, 6.0, 3.0, 3.0, 3.0, 3.0, 3.0, 2.0, 2.0, 2.0, 2.0, 2.0, 7.0, 7.0, 7.0, 7.0, 7.0, 8.0, 8.0, 8.0, 8.0, 8.0, 6.0, 6.0, 6.0, 6.0, 6.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 8.0, 5.0, 5.0, 5.0, 5.0, 5.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 6.0, 4.0, 4.0, 4.0, 4.0, 4.0, 5.0, 5.0, 5.0, 5.0, 5.0, 5.0, 5.0, 5.0, 5.0, 5.0, 8.0, 8.0, 8.0, 8.0, 8.0, 7.0, 7.0, 7.0, 7.0, 7.0, 2.0, 2.0, 2.0, 2.0, 2.0, 4.0, 4.0, 4.0, 4.0, 4.0, 4.0, 4.0, 4.0, 4.0, 4.0, 3.0, 3.0, 3.0, 3.0, 3.0, 8.0, 8.0, 8.0, 8.0, 8.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 7.0, 8.0, 8.0, 8.0, 8.0, 8.0, 2.0]

@compilade compilade mentioned this pull request May 3, 2024
Merged
JoanFM
JoanFM reviewed May 3, 2024
View reviewed changes
convert-hf-to-gguf.py Outdated
if chkhsh == "9c2227e4dd922002fb81bde4fc02b0483ca4f12911410dee2255e4987644e3f8":
# ref: https://huggingface.co/CohereForAI/c4ai-command-r
res = "command-r"
if chkhsh == "9c2227e4dd922002fb81bde4fc02b0483ca4f12911410dee2255e4987644e3f8":
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

just curious, aren't they the exact same hash?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, they are.

@slaren slaren mentioned this pull request May 3, 2024
Merged
@LostRuins
Copy link
Collaborator

Is this duplicated in #7063 ?

@dranger003
Copy link
Contributor

@LostRuins Yes, this is a duplicate.

@sealad886 sealad886 closed this May 6, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@JoanFM JoanFM JoanFM left review comments

@teleprint-me teleprint-me teleprint-me left review comments

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
5 participants
@sealad886 @dranger003 @LostRuins @JoanFM @teleprint-me