Doubao "Query or prefix prompt is too long“ error

[rocing]

Self Checks

• This is only for bug report, if you would like to ask a question, please head to Discussions.
• I have searched for existing issues search for existing issues, including closed ones.
• I confirm that I am using English to submit this report (我已阅读并同意 Language Policy).
• Please do not modify this template :) and fill in all the required fields.

Dify version

0.6.8

Cloud or Self Hosted

Self Hosted (Source)

Steps to reproduce

Tests of various versions of Doubao have yielded "Query or prefix prompt is too long, you can reduce the prefix prompt, or shrink the max token, or switch to a llm with a larger token limit size."
The actual committed prompt length is nowhere near above the maximum limit, and I think this might be a bug.

✔️ Expected Behavior

Submit normally

❌ Actual Behavior

Query or prefix prompt is too long, you can reduce the prefix prompt, or shrink the max token, or switch to a llm with a larger token limit size.

[dosubot]

It appears you're encountering an error due to the total number of tokens for your prompt and the model's response exceeding the Doubao model's context size in Dify version 0.6.8. To address this, you should:

1. Verify Model Configuration: Ensure the model_context_tokens and max_tokens settings in your model configuration accurately reflect the model's limits and your requirements. The model's context size must be large enough to accommodate both your prompt and the desired output.

2. Adjust Prompt Length or Max Tokens: If the sum of your prompt tokens and max_tokens exceeds the model's context size, reduce your prompt's length or lower the max_tokens parameter. This adjustment is crucial to prevent exceeding the model's token limit.

3. Model Limitations: Be aware of the fixed maximum context size of your model. If your requirements frequently approach this limit, consider switching to a model with a larger token capacity.

4. Debugging: Implement logging around token calculations to monitor the usage of tokens by your prompts and the space left for responses. This can help pinpoint unexpected token consumption or inputs that are larger than expected.

The default token limit for the Doubao model is 4096 for both max_prompt_tokens and max_new_tokens. If adjusting the prompt length and max_tokens does not resolve the issue, further investigation into the specific prompts and the Doubao model configuration might be necessary to identify the root cause.

Details

Sources

• Dify : Query or prefix is too long
• 对话内容超限 Query or prefix is too long
• api/core/app/apps/base_app_runner.py
• api/core/model_runtime/model_providers/volcengine_maas/llm/models.py
• api/core/model_runtime/model_providers/openai_api_compatible/llm/llm.py
• api/core/model_runtime/model_providers/openrouter/llm/llm.py

About Dosu
This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

To continue the conversation, mention @dosu.

[rocing]

After modifying the custom tokens configuration, I found that the model began to babble 😄