New context size limit for Llama 3 in settings

[sdudzin]

What happened?

LLama 3 has new context size of 8192 as opposed to 4096 for Llama 2. However the settings dialog limits the context size of 4096 max, also the text description says the limit is 2048.

Relevant log output or stack trace

No response

Steps to reproduce

No response

CodeGPT version

2.7.1

Operating System

macOS

[PhilKes]

You are right, the UI text is a bit misleading/unhelpful and the context size should be model dependent.
The question is should we define hard context size limits at all?
The thing is you can use any model with a larger context size that they have been trained on, usually completion quality just gets much worse at some point, but still if a user wants to do that, why not?

I think I would prefer to not put a hard limit onto the prompt context size input field, and instead only put a hint below it with the context size the model has been trained on (e.g. for Llama 3 8k, CodeQwen 64k, etc.), or even set it as a default value, instead of defining a max value.