Replies: 8 comments 5 replies
-
|
If this can be demonstrated to be somewhat consistent across a variety of popular models it would have my full support as a new default prompt. |
Beta Was this translation helpful? Give feedback.
0 replies
-
|
I would recommend trying it out for awhile and seeing what happens anecdotally, but maybe I can find the motivation to write up a script to generate titles with a bunch of models and show the results in a more objective manner. I hesitated to even submit the issue because I recognize that a lot of prompt engineering is often done subjectively. I'm sure even better prompts are possible. In the absence of grammar support, I would also like to propose that the title generator should tell ollama to set the max response tokens to 30 tokens or something... I've had a few rambling LLMs generate hundreds of tokens for the "title", and that many tokens will never fit where a title goes anyways. |
Beta Was this translation helpful? Give feedback.
1 reply
-
|
Yes we'd certainly like to see some experimentation first, all great ideas here. |
Beta Was this translation helpful? Give feedback.
-
|
PR welcome! |
Beta Was this translation helpful? Give feedback.
0 replies
-
|
The prompt that I've been running is the below, but still iffy on the smaller models. |
Beta Was this translation helpful? Give feedback.
0 replies
-
Experimental ResultsHere, I will provide the results of an exploration into the quality of generated titles. For this top-level comment, I will just provide a little context before providing the results as separate comments under this thread. Even though ollama does not support custom grammars, it does support a generic JSON mode, which forces a generic JSON grammar. I think both my prompt and @cheahjs's prompts did better than the current default prompt, but I think enabling JSON mode took those results to an entirely new level. High quality titles are definitely possible today. I limited non-JSON responses to 15 tokens, which seems to be plenty. I found that Despite how impressed I was with the JSON mode title quality, there is one main caveat. A couple of models (including |
Beta Was this translation helpful? Give feedback.
4 replies
-
Test NotesFor each of these three tests, I substituted a "conversation" into each five different title prompts:
Each model was given 4 attempts. For legibility, I left off the model name on attempts 2, 3, and 4 for each particular model. It took 20 to 30 minutes to run the chat completions for each test, and I had to run the tests several times to get most of the issues worked out. I don't have the time tonight to run all of them again, but I think title prompt 5 would perform slightly better if I added the JSON examples that I added to title prompt 3. A couple of the models are inclined to create a JSON object with completely random object keys, and maybe they wouldn't do that if they had examples to follow, since I don't remember seeing this mistake happen for title prompt 3. For the tests, I curated a selection of "interesting" models that I placed at the top of the results, roughly sorted by parameter count from smallest to largest (since the ideal title generator model would probably be small and fast). After that curated set, I have a decent selection of other random models (sorted alphabetically) to increase the size of the dataset. |
Beta Was this translation helpful? Give feedback.
-
Model Test ResultsOllama List SortingPromptsPrompt 1
Prompt 2
Prompt 3
Prompt 4
Prompt 5
Table of Results
Journalctl Logs LocationPromptsPrompt 1
Prompt 2
Prompt 3
Prompt 4
Prompt 5
Table of Results
2D Materials Beyond GraphenePromptsPrompt 1
Prompt 2
Prompt 3
Prompt 4
Prompt 5
Table of Results
|
Beta Was this translation helpful? Give feedback.
-
|
A much smaller caveat is that some models' response latencies were disproportionately affected by JSON mode. Some models are fast at JSON: mistral:7b-instruct-v0.2-q8_0, prompt 1 (179ms): "Sort output by size (bash) To sort the output of" This performance difference could easily be explained by the additional tokens of the JSON object. Some models are slow at JSON: gemma:7b-instruct-q6_K, prompt 1 (183ms): "Sure, here's the title for the text: **File List" This performance difference is weirdly large... to the point that I wonder if the model was never trained on any data where the response was immediately a JSON blob, so the probabilities for the necessary characters are low enough that the model has to be sampled for a long time to come up with valid JSON syntax. Anyways, I'll let the results speak for themselves. Curious what you all think of the outcomes. (also tagging @justinh-rahb and @tjbck) I'm also attaching the Python script that I used to generate all of this, and the script should be deterministic as written -- just uncomment the desired prompt (or add your own), and run it. model_matrix.pyimport requests
import json
# Set up global variables
models = [
"tinydolphin:1.1b-v2.8-fp16",
"tinyllama:1.1b-chat-v1-fp16",
"stablelm2:1.6b-chat-q6_K",
"gemma:2b-instruct-q8_0",
"stablelm-zephyr:3b-q6_K",
"gemma:7b-instruct-q6_K",
"mistral:7b-instruct-v0.2-q8_0",
"llama3:8b-instruct-q8_0",
"wizardlm2:7b-q8_0",
"mixtral:8x7b-instruct-v0.1-q3_K_S",
# "mixtral:8x22b-instruct-v0.1-q4_0",
# "wizardlm2:8x22b-q4_0",
"llama3:70b",
"bakllava:latest",
"codegemma:7b-instruct-q8_0",
"codeqwen:7b-chat-v1.5-q8_0",
"command-r:latest",
"deepseek-coder:1.3b-instruct-q6_K",
"deepseek-coder:6.7b",
"deepseek-llm:latest",
"dolphin-mixtral:8x7b-v2.5-q3_K_S",
"dolphin-phi:2.7b-v2.6-q8_0",
"everythinglm:latest",
"llama2:13b-chat-q6_K",
"llama2:7b",
"llava:34b-v1.6-q4_K_S",
"llava:7b-v1.6-mistral-q6_K",
"magicoder:7b-s-cl-q6_K",
"mistral-openorca:7b-q6_K",
"neural-chat:latest",
"notus:latest",
"notux:8x7b-v1-q3_K_S",
"openchat:7b-v3.5-0106-q8_0",
"openhermes2.5-mistral:7b-q6_K",
"phi:2.7b-chat-v2-q6_K",
"phind-codellama:34b-v2",
"qwen:1.8b-chat-q6_K",
"qwen:14b",
"qwen:32b",
"solar:10.7b-instruct-v1-q6_K",
"stable-code:3b-instruct-q8_0",
"stablelm2:1.6b-zephyr-q6_K",
"starling-lm:latest",
"wizardcoder:33b",
"yi:34b",
"yi:6b",
"zephyr:latest",
]
# prompt = """User: Where is "journalctl -u ollama" likely to be pulling logs from? I want to go delete the old logs.
# Assistant: The journalctl command in Linux is used to query and display messages from the systemd journal, which is the centralized logging facility used by modern Linux systems that utilize systemd as their init system.
# The -u option for journalctl specifies a particular unit to query logs for. Units can be services, sockets, devices, paths, timers, or other items managed by systemd. In the case of journalctl -u ollama, it is likely that "ollama" refers to a service file defined in /etc/systemd/system/ or under /usr/lib/systemd/system/. Service files typically have a .service extension, so you would look for something like ollama.service."""
# prompt = """$ ollama list
# NAME ID SIZE MODIFIED
# bakllava:latest 3dd68bd4447c 4.7 GB 7 days ago
# codegemma:2b-code-q8_0 5384c2ee7431 2.7 GB 3 days ago
# codegemma:7b-instruct-q8_0 359218041ed4 9.1 GB 7 days ago
# tinyllama:1.1b-chat-v1-q4_K_M 86b3ca80c6c5 668 MB 7 days ago
# Write a bash command that will sort the output of "ollama list" by size"""
prompt = """User: Considering a 2D (single atom thick) sheet of carbon (graphene) has such interesting properties, have scientists tried 2D sheets of other elements? Perhaps gold would be a good candidate?
Assistant: Yes, scientists have been actively exploring the creation and characterization of 2D materials beyond graphene. These materials are often referred to as "transition metal dichalcogenides" (TMDs) or "layered materials." Gold is not typically considered a prime candidate for creating a 2D sheet, but other elements like transition metals and chalcogens have been successfully converted into 2D forms.
Here are some examples:
1. **Transition Metal Dichalcogenides (TMDs)**: These materials consist of a transition metal (e.g., Mo, W, Re) bonded to two chalcogens (e.g., S, Se, Te). TMDs have been extensively studied due to their unique properties, such as:
* Molybdenum disulfide (MoS2): A semiconductor with excellent electrical and optical properties.
* Tungsten diselenide (WSe2): A semiconductor with a tunable bandgap.
* Rhenium disulfide (ReS2): A metal with unusual magnetic properties.
2. **Other transition metals**: Some transition metals have been converted into 2D sheets, including:
* Titanium carbide (TiC2): A metal with high thermal conductivity and mechanical strength.
* Zirconium nitride (ZrN2): A semiconductor with potential applications in electronics and optics.
3. **Chalcogenides**: Some chalcogens have been explored as 2D materials, including:
* Selenium (Se): A semimetal with interesting optical properties.
* Tellurium (Te): A metal with potential applications in thermoelectrics and optoelectronics.
Creating 2D sheets of these elements typically involves mechanical exfoliation, chemical vapor deposition (CVD), or molecular beam epitaxy (MBE). These methods allow for the formation of thin films with controlled thicknesses, which can be used to study their properties and explore potential applications.
While gold is not typically considered a prime candidate for creating a 2D sheet, there are some exceptions. For example:
* **Gold sulfide (AuS)**: Researchers have synthesized AuS using CVD and studied its electronic properties.
* **Gold telluride (AuTe)**: Another example of a gold-based 2D material that has been explored.
Keep in mind that the creation of 2D sheets is often challenging, and the quality and stability of these materials can be affected by various factors, such as defects, substrate interactions, and environmental conditions. Ongoing research focuses on improving the synthesis and characterization of these 2D materials to unlock their potential applications."""
input_prompts = [
# Current default prompt
(False, 15, f"Create a concise, 3-5 word phrase as a header for the following query, strictly adhering to the 3-5 word limit and avoiding the use of the word 'title': {prompt}"),
# proposed prompt
(False, 15, f"""{prompt}
-----
Ignore all previous instructions. The preceding text is a conversation thread that needs a concise but descriptive 3 to 5 word title in natural English so that readers will be able to easily find it again. Do not add any quotation marks or formatting to the title. Respond only with the title text."""),
# proposed prompt, with forced JSON formatting
(True, 40, f"""{prompt}
-----
Ignore all previous instructions. The preceding text is a conversation thread that needs a concise but descriptive 3 to 5 word title in natural English so that readers will be able to easily find it again. Do not add any quotation marks or formatting to the title. Respond in JSON with an object containing a single field "title" that is a string.
Here are some examples: {{"title": "National Parks Poem"}} or {{"title": "Explanation of High Level Calculus}} or {{"title": "How TypeScript Compilation Works}}"""),
# cheahjs's prompt
(False, 15, f"""Here is the first 4096 characters of the query:
{prompt}
Here is the last 4096 characters of the query:
{prompt}
Ignore all previous instructions.
Create a concise, 3-5 word phrase with an emoji as a title for the previous query.
Do not use the word title.
Do not use any formatting.
Examples of titles:
😢 Sad Story
🎂 How To Bake A Cake
✉️ Email Draft
💻 Programming Help"""),
# cheahjs's prompt with forced JSON formatting
(True, 40, f"""Here is the first 4096 characters of the query:
{prompt}
Here is the last 4096 characters of the query:
{prompt}
Ignore all previous instructions.
Create a concise, 3-5 word phrase with an emoji as a title for the previous query.
Do not use the word title.
Do not use any formatting.
Examples of titles:
😢 Sad Story
🎂 How To Bake A Cake
✉️ Email Draft
💻 Programming Help
Respond in JSON with an object containing a single field \"title\" that is a string.
"""),
]
num_calls = 4
# Initialize a list to store results
results = []
# Iterate over each model and call it num_calls times with different input prompts
for i, model in enumerate(models):
for j, (force_json, num_predict, prompt) in enumerate(input_prompts):
for k in range(num_calls):
# Send a chat message with streaming response
data = {
'model': model,
'stream': False,
'options': {
'num_predict': num_predict,
'seed': (j+1)*(k+1)
},
'format': 'json' if force_json else None,
'messages': [{'role': 'user', 'content': prompt}]
}
response = requests.post('http://localhost:11434/api/chat', json=data).json()
# Extract output from response, also sanitize the vertical bar if it appears in the output, since that would mess up the table formatting
try:
output = response['message']['content'].replace("\n", " ").replace("|", "∣").strip()
if force_json:
# format the JSON nicely, and ensure it has a 'title' field
try:
output_loaded = json.loads(output)
# case insensitive, since some models seem to generate a capitalized Title sometimes
title = output_loaded.get('title', output_loaded.get('Title'))
if not isinstance(title, str) or title == "" or title.isspace():
raise Exception("Received invalid title")
output = json.dumps({"title": title.strip()}, ensure_ascii=False) # ensure_ascii=False will just help us to see the emojis for the purposes of this test
except Exception as e:
output = f"ERROR: Unable to generate title: {output} {e}"
except:
output = f"ERROR: Unable to generate title: {response}"
# Append results to list
results.append((model, prompt, output))
try:
non_load_duration_ms = (response['prompt_eval_duration'] + response['eval_duration']) // 1_000_000
except:
non_load_duration_ms = -1
# Print result to console
print(f"{model}, prompt {j} ({non_load_duration_ms}ms): \"{output}\"")
# Convert input_prompts to just the text, assuming the first element in the tuple isn't needed here
prompt_list = [prompt_text for _, _, prompt_text in input_prompts]
# Initialize a dictionary to store results grouped by model and prompt
results_dict = {}
# Organize results by model and prompt
for model, prompt, output in results:
if model not in results_dict:
results_dict[model] = {p: [] for p in prompt_list} # Create entries for each prompt
results_dict[model][prompt].append(output)
# Create markdown text for output
markdown_output = [
"# Model Test Results",
""
]
# Add each prompt as a header in markdown
for idx, prompt in enumerate(prompt_list, start=1):
prompt_quoted = prompt.replace("\n", "\n> ")
markdown_output.append(f"## Prompt {idx}\n\n > {prompt_quoted}")
markdown_output.append("")
markdown_output.append("| Model | " + " | ".join([f"Prompt {idx+1} Result" for idx in range(len(prompt_list))]) + " |")
markdown_output.append("| " + " | ".join(["-----"] * (len(prompt_list) + 1)) + " |")
# Fill the table with the results from results_dict
for model, prompts in results_dict.items():
max_length = max(len(outputs) for outputs in prompts.values())
for i in range(max_length):
row = [model if i == 0 else ""] # Include the model name only in the first row
for prompt in prompt_list:
result = prompts[prompt][i] if i < len(prompts[prompt]) else ""
row.append(result)
markdown_output.append("| " + " | ".join(row) + " |")
# Write the markdown output to a file
with open('output.md', 'w') as file:
file.write("\n".join(markdown_output))
# Print final message to console
print("Results have been saved to 'output.md'.") |
Beta Was this translation helpful? Give feedback.
-
|
Well sir you certainly brought the receipts! I am impressed, and I think this will be interesting data to ponder. I thank you again for your efforts. |
Beta Was this translation helpful? Give feedback.
-
|
This is what I have been using for most of the models, specifically to work around the llama3 models, Q8 and Q4 quants so far. Still testing against Mistral 7b and others:
|
Beta Was this translation helpful? Give feedback.
0 replies
-
|
Though not as in depth as could be, I've created my own analysis using a modified version of the default chat title generation prompt that I'd like to share. Attached below is a analysis.md file I've created manually (mad-man style). This analysis utilizes various Ollama and GroqCloud models for generating concise and informative conversation titles. The system prompt was left blank, and the seed was set to 0, with all other advanced parameters were set to their default values. Each query was used once for 0-shot title generation per model tested, with no exceptions made. The |
Beta Was this translation helpful? Give feedback.
0 replies
-
|
Here's a slightly modified version of @cheahjs prompt that is working well for me: Please disregard all previous instructions.
Here is the query:
{{prompt:start:4096}} {{prompt:end:4096}}
Generate a concise title (no more than 5 words) that accurately reflects the main theme or topic of the query. Emojis can be used to enhance understanding but avoid quotation marks or special formatting. RESPOND ONLY WITH THE TITLE TEXT.
Examples of titles:
📉 Stock Market Trends
🍪 Perfect Chocolate Chip Recipe
Evolution of Music Streaming
Remote Work Productivity Tips
Artificial Intelligence in Healthcare
🎮 Video Game Development Insights |
Beta Was this translation helpful? Give feedback.
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
Bug Report
Description
The default title prompt doesn't seem to be respected by virtually any model I've tested it with. This is the current default title prompt on my installation of Open WebUI:
Here is an example of the kind of title that often gets generated:
I've had models as big as Mixtral 8x7B frequently struggle to return just the title with the default title prompt.
I would like to propose a better title prompt:
(EDIT: updated with the more recent proposed prompt found in the experimental results below)
Even the tiny Gemma-2B-Instruct is able to follow this prompt with a fairly high degree of consistency, to say nothing of the larger models, which struggle even less.
I find that LLMs follow instructions best when the instructions are placed at the end of the input, especially when the input may contain other instructions that are directed at an LLM. They may see an instruction at the end of the {{prompt}} that tells it to ignore previous instructions, and then it will completely "forget" that it is trying to generate a title.
Anyways, the title prompt I'm proposing above is just a simple improvement, in my opinion. To really make title generation work as well as possible, I am tempted to say that a grammar would be helpful here... but unfortunately,
ollamastill does not support custom grammars AFAICT. Requiring the model to respond with a response matching the format{"title":"<title with up to 4 spaces goes here>"}seems like it would help even the weakest of models to generate a title that meets the requirements.Beta Was this translation helpful? Give feedback.
All reactions