examples/quickstart.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# ADALA Quickstart\n",
    "\n",
    "In this notebook, we are going to run through some of the common tasks for creating data labeling agents with ADALA. In this example, we're going to create a data labeling agent for a text classification task - labeling our text samples as either \"Subjective or \"Objective\" statements. \n",
    "\n",
    "This agent will be LLM-based, so we will use [OpenAI's API](https://platform.openai.com/). You will to generate an API key and set it as an environment variable as follows: \n",
    "\n",
    "```\n",
    "export OPENAI_API_KEY=your_openai_api_key\n",
    "```\n",
    "\n",
    "Now, let's begin. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Dataset Creation\n",
    "First, let's use a dataset of product reviews stored in pandas dataframe. This will help us manage our data as we add more attributes, like predictions and labels for subjectivity and objectivity over time. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>text</th>\n",
       "      <th>ground_truth</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>The mic is great.</td>\n",
       "      <td>Subjective</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Will order from them again!</td>\n",
       "      <td>Subjective</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Not loud enough and doesn't turn on like it sh...</td>\n",
       "      <td>Objective</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>The phone doesn't seem to accept anything exce...</td>\n",
       "      <td>Objective</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>All three broke within two months of use.</td>\n",
       "      <td>Objective</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                text ground_truth\n",
       "0                                  The mic is great.   Subjective\n",
       "1                        Will order from them again!   Subjective\n",
       "2  Not loud enough and doesn't turn on like it sh...    Objective\n",
       "3  The phone doesn't seem to accept anything exce...    Objective\n",
       "4          All three broke within two months of use.    Objective"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "\n",
    "df = pd.DataFrame([\n",
    "    [\"The mic is great.\", \"Subjective\"],\n",
    "    [\"Will order from them again!\", \"Subjective\"],\n",
    "    [\"Not loud enough and doesn't turn on like it should.\", \"Objective\"],\n",
    "    [\"The phone doesn't seem to accept anything except CBR mp3s\", \"Objective\"],\n",
    "    [\"All three broke within two months of use.\", \"Objective\"]\n",
    "], columns=[\"text\", \"ground_truth\"])\n",
    "\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We instantiate Dataset that uses this pandas dataframe as a data source. Dataset object takes care of input data schema and data streaming:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create Agent\n",
    "\n",
    "To create Agent, we need to to define 2 things:\n",
    "\n",
    "**Skills** - Agent's abilities are defined as _Skills_. Each agent can possess many different skills. In our case, this agent only has one labeling skill, to produce a classification of Subjective or Objective for a given piece of text.  To define this skill, we will leverage an LLM, passing it instructions and the set of labeles we expect to receive back. \n",
    "\n",
    "**Environment** - that is where the Agent receives ground truth signal to improve its skill. Since we already created ground truth dataset, we can simply refer to the column from the dataframe. In the real world scenario, you may consider using a different environment where ground truth signal can be obtained asynchoronously by gathering real human feedback during agent's learning phase."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\">Agent Instance</span>\n",
       "\n",
       "Environment: StaticEnvironment\n",
       "Skills: subjectivity_detection\n",
       "Runtimes: openai\n",
       "Default Runtime: openai\n",
       "Default Teacher Runtime: openai-gpt4\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[1;34mAgent Instance\u001b[0m\n",
       "\n",
       "Environment: StaticEnvironment\n",
       "Skills: subjectivity_detection\n",
       "Runtimes: openai\n",
       "Default Runtime: openai\n",
       "Default Teacher Runtime: openai-gpt4\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from adala.agents import Agent\n",
    "from adala.environments import StaticEnvironment\n",
    "from adala.skills import ClassificationSkill\n",
    "from adala.runtimes import OpenAIChatRuntime, GuidanceRuntime\n",
    "from rich import print\n",
    "\n",
    "\n",
    "agent = Agent(\n",
    "    # define the agent's labeling skill that should classify text onto 2 categories\n",
    "    skills=ClassificationSkill(\n",
    "        name='subjectivity_detection',\n",
    "        description='Understanding subjective and objective statements from text.',\n",
    "        instructions='Classify a product review as either expressing \"Subjective\" or \"Objective\" statements.',\n",
    "        labels={'prediction': ['Subjective', 'Objective']},\n",
    "        input_template='Input: {text}',\n",
    "        output_template='Output: {prediction}'\n",
    "    ),\n",
    "    \n",
    "    # basic environment extracts ground truth signal from the input records\n",
    "    environment=StaticEnvironment(\n",
    "        df=df,\n",
    "        ground_truth_columns={'prediction': 'ground_truth'}\n",
    "    ),\n",
    "    \n",
    "    runtimes = {\n",
    "        'openai': GuidanceRuntime(),\n",
    "    },\n",
    "    default_runtime='openai',\n",
    "    \n",
    "    teacher_runtimes = {\n",
    "      'openai-gpt3': OpenAIChatRuntime(model='gpt-3.5-turbo'),\n",
    "      'openai-gpt4': OpenAIChatRuntime(model='gpt-4'),\n",
    "    },\n",
    "    \n",
    "    # NOTE! If you don't have an access to gpt4 - replace it with \"openai-gpt3\"\n",
    "    default_teacher_runtime='openai-gpt4'\n",
    ")\n",
    "\n",
    "print(agent)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Learning Agent\n",
    "\n",
    "We will now let Agent learn from the ground truth. After every action, Agent returns its _Experience_, where it stores various observations like predicted data, errors, accuracy, etc."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">\n",
       "\n",
       "=&gt; Iteration #<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0</span>: Getting feedback, analyzing and improving <span style=\"color: #808000; text-decoration-color: #808000\">...</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\n",
       "\n",
       "=> Iteration #\u001b[1;36m0\u001b[0m: Getting feedback, analyzing and improving \u001b[33m...\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">Applying skill: subjectivity_detection\n",
       "</pre>\n"
      ],
      "text/plain": [
       "Applying skill: subjectivity_detection\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|█| 5/5 [00:01<00:00, \n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">Predictions and feedback:\n",
       "</pre>\n"
      ],
      "text/plain": [
       "Predictions and feedback:\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">                                                                                                                   \n",
       " <span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\"> prediction__fb                           </span> <span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\"> text                                     </span> <span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\"> ground_truth </span> <span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\"> prediction </span> \n",
       " ───────────────────────────────────────────────────────────────────────────────────────────────────────────────── \n",
       "  Prediction is correct.                     The mic is great.                          Subjective     Subjective  \n",
       " <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\"> Prediction is correct.                   </span> <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\"> Will order from them again!              </span> <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\"> Subjective   </span> <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\"> Subjective </span> \n",
       "  Prediction is incorrect. Correct answer:   Not loud enough and doesn't turn on like   Objective      Subjective  \n",
       "  \"Objective\"                                it should.                                                            \n",
       " <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\"> Prediction is correct.                   </span> <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\"> The phone doesn't seem to accept         </span> <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\"> Objective    </span> <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\"> Objective  </span> \n",
       " <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                                          </span> <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\"> anything except CBR mp3s                 </span> <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">              </span> <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">            </span> \n",
       "  Prediction is correct.                     All three broke within two months of       Objective      Objective   \n",
       "                                             use.                                                                  \n",
       "                                                                                                                   \n",
       "</pre>\n"
      ],
      "text/plain": [
       "                                                                                                                   \n",
       " \u001b[1;35m \u001b[0m\u001b[1;35mprediction__fb                          \u001b[0m\u001b[1;35m \u001b[0m \u001b[1;35m \u001b[0m\u001b[1;35mtext                                    \u001b[0m\u001b[1;35m \u001b[0m \u001b[1;35m \u001b[0m\u001b[1;35mground_truth\u001b[0m\u001b[1;35m \u001b[0m \u001b[1;35m \u001b[0m\u001b[1;35mprediction\u001b[0m\u001b[1;35m \u001b[0m \n",
       " ───────────────────────────────────────────────────────────────────────────────────────────────────────────────── \n",
       "  Prediction is correct.                     The mic is great.                          Subjective     Subjective  \n",
       " \u001b[2m \u001b[0m\u001b[2mPrediction is correct.                  \u001b[0m\u001b[2m \u001b[0m \u001b[2m \u001b[0m\u001b[2mWill order from them again!             \u001b[0m\u001b[2m \u001b[0m \u001b[2m \u001b[0m\u001b[2mSubjective  \u001b[0m\u001b[2m \u001b[0m \u001b[2m \u001b[0m\u001b[2mSubjective\u001b[0m\u001b[2m \u001b[0m \n",
       "  Prediction is incorrect. Correct answer:   Not loud enough and doesn't turn on like   Objective      Subjective  \n",
       "  \"Objective\"                                it should.                                                            \n",
       " \u001b[2m \u001b[0m\u001b[2mPrediction is correct.                  \u001b[0m\u001b[2m \u001b[0m \u001b[2m \u001b[0m\u001b[2mThe phone doesn't seem to accept        \u001b[0m\u001b[2m \u001b[0m \u001b[2m \u001b[0m\u001b[2mObjective   \u001b[0m\u001b[2m \u001b[0m \u001b[2m \u001b[0m\u001b[2mObjective \u001b[0m\u001b[2m \u001b[0m \n",
       " \u001b[2m                                          \u001b[0m \u001b[2m \u001b[0m\u001b[2manything except CBR mp3s                \u001b[0m\u001b[2m \u001b[0m \u001b[2m              \u001b[0m \u001b[2m            \u001b[0m \n",
       "  Prediction is correct.                     All three broke within two months of       Objective      Objective   \n",
       "                                             use.                                                                  \n",
       "                                                                                                                   \n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #800000; text-decoration-color: #800000; font-weight: bold\">Output to improve: </span><span style=\"color: #008000; text-decoration-color: #008000\">\"prediction\"</span><span style=\"color: #800000; text-decoration-color: #800000; font-weight: bold\"> (</span><span style=\"color: #808000; text-decoration-color: #808000; font-weight: bold\">Skill</span><span style=\"color: #800000; text-decoration-color: #800000; font-weight: bold\">=</span><span style=\"color: #008000; text-decoration-color: #008000\">\"subjectivity_detection\"</span><span style=\"color: #800000; text-decoration-color: #800000; font-weight: bold\">)</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000; font-weight: bold\">Accuracy = </span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">80.00</span><span style=\"color: #800000; text-decoration-color: #800000; font-weight: bold\">%</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[1;31mOutput to improve: \u001b[0m\u001b[32m\"prediction\"\u001b[0m\u001b[1;31m \u001b[0m\u001b[1;31m(\u001b[0m\u001b[1;33mSkill\u001b[0m\u001b[1;31m=\u001b[0m\u001b[32m\"subjectivity_detection\"\u001b[0m\u001b[1;31m)\u001b[0m\n",
       "\u001b[1;31mAccuracy = \u001b[0m\u001b[1;36m80.00\u001b[0m\u001b[1;31m%\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #008000; text-decoration-color: #008000\">\"{system}\"</span>:\n",
       "You are a helpful assistant.\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[32m\"\u001b[0m\u001b[32m{\u001b[0m\u001b[32msystem\u001b[0m\u001b[32m}\u001b[0m\u001b[32m\"\u001b[0m:\n",
       "You are a helpful assistant.\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #008000; text-decoration-color: #008000\">\"{user}\"</span>:\n",
       "\n",
       "A prompt is a text paragraph that outlines the expected actions and instructs the large language model <span style=\"font-weight: bold\">(</span>LLM<span style=\"font-weight: bold\">)</span> to \n",
       "generate a specific output. This prompt is concatenated with the input text, and the model then creates the \n",
       "required output.\n",
       "This describes the full template how the prompt is concatenated with the input to produce the output:\n",
       "\n",
       "```\n",
       "\n",
       "<span style=\"font-weight: bold\">{</span>prompt<span style=\"font-weight: bold\">}</span>\n",
       "Input: <span style=\"font-weight: bold\">{</span>text<span style=\"font-weight: bold\">}</span>\n",
       "Output: <span style=\"font-weight: bold\">{</span>prediction<span style=\"font-weight: bold\">}</span>\n",
       "```\n",
       "\n",
       "Here:\n",
       "- <span style=\"color: #008000; text-decoration-color: #008000\">\"Input: {text}\"</span> is input template,\n",
       "- <span style=\"color: #008000; text-decoration-color: #008000\">\"{prompt}\"</span> is the LLM prompt,\n",
       "- <span style=\"color: #008000; text-decoration-color: #008000\">\"Output: {prediction}\"</span> is the output template.\n",
       "\n",
       "Model can produce erroneous output if a prompt is not well defined. In our collaboration, we’ll work together to \n",
       "refine a prompt. The process consists of two main steps:\n",
       "\n",
       "## Step <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1</span>\n",
       "I will provide you with the current prompt along with prediction examples. Each example contains the input text, \n",
       "the final prediction produced by the model, and the user feedback. User feedback indicates whether the model \n",
       "prediction is correct or not. Your task is to analyze the examples and user feedback, determining whether the \n",
       "existing instruction is describing the task reflected by these examples precisely, and suggests changes to the \n",
       "prompt to address the incorrect predictions.\n",
       "\n",
       "## Step <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2</span>\n",
       "Next, you will carefully review your reasoning in step <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1</span>, integrate the insights to refine the prompt, and provide \n",
       "me with the new prompt that improves the model’s performance.\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[32m\"\u001b[0m\u001b[32m{\u001b[0m\u001b[32muser\u001b[0m\u001b[32m}\u001b[0m\u001b[32m\"\u001b[0m:\n",
       "\n",
       "A prompt is a text paragraph that outlines the expected actions and instructs the large language model \u001b[1m(\u001b[0mLLM\u001b[1m)\u001b[0m to \n",
       "generate a specific output. This prompt is concatenated with the input text, and the model then creates the \n",
       "required output.\n",
       "This describes the full template how the prompt is concatenated with the input to produce the output:\n",
       "\n",
       "```\n",
       "\n",
       "\u001b[1m{\u001b[0mprompt\u001b[1m}\u001b[0m\n",
       "Input: \u001b[1m{\u001b[0mtext\u001b[1m}\u001b[0m\n",
       "Output: \u001b[1m{\u001b[0mprediction\u001b[1m}\u001b[0m\n",
       "```\n",
       "\n",
       "Here:\n",
       "- \u001b[32m\"Input: \u001b[0m\u001b[32m{\u001b[0m\u001b[32mtext\u001b[0m\u001b[32m}\u001b[0m\u001b[32m\"\u001b[0m is input template,\n",
       "- \u001b[32m\"\u001b[0m\u001b[32m{\u001b[0m\u001b[32mprompt\u001b[0m\u001b[32m}\u001b[0m\u001b[32m\"\u001b[0m is the LLM prompt,\n",
       "- \u001b[32m\"Output: \u001b[0m\u001b[32m{\u001b[0m\u001b[32mprediction\u001b[0m\u001b[32m}\u001b[0m\u001b[32m\"\u001b[0m is the output template.\n",
       "\n",
       "Model can produce erroneous output if a prompt is not well defined. In our collaboration, we’ll work together to \n",
       "refine a prompt. The process consists of two main steps:\n",
       "\n",
       "## Step \u001b[1;36m1\u001b[0m\n",
       "I will provide you with the current prompt along with prediction examples. Each example contains the input text, \n",
       "the final prediction produced by the model, and the user feedback. User feedback indicates whether the model \n",
       "prediction is correct or not. Your task is to analyze the examples and user feedback, determining whether the \n",
       "existing instruction is describing the task reflected by these examples precisely, and suggests changes to the \n",
       "prompt to address the incorrect predictions.\n",
       "\n",
       "## Step \u001b[1;36m2\u001b[0m\n",
       "Next, you will carefully review your reasoning in step \u001b[1;36m1\u001b[0m, integrate the insights to refine the prompt, and provide \n",
       "me with the new prompt that improves the model’s performance.\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #008000; text-decoration-color: #008000\">\"{assistant}\"</span>:\n",
       "Sure, I’d be happy to help you with this prompt engineering problem. Please provide me with the current prompt and \n",
       "the examples with user feedback.\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[32m\"\u001b[0m\u001b[32m{\u001b[0m\u001b[32massistant\u001b[0m\u001b[32m}\u001b[0m\u001b[32m\"\u001b[0m:\n",
       "Sure, I’d be happy to help you with this prompt engineering problem. Please provide me with the current prompt and \n",
       "the examples with user feedback.\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #008000; text-decoration-color: #008000\">\"{user}\"</span>:\n",
       "\n",
       "## Current prompt\n",
       "Classify a product review as either expressing <span style=\"color: #008000; text-decoration-color: #008000\">\"Subjective\"</span> or <span style=\"color: #008000; text-decoration-color: #008000\">\"Objective\"</span> statements.\n",
       "\n",
       "## Examples\n",
       "### Example #<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0</span>\n",
       "\n",
       "Input: The mic is great.\n",
       "\n",
       "Output: Subjective\n",
       "\n",
       "User feedback: Prediction is correct.\n",
       "\n",
       "\n",
       "### Example #<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1</span>\n",
       "\n",
       "Input: Will order from them again!\n",
       "\n",
       "Output: Subjective\n",
       "\n",
       "User feedback: Prediction is correct.\n",
       "\n",
       "\n",
       "### Example #<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2</span>\n",
       "\n",
       "Input: Not loud enough and doesn't turn on like it should.\n",
       "\n",
       "Output: Subjective\n",
       "\n",
       "User feedback: Prediction is incorrect. Correct answer: <span style=\"color: #008000; text-decoration-color: #008000\">\"Objective\"</span>\n",
       "\n",
       "\n",
       "### Example #<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">3</span>\n",
       "\n",
       "Input: The phone doesn't seem to accept anything except CBR mp3s\n",
       "\n",
       "Output: Objective\n",
       "\n",
       "User feedback: Prediction is correct.\n",
       "\n",
       "\n",
       "### Example #<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">4</span>\n",
       "\n",
       "Input: All three broke within two months of use.\n",
       "\n",
       "Output: Objective\n",
       "\n",
       "User feedback: Prediction is correct.\n",
       "\n",
       "\n",
       "\n",
       "Summarize your analysis about incorrect predictions and suggest changes to the prompt.\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[32m\"\u001b[0m\u001b[32m{\u001b[0m\u001b[32muser\u001b[0m\u001b[32m}\u001b[0m\u001b[32m\"\u001b[0m:\n",
       "\n",
       "## Current prompt\n",
       "Classify a product review as either expressing \u001b[32m\"Subjective\"\u001b[0m or \u001b[32m\"Objective\"\u001b[0m statements.\n",
       "\n",
       "## Examples\n",
       "### Example #\u001b[1;36m0\u001b[0m\n",
       "\n",
       "Input: The mic is great.\n",
       "\n",
       "Output: Subjective\n",
       "\n",
       "User feedback: Prediction is correct.\n",
       "\n",
       "\n",
       "### Example #\u001b[1;36m1\u001b[0m\n",
       "\n",
       "Input: Will order from them again!\n",
       "\n",
       "Output: Subjective\n",
       "\n",
       "User feedback: Prediction is correct.\n",
       "\n",
       "\n",
       "### Example #\u001b[1;36m2\u001b[0m\n",
       "\n",
       "Input: Not loud enough and doesn't turn on like it should.\n",
       "\n",
       "Output: Subjective\n",
       "\n",
       "User feedback: Prediction is incorrect. Correct answer: \u001b[32m\"Objective\"\u001b[0m\n",
       "\n",
       "\n",
       "### Example #\u001b[1;36m3\u001b[0m\n",
       "\n",
       "Input: The phone doesn't seem to accept anything except CBR mp3s\n",
       "\n",
       "Output: Objective\n",
       "\n",
       "User feedback: Prediction is correct.\n",
       "\n",
       "\n",
       "### Example #\u001b[1;36m4\u001b[0m\n",
       "\n",
       "Input: All three broke within two months of use.\n",
       "\n",
       "Output: Objective\n",
       "\n",
       "User feedback: Prediction is correct.\n",
       "\n",
       "\n",
       "\n",
       "Summarize your analysis about incorrect predictions and suggest changes to the prompt.\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #008000; text-decoration-color: #008000\">\"{assistant}\"</span>:\n",
       "### Analysis \n",
       "\n",
       "From the provided examples, it's evident that the model sometimes struggles to distinguish between objective and \n",
       "subjective statements. Let's analyze the incorrect prediction:\n",
       "\n",
       "- In Example #<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2</span>, the input statement <span style=\"color: #008000; text-decoration-color: #008000\">\"Not loud enough and doesn't turn on like it should.\"</span> was labeled as \n",
       "subjective by the model. However, according to the user feedback, the correct classification should've been \n",
       "<span style=\"color: #008000; text-decoration-color: #008000\">\"objective\"</span>. The user probably categorized it as objective because it appears to present factual information about \n",
       "the product's performance.\n",
       "\n",
       "It seems that while defining subjectivity and objectivity within the context of product reviews, there might be \n",
       "overlaps, leading to potential ambiguity. Subjectivity usually involves personal opinions, feelings, or tastes, \n",
       "while objectivity mostly describes measurable facts about the product itself. But in product reviews, these facts \n",
       "often come off as personal experiences.\n",
       "\n",
       "### Suggestions \n",
       "\n",
       "<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1</span><span style=\"font-weight: bold\">)</span> We could add clearer instructions to the language model about the definition of subjectivity and objectivity. \n",
       "\n",
       "<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2</span><span style=\"font-weight: bold\">)</span> Given that both subjectivity and objectivity can stem from personal experiences in product reviews, the prompt \n",
       "could be more explicit regarding the fact that <span style=\"color: #008000; text-decoration-color: #008000\">'objective'</span> statements could include direct experiences related to \n",
       "the functionality or quality of the product, while <span style=\"color: #008000; text-decoration-color: #008000\">'subjective'</span> ones could be about personal feelings, emotions, or\n",
       "preferences.\n",
       "\n",
       "<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">3</span><span style=\"font-weight: bold\">)</span> Lastly, we can also provide examples to the language model for better understanding.\n",
       "\n",
       "### Revised Prompt\n",
       "\n",
       "<span style=\"color: #008000; text-decoration-color: #008000\">\"Based on the following product review, classify it as 'Subjective' or 'Objective'. If the review primarily </span>\n",
       "<span style=\"color: #008000; text-decoration-color: #008000\">expresses personal feelings, emotions or preferences, it should be classified as 'Subjective'. On the other hand, </span>\n",
       "<span style=\"color: #008000; text-decoration-color: #008000\">if it offers direct experiences related to the functionality or quality of the product without expressing personal </span>\n",
       "<span style=\"color: #008000; text-decoration-color: #008000\">emotions or preferences, classify it as 'Objective'. Remember, even factual information coming from personal </span>\n",
       "<span style=\"color: #008000; text-decoration-color: #008000\">experiences can be considered 'Objective' if it is relating to the product's performance or usability.\"</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[32m\"\u001b[0m\u001b[32m{\u001b[0m\u001b[32massistant\u001b[0m\u001b[32m}\u001b[0m\u001b[32m\"\u001b[0m:\n",
       "### Analysis \n",
       "\n",
       "From the provided examples, it's evident that the model sometimes struggles to distinguish between objective and \n",
       "subjective statements. Let's analyze the incorrect prediction:\n",
       "\n",
       "- In Example #\u001b[1;36m2\u001b[0m, the input statement \u001b[32m\"Not loud enough and doesn't turn on like it should.\"\u001b[0m was labeled as \n",
       "subjective by the model. However, according to the user feedback, the correct classification should've been \n",
       "\u001b[32m\"objective\"\u001b[0m. The user probably categorized it as objective because it appears to present factual information about \n",
       "the product's performance.\n",
       "\n",
       "It seems that while defining subjectivity and objectivity within the context of product reviews, there might be \n",
       "overlaps, leading to potential ambiguity. Subjectivity usually involves personal opinions, feelings, or tastes, \n",
       "while objectivity mostly describes measurable facts about the product itself. But in product reviews, these facts \n",
       "often come off as personal experiences.\n",
       "\n",
       "### Suggestions \n",
       "\n",
       "\u001b[1;36m1\u001b[0m\u001b[1m)\u001b[0m We could add clearer instructions to the language model about the definition of subjectivity and objectivity. \n",
       "\n",
       "\u001b[1;36m2\u001b[0m\u001b[1m)\u001b[0m Given that both subjectivity and objectivity can stem from personal experiences in product reviews, the prompt \n",
       "could be more explicit regarding the fact that \u001b[32m'objective'\u001b[0m statements could include direct experiences related to \n",
       "the functionality or quality of the product, while \u001b[32m'subjective'\u001b[0m ones could be about personal feelings, emotions, or\n",
       "preferences.\n",
       "\n",
       "\u001b[1;36m3\u001b[0m\u001b[1m)\u001b[0m Lastly, we can also provide examples to the language model for better understanding.\n",
       "\n",
       "### Revised Prompt\n",
       "\n",
       "\u001b[32m\"Based on the following product review, classify it as 'Subjective' or 'Objective'. If the review primarily \u001b[0m\n",
       "\u001b[32mexpresses personal feelings, emotions or preferences, it should be classified as 'Subjective'. On the other hand, \u001b[0m\n",
       "\u001b[32mif it offers direct experiences related to the functionality or quality of the product without expressing personal \u001b[0m\n",
       "\u001b[32memotions or preferences, classify it as 'Objective'. Remember, even factual information coming from personal \u001b[0m\n",
       "\u001b[32mexperiences can be considered 'Objective' if it is relating to the product's performance or usability.\"\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #008000; text-decoration-color: #008000\">\"{user}\"</span>:\n",
       "\n",
       "Now please carefully review your reasoning in Step <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1</span> and help with Step <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2</span>: refining the prompt.\n",
       "\n",
       "## Current prompt\n",
       "Classify a product review as either expressing <span style=\"color: #008000; text-decoration-color: #008000\">\"Subjective\"</span> or <span style=\"color: #008000; text-decoration-color: #008000\">\"Objective\"</span> statements.\n",
       "\n",
       "## Follow this guidance to refine the prompt:\n",
       "\n",
       "<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1</span>. The new prompt should should describe the task precisely, and address the points raised in the user feedback.\n",
       "    \n",
       "<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2</span>. The new prompt should be similar to the current instruction, and only differ in the parts that address the \n",
       "issues you identified in Step <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1</span>.\n",
       "    Example:\n",
       "    - Current prompt: <span style=\"color: #008000; text-decoration-color: #008000\">\"The model should generate a summary of the input text.\"</span>\n",
       "    - New prompt: <span style=\"color: #008000; text-decoration-color: #008000\">\"The model should generate a summary of the input text. Pay attention to the original style.\"</span>\n",
       "        \n",
       "<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">3</span>. Reply only with the new prompt. Do not include input and output templates in the prompt.\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[32m\"\u001b[0m\u001b[32m{\u001b[0m\u001b[32muser\u001b[0m\u001b[32m}\u001b[0m\u001b[32m\"\u001b[0m:\n",
       "\n",
       "Now please carefully review your reasoning in Step \u001b[1;36m1\u001b[0m and help with Step \u001b[1;36m2\u001b[0m: refining the prompt.\n",
       "\n",
       "## Current prompt\n",
       "Classify a product review as either expressing \u001b[32m\"Subjective\"\u001b[0m or \u001b[32m\"Objective\"\u001b[0m statements.\n",
       "\n",
       "## Follow this guidance to refine the prompt:\n",
       "\n",
       "\u001b[1;36m1\u001b[0m. The new prompt should should describe the task precisely, and address the points raised in the user feedback.\n",
       "    \n",
       "\u001b[1;36m2\u001b[0m. The new prompt should be similar to the current instruction, and only differ in the parts that address the \n",
       "issues you identified in Step \u001b[1;36m1\u001b[0m.\n",
       "    Example:\n",
       "    - Current prompt: \u001b[32m\"The model should generate a summary of the input text.\"\u001b[0m\n",
       "    - New prompt: \u001b[32m\"The model should generate a summary of the input text. Pay attention to the original style.\"\u001b[0m\n",
       "        \n",
       "\u001b[1;36m3\u001b[0m. Reply only with the new prompt. Do not include input and output templates in the prompt.\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<span style=\"background-color: lightgreen;\">\"</span><span style=\"background-color: lightgreen;\">A</span><span style=\"background-color: lightgreen;\">n</span>a<span style=\"background-color: lightgreen;\">l</span>y<span style=\"background-color: lightgreen;\">z</span><span style=\"background-color: lightgreen;\">e</span> <span style=\"background-color: lightgreen;\">t</span><span style=\"background-color: lightgreen;\">h</span><span style=\"background-color: lightgreen;\">e</span><span style=\"background-color: lightgreen;\"> </span><span style=\"background-color: lightgreen;\">p</span><span style=\"background-color: lightgreen;\">r</span><span style=\"background-color: lightgreen;\">o</span><span style=\"background-color: lightgreen;\">v</span><span style=\"background-color: lightgreen;\">i</span><span style=\"background-color: lightgreen;\">d</span><span style=\"background-color: lightgreen;\">e</span><span style=\"background-color: lightgreen;\">d</span> product review a<span style=\"background-color: lightgreen;\">n</span><span style=\"background-color: lightgreen;\">d</span> <span style=\"background-color: lightgreen;\">d</span>eter<span style=\"background-color: lightgreen;\">m</span><span style=\"background-color: lightgreen;\">i</span><span style=\"background-color: lightgreen;\">n</span><span style=\"background-color: lightgreen;\">e</span><span style=\"background-color: lightgreen;\"> </span><span style=\"background-color: lightgreen;\">w</span><span style=\"background-color: lightgreen;\">h</span><span style=\"background-color: lightgreen;\">e</span><span style=\"background-color: lightgreen;\">t</span><span style=\"background-color: lightgreen;\">h</span><span style=\"background-color: lightgreen;\">e</span><span style=\"background-color: lightgreen;\">r</span><span style=\"background-color: lightgreen;\"> </span><span style=\"background-color: lightgreen;\">i</span><span style=\"background-color: lightgreen;\">t</span><span style=\"background-color: lightgreen;\"> </span><span style=\"background-color: lightgreen;\">p</span><span style=\"background-color: lightgreen;\">r</span><span style=\"background-color: lightgreen;\">i</span><span style=\"background-color: lightgreen;\">m</span><span style=\"background-color: lightgreen;\">a</span><span style=\"background-color: lightgreen;\">r</span><span style=\"background-color: lightgreen;\">i</span><span style=\"background-color: lightgreen;\">l</span><span style=\"background-color: lightgreen;\">y</span> express<span style=\"background-color: lightgreen;\">e</span><span style=\"background-color: lightgreen;\">s</span> <span style=\"background-color: lightgreen;\">'</span>Subjective<span style=\"background-color: lightgreen;\">'</span> or <span style=\"background-color: lightgreen;\">'</span>Objective<span style=\"background-color: lightgreen;\">'</span><span style=\"background-color: lightgreen;\"> </span><span style=\"background-color: lightgreen;\">s</span><span style=\"background-color: lightgreen;\">t</span><span style=\"background-color: lightgreen;\">a</span><span style=\"background-color: lightgreen;\">t</span><span style=\"background-color: lightgreen;\">e</span><span style=\"background-color: lightgreen;\">m</span><span style=\"background-color: lightgreen;\">e</span><span style=\"background-color: lightgreen;\">n</span><span style=\"background-color: lightgreen;\">t</span><span style=\"background-color: lightgreen;\">s</span><span style=\"background-color: lightgreen;\">.</span><span style=\"background-color: lightgreen;\"> </span><span style=\"background-color: lightgreen;\">C</span><span style=\"background-color: lightgreen;\">l</span><span style=\"background-color: lightgreen;\">a</span><span style=\"background-color: lightgreen;\">s</span><span style=\"background-color: lightgreen;\">s</span><span style=\"background-color: lightgreen;\">i</span><span style=\"background-color: lightgreen;\">f</span><span style=\"background-color: lightgreen;\">y</span><span style=\"background-color: lightgreen;\"> </span><span style=\"background-color: lightgreen;\">i</span><span style=\"background-color: lightgreen;\">t</span><span style=\"background-color: lightgreen;\"> </span><span style=\"background-color: lightgreen;\">a</span><span style=\"background-color: lightgreen;\">s</span><span style=\"background-color: lightgreen;\"> </span><span style=\"background-color: lightgreen;\">'</span><span style=\"background-color: lightgreen;\">S</span><span style=\"background-color: lightgreen;\">u</span><span style=\"background-color: lightgreen;\">b</span><span style=\"background-color: lightgreen;\">j</span><span style=\"background-color: lightgreen;\">e</span><span style=\"background-color: lightgreen;\">c</span><span style=\"background-color: lightgreen;\">t</span><span style=\"background-color: lightgreen;\">i</span><span style=\"background-color: lightgreen;\">v</span><span style=\"background-color: lightgreen;\">e</span><span style=\"background-color: lightgreen;\">'</span><span style=\"background-color: lightgreen;\"> </span><span style=\"background-color: lightgreen;\">i</span><span style=\"background-color: lightgreen;\">f</span><span style=\"background-color: lightgreen;\"> </span><span style=\"background-color: lightgreen;\">t</span><span style=\"background-color: lightgreen;\">h</span><span style=\"background-color: lightgreen;\">e</span><span style=\"background-color: lightgreen;\"> </span><span style=\"background-color: lightgreen;\">r</span><span style=\"background-color: lightgreen;\">e</span><span style=\"background-color: lightgreen;\">v</span><span style=\"background-color: lightgreen;\">i</span><span style=\"background-color: lightgreen;\">e</span><span style=\"background-color: lightgreen;\">w</span><span style=\"background-color: lightgreen;\"> </span><span style=\"background-color: lightgreen;\">m</span><span style=\"background-color: lightgreen;\">a</span><span style=\"background-color: lightgreen;\">i</span><span style=\"background-color: lightgreen;\">n</span><span style=\"background-color: lightgreen;\">l</span><span style=\"background-color: lightgreen;\">y</span><span style=\"background-color: lightgreen;\"> </span><span style=\"background-color: lightgreen;\">e</span><span style=\"background-color: lightgreen;\">x</span><span style=\"background-color: lightgreen;\">p</span><span style=\"background-color: lightgreen;\">r</span><span style=\"background-color: lightgreen;\">e</span><span style=\"background-color: lightgreen;\">s</span><span style=\"background-color: lightgreen;\">s</span><span style=\"background-color: lightgreen;\">e</span><span style=\"background-color: lightgreen;\">s</span><span style=\"background-color: lightgreen;\"> </span><span style=\"background-color: lightgreen;\">p</span><span style=\"background-color: lightgreen;\">e</span><span style=\"background-color: lightgreen;\">r</span><span style=\"background-color: lightgreen;\">s</span><span style=\"background-color: lightgreen;\">o</span><span style=\"background-color: lightgreen;\">n</span><span style=\"background-color: lightgreen;\">a</span><span style=\"background-color: lightgreen;\">l</span><span style=\"background-color: lightgreen;\"> </span><span style=\"background-color: lightgreen;\">f</span><span style=\"background-color: lightgreen;\">e</span><span style=\"background-color: lightgreen;\">e</span><span style=\"background-color: lightgreen;\">l</span><span style=\"background-color: lightgreen;\">i</span><span style=\"background-color: lightgreen;\">n</span><span style=\"background-color: lightgreen;\">g</span><span style=\"background-color: lightgreen;\">s</span><span style=\"background-color: lightgreen;\">,</span><span style=\"background-color: lightgreen;\"> </span><span style=\"background-color: lightgreen;\">e</span><span style=\"background-color: lightgreen;\">m</span><span style=\"background-color: lightgreen;\">o</span><span style=\"background-color: lightgreen;\">t</span><span style=\"background-color: lightgreen;\">i</span><span style=\"background-color: lightgreen;\">o</span><span style=\"background-color: lightgreen;\">n</span><span style=\"background-color: lightgreen;\">s</span><span style=\"background-color: lightgreen;\">,</span><span style=\"background-color: lightgreen;\"> </span><span style=\"background-color: lightgreen;\">o</span><span style=\"background-color: lightgreen;\">r</span><span style=\"background-color: lightgreen;\"> </span><span style=\"background-color: lightgreen;\">p</span><span style=\"background-color: lightgreen;\">r</span><span style=\"background-color: lightgreen;\">e</span><span style=\"background-color: lightgreen;\">f</span><span style=\"background-color: lightgreen;\">e</span><span style=\"background-color: lightgreen;\">r</span><span style=\"background-color: lightgreen;\">e</span><span style=\"background-color: lightgreen;\">n</span><span style=\"background-color: lightgreen;\">c</span><span style=\"background-color: lightgreen;\">e</span><span style=\"background-color: lightgreen;\">s</span><span style=\"background-color: lightgreen;\">.</span><span style=\"background-color: lightgreen;\"> </span><span style=\"background-color: lightgreen;\">I</span><span style=\"background-color: lightgreen;\">f</span><span style=\"background-color: lightgreen;\"> </span><span style=\"background-color: lightgreen;\">t</span><span style=\"background-color: lightgreen;\">h</span><span style=\"background-color: lightgreen;\">e</span><span style=\"background-color: lightgreen;\"> </span><span style=\"background-color: lightgreen;\">r</span><span style=\"background-color: lightgreen;\">e</span><span style=\"background-color: lightgreen;\">v</span><span style=\"background-color: lightgreen;\">i</span><span style=\"background-color: lightgreen;\">e</span><span style=\"background-color: lightgreen;\">w</span><span style=\"background-color: lightgreen;\"> </span><span style=\"background-color: lightgreen;\">p</span><span style=\"background-color: lightgreen;\">r</span><span style=\"background-color: lightgreen;\">o</span><span style=\"background-color: lightgreen;\">v</span><span style=\"background-color: lightgreen;\">i</span><span style=\"background-color: lightgreen;\">d</span><span style=\"background-color: lightgreen;\">e</span><span style=\"background-color: lightgreen;\">s</span><span style=\"background-color: lightgreen;\"> </span><span style=\"background-color: lightgreen;\">f</span><span style=\"background-color: lightgreen;\">a</span><span style=\"background-color: lightgreen;\">c</span><span style=\"background-color: lightgreen;\">t</span><span style=\"background-color: lightgreen;\">u</span><span style=\"background-color: lightgreen;\">a</span><span style=\"background-color: lightgreen;\">l</span><span style=\"background-color: lightgreen;\"> </span><span style=\"background-color: lightgreen;\">d</span><span style=\"background-color: lightgreen;\">e</span><span style=\"background-color: lightgreen;\">t</span><span style=\"background-color: lightgreen;\">a</span><span style=\"background-color: lightgreen;\">i</span><span style=\"background-color: lightgreen;\">l</span><span style=\"background-color: lightgreen;\">s</span><span style=\"background-color: lightgreen;\"> </span><span style=\"background-color: lightgreen;\">r</span><span style=\"background-color: lightgreen;\">e</span><span style=\"background-color: lightgreen;\">l</span><span style=\"background-color: lightgreen;\">a</span><span style=\"background-color: lightgreen;\">t</span><span style=\"background-color: lightgreen;\">e</span><span style=\"background-color: lightgreen;\">d</span><span style=\"background-color: lightgreen;\"> </span><span style=\"background-color: lightgreen;\">d</span><span style=\"background-color: lightgreen;\">i</span><span style=\"background-color: lightgreen;\">r</span><span style=\"background-color: lightgreen;\">e</span><span style=\"background-color: lightgreen;\">c</span><span style=\"background-color: lightgreen;\">t</span><span style=\"background-color: lightgreen;\">l</span><span style=\"background-color: lightgreen;\">y</span><span style=\"background-color: lightgreen;\"> </span><span style=\"background-color: lightgreen;\">t</span><span style=\"background-color: lightgreen;\">o</span><span style=\"background-color: lightgreen;\"> </span><span style=\"background-color: lightgreen;\">t</span><span style=\"background-color: lightgreen;\">h</span><span style=\"background-color: lightgreen;\">e</span><span style=\"background-color: lightgreen;\"> </span><span style=\"background-color: lightgreen;\">f</span><span style=\"background-color: lightgreen;\">u</span><span style=\"background-color: lightgreen;\">n</span><span style=\"background-color: lightgreen;\">c</span><span style=\"background-color: lightgreen;\">t</span><span style=\"background-color: lightgreen;\">i</span><span style=\"background-color: lightgreen;\">o</span><span style=\"background-color: lightgreen;\">n</span><span style=\"background-color: lightgreen;\">a</span><span style=\"background-color: lightgreen;\">l</span><span style=\"background-color: lightgreen;\">i</span><span style=\"background-color: lightgreen;\">t</span><span style=\"background-color: lightgreen;\">y</span><span style=\"background-color: lightgreen;\"> </span><span style=\"background-color: lightgreen;\">o</span><span style=\"background-color: lightgreen;\">r</span><span style=\"background-color: lightgreen;\"> </span><span style=\"background-color: lightgreen;\">q</span><span style=\"background-color: lightgreen;\">u</span><span style=\"background-color: lightgreen;\">a</span><span style=\"background-color: lightgreen;\">l</span><span style=\"background-color: lightgreen;\">i</span><span style=\"background-color: lightgreen;\">t</span><span style=\"background-color: lightgreen;\">y</span><span style=\"background-color: lightgreen;\"> </span><span style=\"background-color: lightgreen;\">o</span><span style=\"background-color: lightgreen;\">f</span><span style=\"background-color: lightgreen;\"> </span><span style=\"background-color: lightgreen;\">t</span><span style=\"background-color: lightgreen;\">h</span><span style=\"background-color: lightgreen;\">e</span><span style=\"background-color: lightgreen;\"> </span><span style=\"background-color: lightgreen;\">p</span><span style=\"background-color: lightgreen;\">r</span><span style=\"background-color: lightgreen;\">o</span><span style=\"background-color: lightgreen;\">d</span><span style=\"background-color: lightgreen;\">u</span><span style=\"background-color: lightgreen;\">c</span><span style=\"background-color: lightgreen;\">t</span><span style=\"background-color: lightgreen;\">,</span><span style=\"background-color: lightgreen;\"> </span><span style=\"background-color: lightgreen;\">e</span><span style=\"background-color: lightgreen;\">v</span><span style=\"background-color: lightgreen;\">e</span><span style=\"background-color: lightgreen;\">n</span><span style=\"background-color: lightgreen;\"> </span><span style=\"background-color: lightgreen;\">b</span><span style=\"background-color: lightgreen;\">a</span><span style=\"background-color: lightgreen;\">s</span><span style=\"background-color: lightgreen;\">e</span><span style=\"background-color: lightgreen;\">d</span><span style=\"background-color: lightgreen;\"> </span><span style=\"background-color: lightgreen;\">o</span><span style=\"background-color: lightgreen;\">n</span><span style=\"background-color: lightgreen;\"> </span><span style=\"background-color: lightgreen;\">p</span><span style=\"background-color: lightgreen;\">e</span><span style=\"background-color: lightgreen;\">r</span><span style=\"background-color: lightgreen;\">s</span><span style=\"background-color: lightgreen;\">o</span><span style=\"background-color: lightgreen;\">n</span><span style=\"background-color: lightgreen;\">a</span><span style=\"background-color: lightgreen;\">l</span><span style=\"background-color: lightgreen;\"> </span><span style=\"background-color: lightgreen;\">e</span><span style=\"background-color: lightgreen;\">x</span><span style=\"background-color: lightgreen;\">p</span><span style=\"background-color: lightgreen;\">e</span><span style=\"background-color: lightgreen;\">r</span><span style=\"background-color: lightgreen;\">i</span><span style=\"background-color: lightgreen;\">e</span><span style=\"background-color: lightgreen;\">n</span><span style=\"background-color: lightgreen;\">c</span><span style=\"background-color: lightgreen;\">e</span><span style=\"background-color: lightgreen;\">s</span><span style=\"background-color: lightgreen;\">,</span><span style=\"background-color: lightgreen;\"> </span><span style=\"background-color: lightgreen;\">w</span><span style=\"background-color: lightgreen;\">i</span><span style=\"background-color: lightgreen;\">t</span><span style=\"background-color: lightgreen;\">h</span><span style=\"background-color: lightgreen;\">o</span><span style=\"background-color: lightgreen;\">u</span><span style=\"background-color: lightgreen;\">t</span><span style=\"background-color: lightgreen;\"> </span><span style=\"background-color: lightgreen;\">e</span><span style=\"background-color: lightgreen;\">x</span><span style=\"background-color: lightgreen;\">p</span><span style=\"background-color: lightgreen;\">r</span><span style=\"background-color: lightgreen;\">e</span><span style=\"background-color: lightgreen;\">s</span><span style=\"background-color: lightgreen;\">s</span><span style=\"background-color: lightgreen;\">i</span><span style=\"background-color: lightgreen;\">n</span><span style=\"background-color: lightgreen;\">g</span><span style=\"background-color: lightgreen;\"> </span><span style=\"background-color: lightgreen;\">p</span><span style=\"background-color: lightgreen;\">e</span><span style=\"background-color: lightgreen;\">r</span><span style=\"background-color: lightgreen;\">s</span><span style=\"background-color: lightgreen;\">o</span><span style=\"background-color: lightgreen;\">n</span><span style=\"background-color: lightgreen;\">a</span><span style=\"background-color: lightgreen;\">l</span><span style=\"background-color: lightgreen;\"> </span><span style=\"background-color: lightgreen;\">e</span><span style=\"background-color: lightgreen;\">m</span><span style=\"background-color: lightgreen;\">o</span><span style=\"background-color: lightgreen;\">t</span><span style=\"background-color: lightgreen;\">i</span><span style=\"background-color: lightgreen;\">o</span><span style=\"background-color: lightgreen;\">n</span><span style=\"background-color: lightgreen;\">s</span><span style=\"background-color: lightgreen;\"> </span><span style=\"background-color: lightgreen;\">o</span><span style=\"background-color: lightgreen;\">r</span><span style=\"background-color: lightgreen;\"> </span><span style=\"background-color: lightgreen;\">p</span><span style=\"background-color: lightgreen;\">r</span><span style=\"background-color: lightgreen;\">e</span><span style=\"background-color: lightgreen;\">f</span><span style=\"background-color: lightgreen;\">e</span><span style=\"background-color: lightgreen;\">r</span><span style=\"background-color: lightgreen;\">e</span><span style=\"background-color: lightgreen;\">n</span><span style=\"background-color: lightgreen;\">c</span><span style=\"background-color: lightgreen;\">e</span><span style=\"background-color: lightgreen;\">s</span><span style=\"background-color: lightgreen;\">,</span><span style=\"background-color: lightgreen;\"> </span><span style=\"background-color: lightgreen;\">c</span><span style=\"background-color: lightgreen;\">l</span><span style=\"background-color: lightgreen;\">a</span><span style=\"background-color: lightgreen;\">s</span><span style=\"background-color: lightgreen;\">s</span><span style=\"background-color: lightgreen;\">i</span><span style=\"background-color: lightgreen;\">f</span><span style=\"background-color: lightgreen;\">y</span><span style=\"background-color: lightgreen;\"> </span><span style=\"background-color: lightgreen;\">i</span><span style=\"background-color: lightgreen;\">t</span><span style=\"background-color: lightgreen;\"> </span><span style=\"background-color: lightgreen;\">a</span><span style=\"background-color: lightgreen;\">s</span><span style=\"background-color: lightgreen;\"> </span><span style=\"background-color: lightgreen;\">'</span><span style=\"background-color: lightgreen;\">O</span><span style=\"background-color: lightgreen;\">b</span><span style=\"background-color: lightgreen;\">j</span><span style=\"background-color: lightgreen;\">e</span><span style=\"background-color: lightgreen;\">c</span><span style=\"background-color: lightgreen;\">t</span><span style=\"background-color: lightgreen;\">i</span><span style=\"background-color: lightgreen;\">v</span><span style=\"background-color: lightgreen;\">e</span><span style=\"background-color: lightgreen;\">'</span><span style=\"background-color: lightgreen;\">.</span>\""
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">\n",
       "\n",
       "=&gt; Iteration #<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1</span>: Getting feedback, analyzing and improving <span style=\"color: #808000; text-decoration-color: #808000\">...</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\n",
       "\n",
       "=> Iteration #\u001b[1;36m1\u001b[0m: Getting feedback, analyzing and improving \u001b[33m...\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">Applying skill: subjectivity_detection\n",
       "</pre>\n"
      ],
      "text/plain": [
       "Applying skill: subjectivity_detection\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|█| 5/5 [00:01<00:00, \n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">Predictions and feedback:\n",
       "</pre>\n"
      ],
      "text/plain": [
       "Predictions and feedback:\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">                                                                                                                   \n",
       " <span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\"> prediction__fb                           </span> <span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\"> text                                     </span> <span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\"> ground_truth </span> <span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\"> prediction </span> \n",
       " ───────────────────────────────────────────────────────────────────────────────────────────────────────────────── \n",
       "  Prediction is correct.                     The mic is great.                          Subjective     Subjective  \n",
       " <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\"> Prediction is correct.                   </span> <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\"> Will order from them again!              </span> <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\"> Subjective   </span> <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\"> Subjective </span> \n",
       "  Prediction is incorrect. Correct answer:   Not loud enough and doesn't turn on like   Objective      Subjective  \n",
       "  \"Objective\"                                it should.                                                            \n",
       " <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\"> Prediction is correct.                   </span> <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\"> The phone doesn't seem to accept         </span> <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\"> Objective    </span> <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\"> Objective  </span> \n",
       " <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                                          </span> <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\"> anything except CBR mp3s                 </span> <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">              </span> <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">            </span> \n",
       "  Prediction is correct.                     All three broke within two months of       Objective      Objective   \n",
       "                                             use.                                                                  \n",
       "                                                                                                                   \n",
       "</pre>\n"
      ],
      "text/plain": [
       "                                                                                                                   \n",
       " \u001b[1;35m \u001b[0m\u001b[1;35mprediction__fb                          \u001b[0m\u001b[1;35m \u001b[0m \u001b[1;35m \u001b[0m\u001b[1;35mtext                                    \u001b[0m\u001b[1;35m \u001b[0m \u001b[1;35m \u001b[0m\u001b[1;35mground_truth\u001b[0m\u001b[1;35m \u001b[0m \u001b[1;35m \u001b[0m\u001b[1;35mprediction\u001b[0m\u001b[1;35m \u001b[0m \n",
       " ───────────────────────────────────────────────────────────────────────────────────────────────────────────────── \n",
       "  Prediction is correct.                     The mic is great.                          Subjective     Subjective  \n",
       " \u001b[2m \u001b[0m\u001b[2mPrediction is correct.                  \u001b[0m\u001b[2m \u001b[0m \u001b[2m \u001b[0m\u001b[2mWill order from them again!             \u001b[0m\u001b[2m \u001b[0m \u001b[2m \u001b[0m\u001b[2mSubjective  \u001b[0m\u001b[2m \u001b[0m \u001b[2m \u001b[0m\u001b[2mSubjective\u001b[0m\u001b[2m \u001b[0m \n",
       "  Prediction is incorrect. Correct answer:   Not loud enough and doesn't turn on like   Objective      Subjective  \n",
       "  \"Objective\"                                it should.                                                            \n",
       " \u001b[2m \u001b[0m\u001b[2mPrediction is correct.                  \u001b[0m\u001b[2m \u001b[0m \u001b[2m \u001b[0m\u001b[2mThe phone doesn't seem to accept        \u001b[0m\u001b[2m \u001b[0m \u001b[2m \u001b[0m\u001b[2mObjective   \u001b[0m\u001b[2m \u001b[0m \u001b[2m \u001b[0m\u001b[2mObjective \u001b[0m\u001b[2m \u001b[0m \n",
       " \u001b[2m                                          \u001b[0m \u001b[2m \u001b[0m\u001b[2manything except CBR mp3s                \u001b[0m\u001b[2m \u001b[0m \u001b[2m              \u001b[0m \u001b[2m            \u001b[0m \n",
       "  Prediction is correct.                     All three broke within two months of       Objective      Objective   \n",
       "                                             use.                                                                  \n",
       "                                                                                                                   \n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #800000; text-decoration-color: #800000; font-weight: bold\">Output to improve: </span><span style=\"color: #008000; text-decoration-color: #008000\">\"prediction\"</span><span style=\"color: #800000; text-decoration-color: #800000; font-weight: bold\"> (</span><span style=\"color: #808000; text-decoration-color: #808000; font-weight: bold\">Skill</span><span style=\"color: #800000; text-decoration-color: #800000; font-weight: bold\">=</span><span style=\"color: #008000; text-decoration-color: #008000\">\"subjectivity_detection\"</span><span style=\"color: #800000; text-decoration-color: #800000; font-weight: bold\">)</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000; font-weight: bold\">Accuracy = </span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">80.00</span><span style=\"color: #800000; text-decoration-color: #800000; font-weight: bold\">%</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[1;31mOutput to improve: \u001b[0m\u001b[32m\"prediction\"\u001b[0m\u001b[1;31m \u001b[0m\u001b[1;31m(\u001b[0m\u001b[1;33mSkill\u001b[0m\u001b[1;31m=\u001b[0m\u001b[32m\"subjectivity_detection\"\u001b[0m\u001b[1;31m)\u001b[0m\n",
       "\u001b[1;31mAccuracy = \u001b[0m\u001b[1;36m80.00\u001b[0m\u001b[1;31m%\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #008000; text-decoration-color: #008000\">\"{system}\"</span>:\n",
       "You are a helpful assistant.\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[32m\"\u001b[0m\u001b[32m{\u001b[0m\u001b[32msystem\u001b[0m\u001b[32m}\u001b[0m\u001b[32m\"\u001b[0m:\n",
       "You are a helpful assistant.\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #008000; text-decoration-color: #008000\">\"{user}\"</span>:\n",
       "\n",
       "A prompt is a text paragraph that outlines the expected actions and instructs the large language model <span style=\"font-weight: bold\">(</span>LLM<span style=\"font-weight: bold\">)</span> to \n",
       "generate a specific output. This prompt is concatenated with the input text, and the model then creates the \n",
       "required output.\n",
       "This describes the full template how the prompt is concatenated with the input to produce the output:\n",
       "\n",
       "```\n",
       "\n",
       "<span style=\"font-weight: bold\">{</span>prompt<span style=\"font-weight: bold\">}</span>\n",
       "Input: <span style=\"font-weight: bold\">{</span>text<span style=\"font-weight: bold\">}</span>\n",
       "Output: <span style=\"font-weight: bold\">{</span>prediction<span style=\"font-weight: bold\">}</span>\n",
       "```\n",
       "\n",
       "Here:\n",
       "- <span style=\"color: #008000; text-decoration-color: #008000\">\"Input: {text}\"</span> is input template,\n",
       "- <span style=\"color: #008000; text-decoration-color: #008000\">\"{prompt}\"</span> is the LLM prompt,\n",
       "- <span style=\"color: #008000; text-decoration-color: #008000\">\"Output: {prediction}\"</span> is the output template.\n",
       "\n",
       "Model can produce erroneous output if a prompt is not well defined. In our collaboration, we’ll work together to \n",
       "refine a prompt. The process consists of two main steps:\n",
       "\n",
       "## Step <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1</span>\n",
       "I will provide you with the current prompt along with prediction examples. Each example contains the input text, \n",
       "the final prediction produced by the model, and the user feedback. User feedback indicates whether the model \n",
       "prediction is correct or not. Your task is to analyze the examples and user feedback, determining whether the \n",
       "existing instruction is describing the task reflected by these examples precisely, and suggests changes to the \n",
       "prompt to address the incorrect predictions.\n",
       "\n",
       "## Step <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2</span>\n",
       "Next, you will carefully review your reasoning in step <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1</span>, integrate the insights to refine the prompt, and provide \n",
       "me with the new prompt that improves the model’s performance.\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[32m\"\u001b[0m\u001b[32m{\u001b[0m\u001b[32muser\u001b[0m\u001b[32m}\u001b[0m\u001b[32m\"\u001b[0m:\n",
       "\n",
       "A prompt is a text paragraph that outlines the expected actions and instructs the large language model \u001b[1m(\u001b[0mLLM\u001b[1m)\u001b[0m to \n",
       "generate a specific output. This prompt is concatenated with the input text, and the model then creates the \n",
       "required output.\n",
       "This describes the full template how the prompt is concatenated with the input to produce the output:\n",
       "\n",
       "```\n",
       "\n",
       "\u001b[1m{\u001b[0mprompt\u001b[1m}\u001b[0m\n",
       "Input: \u001b[1m{\u001b[0mtext\u001b[1m}\u001b[0m\n",
       "Output: \u001b[1m{\u001b[0mprediction\u001b[1m}\u001b[0m\n",
       "```\n",
       "\n",
       "Here:\n",
       "- \u001b[32m\"Input: \u001b[0m\u001b[32m{\u001b[0m\u001b[32mtext\u001b[0m\u001b[32m}\u001b[0m\u001b[32m\"\u001b[0m is input template,\n",
       "- \u001b[32m\"\u001b[0m\u001b[32m{\u001b[0m\u001b[32mprompt\u001b[0m\u001b[32m}\u001b[0m\u001b[32m\"\u001b[0m is the LLM prompt,\n",
       "- \u001b[32m\"Output: \u001b[0m\u001b[32m{\u001b[0m\u001b[32mprediction\u001b[0m\u001b[32m}\u001b[0m\u001b[32m\"\u001b[0m is the output template.\n",
       "\n",
       "Model can produce erroneous output if a prompt is not well defined. In our collaboration, we’ll work together to \n",
       "refine a prompt. The process consists of two main steps:\n",
       "\n",
       "## Step \u001b[1;36m1\u001b[0m\n",
       "I will provide you with the current prompt along with prediction examples. Each example contains the input text, \n",
       "the final prediction produced by the model, and the user feedback. User feedback indicates whether the model \n",
       "prediction is correct or not. Your task is to analyze the examples and user feedback, determining whether the \n",
       "existing instruction is describing the task reflected by these examples precisely, and suggests changes to the \n",
       "prompt to address the incorrect predictions.\n",
       "\n",
       "## Step \u001b[1;36m2\u001b[0m\n",
       "Next, you will carefully review your reasoning in step \u001b[1;36m1\u001b[0m, integrate the insights to refine the prompt, and provide \n",
       "me with the new prompt that improves the model’s performance.\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #008000; text-decoration-color: #008000\">\"{assistant}\"</span>:\n",
       "Sure, I’d be happy to help you with this prompt engineering problem. Please provide me with the current prompt and \n",
       "the examples with user feedback.\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[32m\"\u001b[0m\u001b[32m{\u001b[0m\u001b[32massistant\u001b[0m\u001b[32m}\u001b[0m\u001b[32m\"\u001b[0m:\n",
       "Sure, I’d be happy to help you with this prompt engineering problem. Please provide me with the current prompt and \n",
       "the examples with user feedback.\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #008000; text-decoration-color: #008000\">\"{user}\"</span>:\n",
       "\n",
       "## Current prompt\n",
       "<span style=\"color: #008000; text-decoration-color: #008000\">\"Analyze the provided product review and determine whether it primarily expresses 'Subjective' or 'Objective' </span>\n",
       "<span style=\"color: #008000; text-decoration-color: #008000\">statements. Classify it as 'Subjective' if the review mainly expresses personal feelings, emotions, or preferences.</span>\n",
       "<span style=\"color: #008000; text-decoration-color: #008000\">If the review provides factual details related directly to the functionality or quality of the product, even based </span>\n",
       "<span style=\"color: #008000; text-decoration-color: #008000\">on personal experiences, without expressing personal emotions or preferences, classify it as 'Objective'.\"</span>\n",
       "\n",
       "## Examples\n",
       "### Example #<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0</span>\n",
       "\n",
       "Input: The mic is great.\n",
       "\n",
       "Output: Subjective\n",
       "\n",
       "User feedback: Prediction is correct.\n",
       "\n",
       "\n",
       "### Example #<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1</span>\n",
       "\n",
       "Input: Will order from them again!\n",
       "\n",
       "Output: Subjective\n",
       "\n",
       "User feedback: Prediction is correct.\n",
       "\n",
       "\n",
       "### Example #<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2</span>\n",
       "\n",
       "Input: Not loud enough and doesn't turn on like it should.\n",
       "\n",
       "Output: Subjective\n",
       "\n",
       "User feedback: Prediction is incorrect. Correct answer: <span style=\"color: #008000; text-decoration-color: #008000\">\"Objective\"</span>\n",
       "\n",
       "\n",
       "### Example #<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">3</span>\n",
       "\n",
       "Input: The phone doesn't seem to accept anything except CBR mp3s\n",
       "\n",
       "Output: Objective\n",
       "\n",
       "User feedback: Prediction is correct.\n",
       "\n",
       "\n",
       "### Example #<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">4</span>\n",
       "\n",
       "Input: All three broke within two months of use.\n",
       "\n",
       "Output: Objective\n",
       "\n",
       "User feedback: Prediction is correct.\n",
       "\n",
       "\n",
       "\n",
       "Summarize your analysis about incorrect predictions and suggest changes to the prompt.\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[32m\"\u001b[0m\u001b[32m{\u001b[0m\u001b[32muser\u001b[0m\u001b[32m}\u001b[0m\u001b[32m\"\u001b[0m:\n",
       "\n",
       "## Current prompt\n",
       "\u001b[32m\"Analyze the provided product review and determine whether it primarily expresses 'Subjective' or 'Objective' \u001b[0m\n",
       "\u001b[32mstatements. Classify it as 'Subjective' if the review mainly expresses personal feelings, emotions, or preferences.\u001b[0m\n",
       "\u001b[32mIf the review provides factual details related directly to the functionality or quality of the product, even based \u001b[0m\n",
       "\u001b[32mon personal experiences, without expressing personal emotions or preferences, classify it as 'Objective'.\"\u001b[0m\n",
       "\n",
       "## Examples\n",
       "### Example #\u001b[1;36m0\u001b[0m\n",
       "\n",
       "Input: The mic is great.\n",
       "\n",
       "Output: Subjective\n",
       "\n",
       "User feedback: Prediction is correct.\n",
       "\n",
       "\n",
       "### Example #\u001b[1;36m1\u001b[0m\n",
       "\n",
       "Input: Will order from them again!\n",
       "\n",
       "Output: Subjective\n",
       "\n",
       "User feedback: Prediction is correct.\n",
       "\n",
       "\n",
       "### Example #\u001b[1;36m2\u001b[0m\n",
       "\n",
       "Input: Not loud enough and doesn't turn on like it should.\n",
       "\n",
       "Output: Subjective\n",
       "\n",
       "User feedback: Prediction is incorrect. Correct answer: \u001b[32m\"Objective\"\u001b[0m\n",
       "\n",
       "\n",
       "### Example #\u001b[1;36m3\u001b[0m\n",
       "\n",
       "Input: The phone doesn't seem to accept anything except CBR mp3s\n",
       "\n",
       "Output: Objective\n",
       "\n",
       "User feedback: Prediction is correct.\n",
       "\n",
       "\n",
       "### Example #\u001b[1;36m4\u001b[0m\n",
       "\n",
       "Input: All three broke within two months of use.\n",
       "\n",
       "Output: Objective\n",
       "\n",
       "User feedback: Prediction is correct.\n",
       "\n",
       "\n",
       "\n",
       "Summarize your analysis about incorrect predictions and suggest changes to the prompt.\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #008000; text-decoration-color: #008000\">\"{assistant}\"</span>:\n",
       "Looking at the examples you've provided, it appears the model struggles with differentiating subjective and \n",
       "objective statements when the commentary is based on personal experiences but is factual in nature. \n",
       "\n",
       "For instance, in Example #<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2</span>, the user states, <span style=\"color: #008000; text-decoration-color: #008000\">\"Not loud enough and doesn't turn on like it should.\"</span> While this is \n",
       "drawn from personal observation, the feedback does not express any personal feelings or preferences, but relies on \n",
       "the product's performance. This should be classified as an objective statement, but the model classified it as \n",
       "subjective.\n",
       "\n",
       "The current prompt interprets any personal experience as subjective, which can lead to errors. However, statements \n",
       "based on personal observations can still be objective if the user simply describes the functionality or performance\n",
       "of a product without expressing emotions or preferences.\n",
       "\n",
       "To address this, we may want to refine the prompt to underline that not all personal experience-based inputs are \n",
       "subjective. One possible amendment could be:\n",
       "\n",
       "<span style=\"color: #008000; text-decoration-color: #008000\">\"Analyze the provided product review and determine whether it primarily expresses 'Subjective' or 'Objective' </span>\n",
       "<span style=\"color: #008000; text-decoration-color: #008000\">statements. Classify it as 'Subjective' if the review mainly expresses personal feelings, emotions, or preferences.</span>\n",
       "<span style=\"color: #008000; text-decoration-color: #008000\">If the review offers factual details about the product’s performance or quality, even if these details are grounded</span>\n",
       "<span style=\"color: #008000; text-decoration-color: #008000\">in personal experiences, but does not express personal feelings, emotions, or preferences, classify it as </span>\n",
       "<span style=\"color: #008000; text-decoration-color: #008000\">'Objective'. Note that a personal experience can be stated objectively if it only describes the product's </span>\n",
       "<span style=\"color: #008000; text-decoration-color: #008000\">functionality without expressing personal emotions or preferences.\"</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[32m\"\u001b[0m\u001b[32m{\u001b[0m\u001b[32massistant\u001b[0m\u001b[32m}\u001b[0m\u001b[32m\"\u001b[0m:\n",
       "Looking at the examples you've provided, it appears the model struggles with differentiating subjective and \n",
       "objective statements when the commentary is based on personal experiences but is factual in nature. \n",
       "\n",
       "For instance, in Example #\u001b[1;36m2\u001b[0m, the user states, \u001b[32m\"Not loud enough and doesn't turn on like it should.\"\u001b[0m While this is \n",
       "drawn from personal observation, the feedback does not express any personal feelings or preferences, but relies on \n",
       "the product's performance. This should be classified as an objective statement, but the model classified it as \n",
       "subjective.\n",
       "\n",
       "The current prompt interprets any personal experience as subjective, which can lead to errors. However, statements \n",
       "based on personal observations can still be objective if the user simply describes the functionality or performance\n",
       "of a product without expressing emotions or preferences.\n",
       "\n",
       "To address this, we may want to refine the prompt to underline that not all personal experience-based inputs are \n",
       "subjective. One possible amendment could be:\n",
       "\n",
       "\u001b[32m\"Analyze the provided product review and determine whether it primarily expresses 'Subjective' or 'Objective' \u001b[0m\n",
       "\u001b[32mstatements. Classify it as 'Subjective' if the review mainly expresses personal feelings, emotions, or preferences.\u001b[0m\n",
       "\u001b[32mIf the review offers factual details about the product’s performance or quality, even if these details are grounded\u001b[0m\n",
       "\u001b[32min personal experiences, but does not express personal feelings, emotions, or preferences, classify it as \u001b[0m\n",
       "\u001b[32m'Objective'. Note that a personal experience can be stated objectively if it only describes the product's \u001b[0m\n",
       "\u001b[32mfunctionality without expressing personal emotions or preferences.\"\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #008000; text-decoration-color: #008000\">\"{user}\"</span>:\n",
       "\n",
       "Now please carefully review your reasoning in Step <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1</span> and help with Step <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2</span>: refining the prompt.\n",
       "\n",
       "## Current prompt\n",
       "<span style=\"color: #008000; text-decoration-color: #008000\">\"Analyze the provided product review and determine whether it primarily expresses 'Subjective' or 'Objective' </span>\n",
       "<span style=\"color: #008000; text-decoration-color: #008000\">statements. Classify it as 'Subjective' if the review mainly expresses personal feelings, emotions, or preferences.</span>\n",
       "<span style=\"color: #008000; text-decoration-color: #008000\">If the review provides factual details related directly to the functionality or quality of the product, even based </span>\n",
       "<span style=\"color: #008000; text-decoration-color: #008000\">on personal experiences, without expressing personal emotions or preferences, classify it as 'Objective'.\"</span>\n",
       "\n",
       "## Follow this guidance to refine the prompt:\n",
       "\n",
       "<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1</span>. The new prompt should should describe the task precisely, and address the points raised in the user feedback.\n",
       "    \n",
       "<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2</span>. The new prompt should be similar to the current instruction, and only differ in the parts that address the \n",
       "issues you identified in Step <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1</span>.\n",
       "    Example:\n",
       "    - Current prompt: <span style=\"color: #008000; text-decoration-color: #008000\">\"The model should generate a summary of the input text.\"</span>\n",
       "    - New prompt: <span style=\"color: #008000; text-decoration-color: #008000\">\"The model should generate a summary of the input text. Pay attention to the original style.\"</span>\n",
       "        \n",
       "<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">3</span>. Reply only with the new prompt. Do not include input and output templates in the prompt.\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[32m\"\u001b[0m\u001b[32m{\u001b[0m\u001b[32muser\u001b[0m\u001b[32m}\u001b[0m\u001b[32m\"\u001b[0m:\n",
       "\n",
       "Now please carefully review your reasoning in Step \u001b[1;36m1\u001b[0m and help with Step \u001b[1;36m2\u001b[0m: refining the prompt.\n",
       "\n",
       "## Current prompt\n",
       "\u001b[32m\"Analyze the provided product review and determine whether it primarily expresses 'Subjective' or 'Objective' \u001b[0m\n",
       "\u001b[32mstatements. Classify it as 'Subjective' if the review mainly expresses personal feelings, emotions, or preferences.\u001b[0m\n",
       "\u001b[32mIf the review provides factual details related directly to the functionality or quality of the product, even based \u001b[0m\n",
       "\u001b[32mon personal experiences, without expressing personal emotions or preferences, classify it as 'Objective'.\"\u001b[0m\n",
       "\n",
       "## Follow this guidance to refine the prompt:\n",
       "\n",
       "\u001b[1;36m1\u001b[0m. The new prompt should should describe the task precisely, and address the points raised in the user feedback.\n",
       "    \n",
       "\u001b[1;36m2\u001b[0m. The new prompt should be similar to the current instruction, and only differ in the parts that address the \n",
       "issues you identified in Step \u001b[1;36m1\u001b[0m.\n",
       "    Example:\n",
       "    - Current prompt: \u001b[32m\"The model should generate a summary of the input text.\"\u001b[0m\n",
       "    - New prompt: \u001b[32m\"The model should generate a summary of the input text. Pay attention to the original style.\"\u001b[0m\n",
       "        \n",
       "\u001b[1;36m3\u001b[0m. Reply only with the new prompt. Do not include input and output templates in the prompt.\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "\"Analyze the provided product review and a<span style=\"background-color: lightgreen;\">s</span><span style=\"background-color: lightgreen;\">c</span><span style=\"background-color: lightgreen;\">e</span>r<span style=\"background-color: lightgreen;\">t</span><span style=\"background-color: lightgreen;\">a</span>i<span style=\"background-color: lightgreen;\">n</span><span style=\"background-color: lightgreen;\"> </span><span style=\"background-color: lightgreen;\">w</span><span style=\"background-color: lightgreen;\">h</span><span style=\"background-color: lightgreen;\">e</span><span style=\"background-color: lightgreen;\">t</span><span style=\"background-color: lightgreen;\">h</span><span style=\"background-color: lightgreen;\">e</span><span style=\"background-color: lightgreen;\">r</span><span style=\"background-color: lightgreen;\"> </span><span style=\"background-color: lightgreen;\">i</span><span style=\"background-color: lightgreen;\">t</span><span style=\"background-color: lightgreen;\"> </span><span style=\"background-color: lightgreen;\">p</span><span style=\"background-color: lightgreen;\">r</span><span style=\"background-color: lightgreen;\">e</span><span style=\"background-color: lightgreen;\">d</span><span style=\"background-color: lightgreen;\">o</span><span style=\"background-color: lightgreen;\">m</span><span style=\"background-color: lightgreen;\">i</span><span style=\"background-color: lightgreen;\">n</span><span style=\"background-color: lightgreen;\">a</span><span style=\"background-color: lightgreen;\">n</span><span style=\"background-color: lightgreen;\">t</span>ly expresses 'Subjective' or 'Objective' statements. Classify it as 'Subjective' if the review <span style=\"background-color: lightgreen;\">p</span><span style=\"background-color: lightgreen;\">r</span>i<span style=\"background-color: lightgreen;\">m</span><span style=\"background-color: lightgreen;\">a</span><span style=\"background-color: lightgreen;\">r</span><span style=\"background-color: lightgreen;\">i</span>ly<span style=\"background-color: lightgreen;\"> </span><span style=\"background-color: lightgreen;\">d</span><span style=\"background-color: lightgreen;\">i</span><span style=\"background-color: lightgreen;\">s</span><span style=\"background-color: lightgreen;\">c</span><span style=\"background-color: lightgreen;\">l</span><span style=\"background-color: lightgreen;\">o</span><span style=\"background-color: lightgreen;\">s</span><span style=\"background-color: lightgreen;\">e</span><span style=\"background-color: lightgreen;\">s</span><span style=\"background-color: lightgreen;\"> </span><span style=\"background-color: lightgreen;\">p</span><span style=\"background-color: lightgreen;\">e</span><span style=\"background-color: lightgreen;\">r</span><span style=\"background-color: lightgreen;\">s</span><span style=\"background-color: lightgreen;\">o</span><span style=\"background-color: lightgreen;\">n</span><span style=\"background-color: lightgreen;\">a</span><span style=\"background-color: lightgreen;\">l</span><span style=\"background-color: lightgreen;\"> </span><span style=\"background-color: lightgreen;\">f</span><span style=\"background-color: lightgreen;\">e</span><span style=\"background-color: lightgreen;\">e</span><span style=\"background-color: lightgreen;\">l</span><span style=\"background-color: lightgreen;\">i</span><span style=\"background-color: lightgreen;\">n</span><span style=\"background-color: lightgreen;\">g</span><span style=\"background-color: lightgreen;\">s</span><span style=\"background-color: lightgreen;\">,</span><span style=\"background-color: lightgreen;\"> </span><span style=\"background-color: lightgreen;\">e</span><span style=\"background-color: lightgreen;\">m</span><span style=\"background-color: lightgreen;\">o</span><span style=\"background-color: lightgreen;\">t</span><span style=\"background-color: lightgreen;\">i</span><span style=\"background-color: lightgreen;\">o</span><span style=\"background-color: lightgreen;\">n</span><span style=\"background-color: lightgreen;\">s</span><span style=\"background-color: lightgreen;\">,</span><span style=\"background-color: lightgreen;\"> </span><span style=\"background-color: lightgreen;\">o</span><span style=\"background-color: lightgreen;\">r</span><span style=\"background-color: lightgreen;\"> </span><span style=\"background-color: lightgreen;\">p</span><span style=\"background-color: lightgreen;\">r</span><span style=\"background-color: lightgreen;\">e</span><span style=\"background-color: lightgreen;\">f</span><span style=\"background-color: lightgreen;\">e</span><span style=\"background-color: lightgreen;\">r</span><span style=\"background-color: lightgreen;\">e</span><span style=\"background-color: lightgreen;\">n</span><span style=\"background-color: lightgreen;\">c</span><span style=\"background-color: lightgreen;\">e</span><span style=\"background-color: lightgreen;\">s</span><span style=\"background-color: lightgreen;\">.</span><span style=\"background-color: lightgreen;\"> </span><span style=\"background-color: lightgreen;\">H</span><span style=\"background-color: lightgreen;\">o</span><span style=\"background-color: lightgreen;\">w</span><span style=\"background-color: lightgreen;\">e</span><span style=\"background-color: lightgreen;\">v</span><span style=\"background-color: lightgreen;\">e</span><span style=\"background-color: lightgreen;\">r</span><span style=\"background-color: lightgreen;\">,</span><span style=\"background-color: lightgreen;\"> </span><span style=\"background-color: lightgreen;\">s</span><span style=\"background-color: lightgreen;\">h</span><span style=\"background-color: lightgreen;\">o</span><span style=\"background-color: lightgreen;\">u</span><span style=\"background-color: lightgreen;\">l</span><span style=\"background-color: lightgreen;\">d</span><span style=\"background-color: lightgreen;\"> </span><span style=\"background-color: lightgreen;\">t</span><span style=\"background-color: lightgreen;\">h</span><span style=\"background-color: lightgreen;\">e</span><span style=\"background-color: lightgreen;\"> </span><span style=\"background-color: lightgreen;\">r</span><span style=\"background-color: lightgreen;\">e</span><span style=\"background-color: lightgreen;\">v</span><span style=\"background-color: lightgreen;\">i</span><span style=\"background-color: lightgreen;\">e</span><span style=\"background-color: lightgreen;\">w</span><span style=\"background-color: lightgreen;\"> </span><span style=\"background-color: lightgreen;\">d</span><span style=\"background-color: lightgreen;\">e</span><span style=\"background-color: lightgreen;\">s</span><span style=\"background-color: lightgreen;\">c</span><span style=\"background-color: lightgreen;\">r</span><span style=\"background-color: lightgreen;\">i</span><span style=\"background-color: lightgreen;\">b</span><span style=\"background-color: lightgreen;\">e</span><span style=\"background-color: lightgreen;\"> </span><span style=\"background-color: lightgreen;\">f</span><span style=\"background-color: lightgreen;\">a</span><span style=\"background-color: lightgreen;\">c</span><span style=\"background-color: lightgreen;\">t</span><span style=\"background-color: lightgreen;\">u</span><span style=\"background-color: lightgreen;\">a</span><span style=\"background-color: lightgreen;\">l</span><span style=\"background-color: lightgreen;\"> </span><span style=\"background-color: lightgreen;\">d</span><span style=\"background-color: lightgreen;\">e</span><span style=\"background-color: lightgreen;\">t</span><span style=\"background-color: lightgreen;\">a</span><span style=\"background-color: lightgreen;\">i</span><span style=\"background-color: lightgreen;\">l</span><span style=\"background-color: lightgreen;\">s</span><span style=\"background-color: lightgreen;\"> </span><span style=\"background-color: lightgreen;\">a</span><span style=\"background-color: lightgreen;\">b</span><span style=\"background-color: lightgreen;\">o</span><span style=\"background-color: lightgreen;\">u</span><span style=\"background-color: lightgreen;\">t</span><span style=\"background-color: lightgreen;\"> </span><span style=\"background-color: lightgreen;\">t</span><span style=\"background-color: lightgreen;\">h</span><span style=\"background-color: lightgreen;\">e</span><span style=\"background-color: lightgreen;\"> </span><span style=\"background-color: lightgreen;\">p</span><span style=\"background-color: lightgreen;\">r</span><span style=\"background-color: lightgreen;\">o</span><span style=\"background-color: lightgreen;\">d</span><span style=\"background-color: lightgreen;\">u</span><span style=\"background-color: lightgreen;\">c</span><span style=\"background-color: lightgreen;\">t</span><span style=\"background-color: lightgreen;\">’</span><span style=\"background-color: lightgreen;\">s</span><span style=\"background-color: lightgreen;\"> </span><span style=\"background-color: lightgreen;\">f</span><span style=\"background-color: lightgreen;\">u</span><span style=\"background-color: lightgreen;\">n</span><span style=\"background-color: lightgreen;\">c</span><span style=\"background-color: lightgreen;\">t</span><span style=\"background-color: lightgreen;\">i</span><span style=\"background-color: lightgreen;\">o</span><span style=\"background-color: lightgreen;\">n</span><span style=\"background-color: lightgreen;\">a</span><span style=\"background-color: lightgreen;\">l</span><span style=\"background-color: lightgreen;\">i</span><span style=\"background-color: lightgreen;\">t</span><span style=\"background-color: lightgreen;\">y</span><span style=\"background-color: lightgreen;\"> </span><span style=\"background-color: lightgreen;\">o</span><span style=\"background-color: lightgreen;\">r</span><span style=\"background-color: lightgreen;\"> </span><span style=\"background-color: lightgreen;\">p</span><span style=\"background-color: lightgreen;\">e</span><span style=\"background-color: lightgreen;\">r</span><span style=\"background-color: lightgreen;\">f</span><span style=\"background-color: lightgreen;\">o</span><span style=\"background-color: lightgreen;\">r</span><span style=\"background-color: lightgreen;\">m</span><span style=\"background-color: lightgreen;\">a</span><span style=\"background-color: lightgreen;\">n</span><span style=\"background-color: lightgreen;\">c</span><span style=\"background-color: lightgreen;\">e</span><span style=\"background-color: lightgreen;\">,</span><span style=\"background-color: lightgreen;\"> </span><span style=\"background-color: lightgreen;\">e</span><span style=\"background-color: lightgreen;\">v</span><span style=\"background-color: lightgreen;\">e</span><span style=\"background-color: lightgreen;\">n</span><span style=\"background-color: lightgreen;\"> </span><span style=\"background-color: lightgreen;\">i</span><span style=\"background-color: lightgreen;\">f</span><span style=\"background-color: lightgreen;\"> </span><span style=\"background-color: lightgreen;\">t</span><span style=\"background-color: lightgreen;\">h</span><span style=\"background-color: lightgreen;\">e</span><span style=\"background-color: lightgreen;\">s</span><span style=\"background-color: lightgreen;\">e</span><span style=\"background-color: lightgreen;\"> </span><span style=\"background-color: lightgreen;\">d</span><span style=\"background-color: lightgreen;\">e</span><span style=\"background-color: lightgreen;\">t</span><span style=\"background-color: lightgreen;\">a</span><span style=\"background-color: lightgreen;\">i</span><span style=\"background-color: lightgreen;\">l</span><span style=\"background-color: lightgreen;\">s</span><span style=\"background-color: lightgreen;\"> </span><span style=\"background-color: lightgreen;\">s</span><span style=\"background-color: lightgreen;\">t</span><span style=\"background-color: lightgreen;\">e</span><span style=\"background-color: lightgreen;\">m</span><span style=\"background-color: lightgreen;\"> </span><span style=\"background-color: lightgreen;\">f</span><span style=\"background-color: lightgreen;\">r</span><span style=\"background-color: lightgreen;\">o</span><span style=\"background-color: lightgreen;\">m</span><span style=\"background-color: lightgreen;\"> </span><span style=\"background-color: lightgreen;\">p</span><span style=\"background-color: lightgreen;\">e</span><span style=\"background-color: lightgreen;\">r</span><span style=\"background-color: lightgreen;\">s</span><span style=\"background-color: lightgreen;\">o</span><span style=\"background-color: lightgreen;\">n</span><span style=\"background-color: lightgreen;\">a</span><span style=\"background-color: lightgreen;\">l</span> exper<span style=\"background-color: lightgreen;\">i</span>e<span style=\"background-color: lightgreen;\">n</span><span style=\"background-color: lightgreen;\">c</span>e<span style=\"background-color: lightgreen;\">s</span><span style=\"background-color: lightgreen;\">,</span><span style=\"background-color: lightgreen;\"> </span><span style=\"background-color: lightgreen;\">y</span><span style=\"background-color: lightgreen;\">e</span><span style=\"background-color: lightgreen;\">t</span><span style=\"background-color: lightgreen;\"> </span><span style=\"background-color: lightgreen;\">d</span><span style=\"background-color: lightgreen;\">o</span><span style=\"background-color: lightgreen;\"> </span><span style=\"background-color: lightgreen;\">n</span><span style=\"background-color: lightgreen;\">o</span><span style=\"background-color: lightgreen;\">t</span><span style=\"background-color: lightgreen;\"> </span><span style=\"background-color: lightgreen;\">a</span><span style=\"background-color: lightgreen;\">s</span><span style=\"background-color: lightgreen;\">s</span><span style=\"background-color: lightgreen;\">e</span><span style=\"background-color: lightgreen;\">r</span><span style=\"background-color: lightgreen;\">t</span><span style=\"background-color: lightgreen;\"> </span><span style=\"background-color: lightgreen;\">p</span><span style=\"background-color: lightgreen;\">e</span><span style=\"background-color: lightgreen;\">r</span><span style=\"background-color: lightgreen;\">s</span><span style=\"background-color: lightgreen;\">o</span><span style=\"background-color: lightgreen;\">n</span><span style=\"background-color: lightgreen;\">a</span>l<span style=\"background-color: lightgreen;\"> </span><span style=\"background-color: lightgreen;\">e</span><span style=\"background-color: lightgreen;\">m</span><span style=\"background-color: lightgreen;\">o</span><span style=\"background-color: lightgreen;\">t</span><span style=\"background-color: lightgreen;\">i</span><span style=\"background-color: lightgreen;\">o</span><span style=\"background-color: lightgreen;\">n</span><span style=\"background-color: lightgreen;\">s</span><span style=\"background-color: lightgreen;\"> </span><span style=\"background-color: lightgreen;\">o</span><span style=\"background-color: lightgreen;\">r</span><span style=\"background-color: lightgreen;\"> </span><span style=\"background-color: lightgreen;\">p</span><span style=\"background-color: lightgreen;\">r</span><span style=\"background-color: lightgreen;\">e</span><span style=\"background-color: lightgreen;\">f</span><span style=\"background-color: lightgreen;\">e</span><span style=\"background-color: lightgreen;\">r</span><span style=\"background-color: lightgreen;\">e</span><span style=\"background-color: lightgreen;\">n</span><span style=\"background-color: lightgreen;\">c</span><span style=\"background-color: lightgreen;\">e</span><span style=\"background-color: lightgreen;\">s</span><span style=\"background-color: lightgreen;\">,</span><span style=\"background-color: lightgreen;\"> </span><span style=\"background-color: lightgreen;\">c</span><span style=\"background-color: lightgreen;\">l</span><span style=\"background-color: lightgreen;\">a</span><span style=\"background-color: lightgreen;\">s</span><span style=\"background-color: lightgreen;\">s</span><span style=\"background-color: lightgreen;\">i</span><span style=\"background-color: lightgreen;\">f</span><span style=\"background-color: lightgreen;\">y</span><span style=\"background-color: lightgreen;\"> </span><span style=\"background-color: lightgreen;\">i</span><span style=\"background-color: lightgreen;\">t</span><span style=\"background-color: lightgreen;\"> </span><span style=\"background-color: lightgreen;\">a</span><span style=\"background-color: lightgreen;\">s</span><span style=\"background-color: lightgreen;\"> </span><span style=\"background-color: lightgreen;\">'</span><span style=\"background-color: lightgreen;\">O</span><span style=\"background-color: lightgreen;\">b</span><span style=\"background-color: lightgreen;\">j</span><span style=\"background-color: lightgreen;\">e</span><span style=\"background-color: lightgreen;\">c</span><span style=\"background-color: lightgreen;\">t</span><span style=\"background-color: lightgreen;\">i</span><span style=\"background-color: lightgreen;\">v</span><span style=\"background-color: lightgreen;\">e</span><span style=\"background-color: lightgreen;\">'</span><span style=\"background-color: lightgreen;\">.</span><span style=\"background-color: lightgreen;\"> </span><span style=\"background-color: lightgreen;\">K</span><span style=\"background-color: lightgreen;\">e</span><span style=\"background-color: lightgreen;\">e</span><span style=\"background-color: lightgreen;\">p</span><span style=\"background-color: lightgreen;\"> </span><span style=\"background-color: lightgreen;\">i</span><span style=\"background-color: lightgreen;\">n</span><span style=\"background-color: lightgreen;\"> </span><span style=\"background-color: lightgreen;\">m</span><span style=\"background-color: lightgreen;\">i</span><span style=\"background-color: lightgreen;\">n</span><span style=\"background-color: lightgreen;\">d</span><span style=\"background-color: lightgreen;\"> </span><span style=\"background-color: lightgreen;\">t</span><span style=\"background-color: lightgreen;\">h</span><span style=\"background-color: lightgreen;\">a</span><span style=\"background-color: lightgreen;\">t</span><span style=\"background-color: lightgreen;\"> </span><span style=\"background-color: lightgreen;\">a</span><span style=\"background-color: lightgreen;\"> </span><span style=\"background-color: lightgreen;\">r</span><span style=\"background-color: lightgreen;\">e</span><span style=\"background-color: lightgreen;\">v</span><span style=\"background-color: lightgreen;\">i</span><span style=\"background-color: lightgreen;\">e</span><span style=\"background-color: lightgreen;\">w</span><span style=\"background-color: lightgreen;\"> </span><span style=\"background-color: lightgreen;\">c</span><span style=\"background-color: lightgreen;\">a</span><span style=\"background-color: lightgreen;\">n</span><span style=\"background-color: lightgreen;\"> </span><span style=\"background-color: lightgreen;\">d</span><span style=\"background-color: lightgreen;\">r</span><span style=\"background-color: lightgreen;\">a</span><span style=\"background-color: lightgreen;\">w</span><span style=\"background-color: lightgreen;\"> </span><span style=\"background-color: lightgreen;\">f</span><span style=\"background-color: lightgreen;\">r</span><span style=\"background-color: lightgreen;\">o</span><span style=\"background-color: lightgreen;\">m</span><span style=\"background-color: lightgreen;\"> </span><span style=\"background-color: lightgreen;\">p</span><span style=\"background-color: lightgreen;\">e</span><span style=\"background-color: lightgreen;\">r</span><span style=\"background-color: lightgreen;\">s</span><span style=\"background-color: lightgreen;\">o</span><span style=\"background-color: lightgreen;\">n</span><span style=\"background-color: lightgreen;\">a</span><span style=\"background-color: lightgreen;\">l</span><span style=\"background-color: lightgreen;\"> </span><span style=\"background-color: lightgreen;\">e</span><span style=\"background-color: lightgreen;\">x</span><span style=\"background-color: lightgreen;\">p</span><span style=\"background-color: lightgreen;\">e</span><span style=\"background-color: lightgreen;\">r</span><span style=\"background-color: lightgreen;\">i</span><span style=\"background-color: lightgreen;\">e</span><span style=\"background-color: lightgreen;\">n</span><span style=\"background-color: lightgreen;\">c</span><span style=\"background-color: lightgreen;\">e</span><span style=\"background-color: lightgreen;\">s</span><span style=\"background-color: lightgreen;\"> </span><span style=\"background-color: lightgreen;\">y</span><span style=\"background-color: lightgreen;\">e</span><span style=\"background-color: lightgreen;\">t</span><span style=\"background-color: lightgreen;\"> </span><span style=\"background-color: lightgreen;\">s</span><span style=\"background-color: lightgreen;\">t</span><span style=\"background-color: lightgreen;\">i</span><span style=\"background-color: lightgreen;\">l</span><span style=\"background-color: lightgreen;\">l</span><span style=\"background-color: lightgreen;\"> </span><span style=\"background-color: lightgreen;\">r</span><span style=\"background-color: lightgreen;\">e</span><span style=\"background-color: lightgreen;\">m</span><span style=\"background-color: lightgreen;\">a</span><span style=\"background-color: lightgreen;\">i</span><span style=\"background-color: lightgreen;\">n</span><span style=\"background-color: lightgreen;\"> </span><span style=\"background-color: lightgreen;\">'</span><span style=\"background-color: lightgreen;\">O</span><span style=\"background-color: lightgreen;\">b</span><span style=\"background-color: lightgreen;\">j</span><span style=\"background-color: lightgreen;\">e</span><span style=\"background-color: lightgreen;\">c</span><span style=\"background-color: lightgreen;\">t</span><span style=\"background-color: lightgreen;\">i</span><span style=\"background-color: lightgreen;\">v</span><span style=\"background-color: lightgreen;\">e</span><span style=\"background-color: lightgreen;\">'</span><span style=\"background-color: lightgreen;\"> </span><span style=\"background-color: lightgreen;\">i</span><span style=\"background-color: lightgreen;\">f</span><span style=\"background-color: lightgreen;\"> </span><span style=\"background-color: lightgreen;\">i</span><span style=\"background-color: lightgreen;\">t</span><span style=\"background-color: lightgreen;\"> </span><span style=\"background-color: lightgreen;\">i</span><span style=\"background-color: lightgreen;\">m</span><span style=\"background-color: lightgreen;\">p</span><span style=\"background-color: lightgreen;\">a</span><span style=\"background-color: lightgreen;\">r</span><span style=\"background-color: lightgreen;\">t</span><span style=\"background-color: lightgreen;\">s</span><span style=\"background-color: lightgreen;\"> </span><span style=\"background-color: lightgreen;\">i</span><span style=\"background-color: lightgreen;\">n</span><span style=\"background-color: lightgreen;\">f</span><span style=\"background-color: lightgreen;\">o</span><span style=\"background-color: lightgreen;\">r</span><span style=\"background-color: lightgreen;\">m</span><span style=\"background-color: lightgreen;\">a</span><span style=\"background-color: lightgreen;\">t</span><span style=\"background-color: lightgreen;\">i</span><span style=\"background-color: lightgreen;\">o</span><span style=\"background-color: lightgreen;\">n</span><span style=\"background-color: lightgreen;\"> </span><span style=\"background-color: lightgreen;\">a</span><span style=\"background-color: lightgreen;\">b</span><span style=\"background-color: lightgreen;\">o</span><span style=\"background-color: lightgreen;\">u</span><span style=\"background-color: lightgreen;\">t</span><span style=\"background-color: lightgreen;\"> </span><span style=\"background-color: lightgreen;\">t</span><span style=\"background-color: lightgreen;\">h</span><span style=\"background-color: lightgreen;\">e</span><span style=\"background-color: lightgreen;\"> </span><span style=\"background-color: lightgreen;\">p</span><span style=\"background-color: lightgreen;\">r</span><span style=\"background-color: lightgreen;\">o</span><span style=\"background-color: lightgreen;\">d</span><span style=\"background-color: lightgreen;\">u</span><span style=\"background-color: lightgreen;\">c</span><span style=\"background-color: lightgreen;\">t</span><span style=\"background-color: lightgreen;\">'</span><span style=\"background-color: lightgreen;\">s</span><span style=\"background-color: lightgreen;\"> </span><span style=\"background-color: lightgreen;\">p</span><span style=\"background-color: lightgreen;\">e</span><span style=\"background-color: lightgreen;\">r</span><span style=\"background-color: lightgreen;\">f</span><span style=\"background-color: lightgreen;\">o</span><span style=\"background-color: lightgreen;\">r</span><span style=\"background-color: lightgreen;\">m</span><span style=\"background-color: lightgreen;\">a</span><span style=\"background-color: lightgreen;\">n</span><span style=\"background-color: lightgreen;\">c</span><span style=\"background-color: lightgreen;\">e</span><span style=\"background-color: lightgreen;\"> </span><span style=\"background-color: lightgreen;\">o</span><span style=\"background-color: lightgreen;\">r</span><span style=\"background-color: lightgreen;\"> </span><span style=\"background-color: lightgreen;\">f</span><span style=\"background-color: lightgreen;\">u</span><span style=\"background-color: lightgreen;\">n</span><span style=\"background-color: lightgreen;\">c</span><span style=\"background-color: lightgreen;\">t</span><span style=\"background-color: lightgreen;\">i</span><span style=\"background-color: lightgreen;\">o</span><span style=\"background-color: lightgreen;\">n</span><span style=\"background-color: lightgreen;\">a</span><span style=\"background-color: lightgreen;\">l</span><span style=\"background-color: lightgreen;\">i</span><span style=\"background-color: lightgreen;\">t</span><span style=\"background-color: lightgreen;\">y</span><span style=\"background-color: lightgreen;\"> </span><span style=\"background-color: lightgreen;\">w</span><span style=\"background-color: lightgreen;\">i</span><span style=\"background-color: lightgreen;\">t</span><span style=\"background-color: lightgreen;\">h</span><span style=\"background-color: lightgreen;\">o</span><span style=\"background-color: lightgreen;\">u</span><span style=\"background-color: lightgreen;\">t</span><span style=\"background-color: lightgreen;\"> </span><span style=\"background-color: lightgreen;\">d</span><span style=\"background-color: lightgreen;\">e</span><span style=\"background-color: lightgreen;\">c</span><span style=\"background-color: lightgreen;\">l</span><span style=\"background-color: lightgreen;\">a</span><span style=\"background-color: lightgreen;\">r</span>ing personal emotions or preferences.\""
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">\n",
       "\n",
       "=&gt; Iteration #<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2</span>: Getting feedback, analyzing and improving <span style=\"color: #808000; text-decoration-color: #808000\">...</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\n",
       "\n",
       "=> Iteration #\u001b[1;36m2\u001b[0m: Getting feedback, analyzing and improving \u001b[33m...\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">Applying skill: subjectivity_detection\n",
       "</pre>\n"
      ],
      "text/plain": [
       "Applying skill: subjectivity_detection\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|█| 5/5 [00:01<00:00, \n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">Predictions and feedback:\n",
       "</pre>\n"
      ],
      "text/plain": [
       "Predictions and feedback:\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">                                                                                                                   \n",
       " <span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\"> prediction__fb                           </span> <span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\"> text                                     </span> <span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\"> ground_truth </span> <span style=\"color: #800080; text-decoration-color: #800080; font-weight: bold\"> prediction </span> \n",
       " ───────────────────────────────────────────────────────────────────────────────────────────────────────────────── \n",
       "  Prediction is correct.                     The mic is great.                          Subjective     Subjective  \n",
       " <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\"> Prediction is correct.                   </span> <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\"> Will order from them again!              </span> <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\"> Subjective   </span> <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\"> Subjective </span> \n",
       "  Prediction is incorrect. Correct answer:   Not loud enough and doesn't turn on like   Objective      Subjective  \n",
       "  \"Objective\"                                it should.                                                            \n",
       " <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\"> Prediction is correct.                   </span> <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\"> The phone doesn't seem to accept         </span> <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\"> Objective    </span> <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\"> Objective  </span> \n",
       " <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">                                          </span> <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\"> anything except CBR mp3s                 </span> <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">              </span> <span style=\"color: #7f7f7f; text-decoration-color: #7f7f7f\">            </span> \n",
       "  Prediction is correct.                     All three broke within two months of       Objective      Objective   \n",
       "                                             use.                                                                  \n",
       "                                                                                                                   \n",
       "</pre>\n"
      ],
      "text/plain": [
       "                                                                                                                   \n",
       " \u001b[1;35m \u001b[0m\u001b[1;35mprediction__fb                          \u001b[0m\u001b[1;35m \u001b[0m \u001b[1;35m \u001b[0m\u001b[1;35mtext                                    \u001b[0m\u001b[1;35m \u001b[0m \u001b[1;35m \u001b[0m\u001b[1;35mground_truth\u001b[0m\u001b[1;35m \u001b[0m \u001b[1;35m \u001b[0m\u001b[1;35mprediction\u001b[0m\u001b[1;35m \u001b[0m \n",
       " ───────────────────────────────────────────────────────────────────────────────────────────────────────────────── \n",
       "  Prediction is correct.                     The mic is great.                          Subjective     Subjective  \n",
       " \u001b[2m \u001b[0m\u001b[2mPrediction is correct.                  \u001b[0m\u001b[2m \u001b[0m \u001b[2m \u001b[0m\u001b[2mWill order from them again!             \u001b[0m\u001b[2m \u001b[0m \u001b[2m \u001b[0m\u001b[2mSubjective  \u001b[0m\u001b[2m \u001b[0m \u001b[2m \u001b[0m\u001b[2mSubjective\u001b[0m\u001b[2m \u001b[0m \n",
       "  Prediction is incorrect. Correct answer:   Not loud enough and doesn't turn on like   Objective      Subjective  \n",
       "  \"Objective\"                                it should.                                                            \n",
       " \u001b[2m \u001b[0m\u001b[2mPrediction is correct.                  \u001b[0m\u001b[2m \u001b[0m \u001b[2m \u001b[0m\u001b[2mThe phone doesn't seem to accept        \u001b[0m\u001b[2m \u001b[0m \u001b[2m \u001b[0m\u001b[2mObjective   \u001b[0m\u001b[2m \u001b[0m \u001b[2m \u001b[0m\u001b[2mObjective \u001b[0m\u001b[2m \u001b[0m \n",
       " \u001b[2m                                          \u001b[0m \u001b[2m \u001b[0m\u001b[2manything except CBR mp3s                \u001b[0m\u001b[2m \u001b[0m \u001b[2m              \u001b[0m \u001b[2m            \u001b[0m \n",
       "  Prediction is correct.                     All three broke within two months of       Objective      Objective   \n",
       "                                             use.                                                                  \n",
       "                                                                                                                   \n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #800000; text-decoration-color: #800000; font-weight: bold\">Output to improve: </span><span style=\"color: #008000; text-decoration-color: #008000\">\"prediction\"</span><span style=\"color: #800000; text-decoration-color: #800000; font-weight: bold\"> (</span><span style=\"color: #808000; text-decoration-color: #808000; font-weight: bold\">Skill</span><span style=\"color: #800000; text-decoration-color: #800000; font-weight: bold\">=</span><span style=\"color: #008000; text-decoration-color: #008000\">\"subjectivity_detection\"</span><span style=\"color: #800000; text-decoration-color: #800000; font-weight: bold\">)</span>\n",
       "<span style=\"color: #800000; text-decoration-color: #800000; font-weight: bold\">Accuracy = </span><span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">80.00</span><span style=\"color: #800000; text-decoration-color: #800000; font-weight: bold\">%</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[1;31mOutput to improve: \u001b[0m\u001b[32m\"prediction\"\u001b[0m\u001b[1;31m \u001b[0m\u001b[1;31m(\u001b[0m\u001b[1;33mSkill\u001b[0m\u001b[1;31m=\u001b[0m\u001b[32m\"subjectivity_detection\"\u001b[0m\u001b[1;31m)\u001b[0m\n",
       "\u001b[1;31mAccuracy = \u001b[0m\u001b[1;36m80.00\u001b[0m\u001b[1;31m%\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #008000; text-decoration-color: #008000\">\"{system}\"</span>:\n",
       "You are a helpful assistant.\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[32m\"\u001b[0m\u001b[32m{\u001b[0m\u001b[32msystem\u001b[0m\u001b[32m}\u001b[0m\u001b[32m\"\u001b[0m:\n",
       "You are a helpful assistant.\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #008000; text-decoration-color: #008000\">\"{user}\"</span>:\n",
       "\n",
       "A prompt is a text paragraph that outlines the expected actions and instructs the large language model <span style=\"font-weight: bold\">(</span>LLM<span style=\"font-weight: bold\">)</span> to \n",
       "generate a specific output. This prompt is concatenated with the input text, and the model then creates the \n",
       "required output.\n",
       "This describes the full template how the prompt is concatenated with the input to produce the output:\n",
       "\n",
       "```\n",
       "\n",
       "<span style=\"font-weight: bold\">{</span>prompt<span style=\"font-weight: bold\">}</span>\n",
       "Input: <span style=\"font-weight: bold\">{</span>text<span style=\"font-weight: bold\">}</span>\n",
       "Output: <span style=\"font-weight: bold\">{</span>prediction<span style=\"font-weight: bold\">}</span>\n",
       "```\n",
       "\n",
       "Here:\n",
       "- <span style=\"color: #008000; text-decoration-color: #008000\">\"Input: {text}\"</span> is input template,\n",
       "- <span style=\"color: #008000; text-decoration-color: #008000\">\"{prompt}\"</span> is the LLM prompt,\n",
       "- <span style=\"color: #008000; text-decoration-color: #008000\">\"Output: {prediction}\"</span> is the output template.\n",
       "\n",
       "Model can produce erroneous output if a prompt is not well defined. In our collaboration, we’ll work together to \n",
       "refine a prompt. The process consists of two main steps:\n",
       "\n",
       "## Step <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1</span>\n",
       "I will provide you with the current prompt along with prediction examples. Each example contains the input text, \n",
       "the final prediction produced by the model, and the user feedback. User feedback indicates whether the model \n",
       "prediction is correct or not. Your task is to analyze the examples and user feedback, determining whether the \n",
       "existing instruction is describing the task reflected by these examples precisely, and suggests changes to the \n",
       "prompt to address the incorrect predictions.\n",
       "\n",
       "## Step <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2</span>\n",
       "Next, you will carefully review your reasoning in step <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1</span>, integrate the insights to refine the prompt, and provide \n",
       "me with the new prompt that improves the model’s performance.\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[32m\"\u001b[0m\u001b[32m{\u001b[0m\u001b[32muser\u001b[0m\u001b[32m}\u001b[0m\u001b[32m\"\u001b[0m:\n",
       "\n",
       "A prompt is a text paragraph that outlines the expected actions and instructs the large language model \u001b[1m(\u001b[0mLLM\u001b[1m)\u001b[0m to \n",
       "generate a specific output. This prompt is concatenated with the input text, and the model then creates the \n",
       "required output.\n",
       "This describes the full template how the prompt is concatenated with the input to produce the output:\n",
       "\n",
       "```\n",
       "\n",
       "\u001b[1m{\u001b[0mprompt\u001b[1m}\u001b[0m\n",
       "Input: \u001b[1m{\u001b[0mtext\u001b[1m}\u001b[0m\n",
       "Output: \u001b[1m{\u001b[0mprediction\u001b[1m}\u001b[0m\n",
       "```\n",
       "\n",
       "Here:\n",
       "- \u001b[32m\"Input: \u001b[0m\u001b[32m{\u001b[0m\u001b[32mtext\u001b[0m\u001b[32m}\u001b[0m\u001b[32m\"\u001b[0m is input template,\n",
       "- \u001b[32m\"\u001b[0m\u001b[32m{\u001b[0m\u001b[32mprompt\u001b[0m\u001b[32m}\u001b[0m\u001b[32m\"\u001b[0m is the LLM prompt,\n",
       "- \u001b[32m\"Output: \u001b[0m\u001b[32m{\u001b[0m\u001b[32mprediction\u001b[0m\u001b[32m}\u001b[0m\u001b[32m\"\u001b[0m is the output template.\n",
       "\n",
       "Model can produce erroneous output if a prompt is not well defined. In our collaboration, we’ll work together to \n",
       "refine a prompt. The process consists of two main steps:\n",
       "\n",
       "## Step \u001b[1;36m1\u001b[0m\n",
       "I will provide you with the current prompt along with prediction examples. Each example contains the input text, \n",
       "the final prediction produced by the model, and the user feedback. User feedback indicates whether the model \n",
       "prediction is correct or not. Your task is to analyze the examples and user feedback, determining whether the \n",
       "existing instruction is describing the task reflected by these examples precisely, and suggests changes to the \n",
       "prompt to address the incorrect predictions.\n",
       "\n",
       "## Step \u001b[1;36m2\u001b[0m\n",
       "Next, you will carefully review your reasoning in step \u001b[1;36m1\u001b[0m, integrate the insights to refine the prompt, and provide \n",
       "me with the new prompt that improves the model’s performance.\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #008000; text-decoration-color: #008000\">\"{assistant}\"</span>:\n",
       "Sure, I’d be happy to help you with this prompt engineering problem. Please provide me with the current prompt and \n",
       "the examples with user feedback.\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[32m\"\u001b[0m\u001b[32m{\u001b[0m\u001b[32massistant\u001b[0m\u001b[32m}\u001b[0m\u001b[32m\"\u001b[0m:\n",
       "Sure, I’d be happy to help you with this prompt engineering problem. Please provide me with the current prompt and \n",
       "the examples with user feedback.\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #008000; text-decoration-color: #008000\">\"{user}\"</span>:\n",
       "\n",
       "## Current prompt\n",
       "<span style=\"color: #008000; text-decoration-color: #008000\">\"Analyze the provided product review and ascertain whether it predominantly expresses 'Subjective' or 'Objective' </span>\n",
       "<span style=\"color: #008000; text-decoration-color: #008000\">statements. Classify it as 'Subjective' if the review primarily discloses personal feelings, emotions, or </span>\n",
       "<span style=\"color: #008000; text-decoration-color: #008000\">preferences. However, should the review describe factual details about the product’s functionality or performance, </span>\n",
       "<span style=\"color: #008000; text-decoration-color: #008000\">even if these details stem from personal experiences, yet do not assert personal emotions or preferences, classify </span>\n",
       "<span style=\"color: #008000; text-decoration-color: #008000\">it as 'Objective'. Keep in mind that a review can draw from personal experiences yet still remain 'Objective' if it</span>\n",
       "<span style=\"color: #008000; text-decoration-color: #008000\">imparts information about the product's performance or functionality without declaring personal emotions or </span>\n",
       "<span style=\"color: #008000; text-decoration-color: #008000\">preferences.\"</span>\n",
       "\n",
       "## Examples\n",
       "### Example #<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">0</span>\n",
       "\n",
       "Input: The mic is great.\n",
       "\n",
       "Output: Subjective\n",
       "\n",
       "User feedback: Prediction is correct.\n",
       "\n",
       "\n",
       "### Example #<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1</span>\n",
       "\n",
       "Input: Will order from them again!\n",
       "\n",
       "Output: Subjective\n",
       "\n",
       "User feedback: Prediction is correct.\n",
       "\n",
       "\n",
       "### Example #<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2</span>\n",
       "\n",
       "Input: Not loud enough and doesn't turn on like it should.\n",
       "\n",
       "Output: Subjective\n",
       "\n",
       "User feedback: Prediction is incorrect. Correct answer: <span style=\"color: #008000; text-decoration-color: #008000\">\"Objective\"</span>\n",
       "\n",
       "\n",
       "### Example #<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">3</span>\n",
       "\n",
       "Input: The phone doesn't seem to accept anything except CBR mp3s\n",
       "\n",
       "Output: Objective\n",
       "\n",
       "User feedback: Prediction is correct.\n",
       "\n",
       "\n",
       "### Example #<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">4</span>\n",
       "\n",
       "Input: All three broke within two months of use.\n",
       "\n",
       "Output: Objective\n",
       "\n",
       "User feedback: Prediction is correct.\n",
       "\n",
       "\n",
       "\n",
       "Summarize your analysis about incorrect predictions and suggest changes to the prompt.\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[32m\"\u001b[0m\u001b[32m{\u001b[0m\u001b[32muser\u001b[0m\u001b[32m}\u001b[0m\u001b[32m\"\u001b[0m:\n",
       "\n",
       "## Current prompt\n",
       "\u001b[32m\"Analyze the provided product review and ascertain whether it predominantly expresses 'Subjective' or 'Objective' \u001b[0m\n",
       "\u001b[32mstatements. Classify it as 'Subjective' if the review primarily discloses personal feelings, emotions, or \u001b[0m\n",
       "\u001b[32mpreferences. However, should the review describe factual details about the product’s functionality or performance, \u001b[0m\n",
       "\u001b[32meven if these details stem from personal experiences, yet do not assert personal emotions or preferences, classify \u001b[0m\n",
       "\u001b[32mit as 'Objective'. Keep in mind that a review can draw from personal experiences yet still remain 'Objective' if it\u001b[0m\n",
       "\u001b[32mimparts information about the product's performance or functionality without declaring personal emotions or \u001b[0m\n",
       "\u001b[32mpreferences.\"\u001b[0m\n",
       "\n",
       "## Examples\n",
       "### Example #\u001b[1;36m0\u001b[0m\n",
       "\n",
       "Input: The mic is great.\n",
       "\n",
       "Output: Subjective\n",
       "\n",
       "User feedback: Prediction is correct.\n",
       "\n",
       "\n",
       "### Example #\u001b[1;36m1\u001b[0m\n",
       "\n",
       "Input: Will order from them again!\n",
       "\n",
       "Output: Subjective\n",
       "\n",
       "User feedback: Prediction is correct.\n",
       "\n",
       "\n",
       "### Example #\u001b[1;36m2\u001b[0m\n",
       "\n",
       "Input: Not loud enough and doesn't turn on like it should.\n",
       "\n",
       "Output: Subjective\n",
       "\n",
       "User feedback: Prediction is incorrect. Correct answer: \u001b[32m\"Objective\"\u001b[0m\n",
       "\n",
       "\n",
       "### Example #\u001b[1;36m3\u001b[0m\n",
       "\n",
       "Input: The phone doesn't seem to accept anything except CBR mp3s\n",
       "\n",
       "Output: Objective\n",
       "\n",
       "User feedback: Prediction is correct.\n",
       "\n",
       "\n",
       "### Example #\u001b[1;36m4\u001b[0m\n",
       "\n",
       "Input: All three broke within two months of use.\n",
       "\n",
       "Output: Objective\n",
       "\n",
       "User feedback: Prediction is correct.\n",
       "\n",
       "\n",
       "\n",
       "Summarize your analysis about incorrect predictions and suggest changes to the prompt.\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #008000; text-decoration-color: #008000\">\"{assistant}\"</span>:\n",
       "## Analysis \n",
       "\n",
       "Looking at the examples provided, it appears that the model mostly predicts correctly, except for Example #<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2</span>. \n",
       "\n",
       "For Example #<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2</span>, the user feedback suggests that the model's prediction is incorrect. The model categorized it as \n",
       "<span style=\"color: #008000; text-decoration-color: #008000\">'Subjective'</span> when it should be <span style=\"color: #008000; text-decoration-color: #008000\">'Objective'</span>. The user comment <span style=\"color: #008000; text-decoration-color: #008000\">\"Not loud enough and doesn't turn on like it should\"</span> \n",
       "is a factual statement based on the user's experiences with the product. This reflects the fact that the product \n",
       "does not operate as it should, according to its specifications or expected performance, making it an <span style=\"color: #008000; text-decoration-color: #008000\">'Objective'</span> \n",
       "statement.\n",
       "\n",
       "I believe that this discrepancy might be due to the negative aspect of this review. The model could have \n",
       "potentially interpreted the negative sentiment as a sign of subjectivity.\n",
       "\n",
       "## Prompt Suggestion \n",
       "\n",
       "Based on the analysis, we can refine the prompt by adding more clarity around subjective and objective attitudes, \n",
       "both positive and negative. Negative criticism that is based on factual aspects of a product or service should also\n",
       "be seen as <span style=\"color: #008000; text-decoration-color: #008000\">'Objective'</span>. We could possibly add a line emphasizing this in the prompt. \n",
       "\n",
       "Here's a suggestion:\n",
       "\n",
       "<span style=\"color: #008000; text-decoration-color: #008000\">\"Analyze the provided product review and ascertain whether it predominantly expresses 'Subjective' or 'Objective' </span>\n",
       "<span style=\"color: #008000; text-decoration-color: #008000\">statements. Classify it as 'Subjective' if the review primarily discloses personal feelings, emotions, or </span>\n",
       "<span style=\"color: #008000; text-decoration-color: #008000\">preferences, regardless of them being positive or negative. However, should the review describe factual details </span>\n",
       "<span style=\"color: #008000; text-decoration-color: #008000\">about the product’s functionality or performance, even if these details reveal issues or negative aspects of the </span>\n",
       "<span style=\"color: #008000; text-decoration-color: #008000\">product, classify it as 'Objective'. A review can draw from personal experiences yet still remain 'Objective' if it</span>\n",
       "<span style=\"color: #008000; text-decoration-color: #008000\">imparts information about the product's performance or functionality without declaring personal emotions or </span>\n",
       "<span style=\"color: #008000; text-decoration-color: #008000\">preferences. Pay attention to whether the statements hold true irrespective of who is using the product.\"</span>\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[32m\"\u001b[0m\u001b[32m{\u001b[0m\u001b[32massistant\u001b[0m\u001b[32m}\u001b[0m\u001b[32m\"\u001b[0m:\n",
       "## Analysis \n",
       "\n",
       "Looking at the examples provided, it appears that the model mostly predicts correctly, except for Example #\u001b[1;36m2\u001b[0m. \n",
       "\n",
       "For Example #\u001b[1;36m2\u001b[0m, the user feedback suggests that the model's prediction is incorrect. The model categorized it as \n",
       "\u001b[32m'Subjective'\u001b[0m when it should be \u001b[32m'Objective'\u001b[0m. The user comment \u001b[32m\"Not loud enough and doesn't turn on like it should\"\u001b[0m \n",
       "is a factual statement based on the user's experiences with the product. This reflects the fact that the product \n",
       "does not operate as it should, according to its specifications or expected performance, making it an \u001b[32m'Objective'\u001b[0m \n",
       "statement.\n",
       "\n",
       "I believe that this discrepancy might be due to the negative aspect of this review. The model could have \n",
       "potentially interpreted the negative sentiment as a sign of subjectivity.\n",
       "\n",
       "## Prompt Suggestion \n",
       "\n",
       "Based on the analysis, we can refine the prompt by adding more clarity around subjective and objective attitudes, \n",
       "both positive and negative. Negative criticism that is based on factual aspects of a product or service should also\n",
       "be seen as \u001b[32m'Objective'\u001b[0m. We could possibly add a line emphasizing this in the prompt. \n",
       "\n",
       "Here's a suggestion:\n",
       "\n",
       "\u001b[32m\"Analyze the provided product review and ascertain whether it predominantly expresses 'Subjective' or 'Objective' \u001b[0m\n",
       "\u001b[32mstatements. Classify it as 'Subjective' if the review primarily discloses personal feelings, emotions, or \u001b[0m\n",
       "\u001b[32mpreferences, regardless of them being positive or negative. However, should the review describe factual details \u001b[0m\n",
       "\u001b[32mabout the product’s functionality or performance, even if these details reveal issues or negative aspects of the \u001b[0m\n",
       "\u001b[32mproduct, classify it as 'Objective'. A review can draw from personal experiences yet still remain 'Objective' if it\u001b[0m\n",
       "\u001b[32mimparts information about the product's performance or functionality without declaring personal emotions or \u001b[0m\n",
       "\u001b[32mpreferences. Pay attention to whether the statements hold true irrespective of who is using the product.\"\u001b[0m\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #008000; text-decoration-color: #008000\">\"{user}\"</span>:\n",
       "\n",
       "Now please carefully review your reasoning in Step <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1</span> and help with Step <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2</span>: refining the prompt.\n",
       "\n",
       "## Current prompt\n",
       "<span style=\"color: #008000; text-decoration-color: #008000\">\"Analyze the provided product review and ascertain whether it predominantly expresses 'Subjective' or 'Objective' </span>\n",
       "<span style=\"color: #008000; text-decoration-color: #008000\">statements. Classify it as 'Subjective' if the review primarily discloses personal feelings, emotions, or </span>\n",
       "<span style=\"color: #008000; text-decoration-color: #008000\">preferences. However, should the review describe factual details about the product’s functionality or performance, </span>\n",
       "<span style=\"color: #008000; text-decoration-color: #008000\">even if these details stem from personal experiences, yet do not assert personal emotions or preferences, classify </span>\n",
       "<span style=\"color: #008000; text-decoration-color: #008000\">it as 'Objective'. Keep in mind that a review can draw from personal experiences yet still remain 'Objective' if it</span>\n",
       "<span style=\"color: #008000; text-decoration-color: #008000\">imparts information about the product's performance or functionality without declaring personal emotions or </span>\n",
       "<span style=\"color: #008000; text-decoration-color: #008000\">preferences.\"</span>\n",
       "\n",
       "## Follow this guidance to refine the prompt:\n",
       "\n",
       "<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1</span>. The new prompt should should describe the task precisely, and address the points raised in the user feedback.\n",
       "    \n",
       "<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">2</span>. The new prompt should be similar to the current instruction, and only differ in the parts that address the \n",
       "issues you identified in Step <span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">1</span>.\n",
       "    Example:\n",
       "    - Current prompt: <span style=\"color: #008000; text-decoration-color: #008000\">\"The model should generate a summary of the input text.\"</span>\n",
       "    - New prompt: <span style=\"color: #008000; text-decoration-color: #008000\">\"The model should generate a summary of the input text. Pay attention to the original style.\"</span>\n",
       "        \n",
       "<span style=\"color: #008080; text-decoration-color: #008080; font-weight: bold\">3</span>. Reply only with the new prompt. Do not include input and output templates in the prompt.\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[32m\"\u001b[0m\u001b[32m{\u001b[0m\u001b[32muser\u001b[0m\u001b[32m}\u001b[0m\u001b[32m\"\u001b[0m:\n",
       "\n",
       "Now please carefully review your reasoning in Step \u001b[1;36m1\u001b[0m and help with Step \u001b[1;36m2\u001b[0m: refining the prompt.\n",
       "\n",
       "## Current prompt\n",
       "\u001b[32m\"Analyze the provided product review and ascertain whether it predominantly expresses 'Subjective' or 'Objective' \u001b[0m\n",
       "\u001b[32mstatements. Classify it as 'Subjective' if the review primarily discloses personal feelings, emotions, or \u001b[0m\n",
       "\u001b[32mpreferences. However, should the review describe factual details about the product’s functionality or performance, \u001b[0m\n",
       "\u001b[32meven if these details stem from personal experiences, yet do not assert personal emotions or preferences, classify \u001b[0m\n",
       "\u001b[32mit as 'Objective'. Keep in mind that a review can draw from personal experiences yet still remain 'Objective' if it\u001b[0m\n",
       "\u001b[32mimparts information about the product's performance or functionality without declaring personal emotions or \u001b[0m\n",
       "\u001b[32mpreferences.\"\u001b[0m\n",
       "\n",
       "## Follow this guidance to refine the prompt:\n",
       "\n",
       "\u001b[1;36m1\u001b[0m. The new prompt should should describe the task precisely, and address the points raised in the user feedback.\n",
       "    \n",
       "\u001b[1;36m2\u001b[0m. The new prompt should be similar to the current instruction, and only differ in the parts that address the \n",
       "issues you identified in Step \u001b[1;36m1\u001b[0m.\n",
       "    Example:\n",
       "    - Current prompt: \u001b[32m\"The model should generate a summary of the input text.\"\u001b[0m\n",
       "    - New prompt: \u001b[32m\"The model should generate a summary of the input text. Pay attention to the original style.\"\u001b[0m\n",
       "        \n",
       "\u001b[1;36m3\u001b[0m. Reply only with the new prompt. Do not include input and output templates in the prompt.\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "\"Analyze the provided product review and c<span style=\"background-color: lightgreen;\">l</span>a<span style=\"background-color: lightgreen;\">s</span><span style=\"background-color: lightgreen;\">s</span>i<span style=\"background-color: lightgreen;\">f</span><span style=\"background-color: lightgreen;\">y</span> <span style=\"background-color: lightgreen;\">i</span><span style=\"background-color: lightgreen;\">t</span><span style=\"background-color: lightgreen;\"> </span><span style=\"background-color: lightgreen;\">a</span><span style=\"background-color: lightgreen;\">s</span><span style=\"background-color: lightgreen;\"> </span><span style=\"background-color: lightgreen;\">'</span><span style=\"background-color: lightgreen;\">S</span><span style=\"background-color: lightgreen;\">u</span><span style=\"background-color: lightgreen;\">b</span><span style=\"background-color: lightgreen;\">j</span><span style=\"background-color: lightgreen;\">e</span><span style=\"background-color: lightgreen;\">c</span><span style=\"background-color: lightgreen;\">t</span><span style=\"background-color: lightgreen;\">i</span><span style=\"background-color: lightgreen;\">v</span><span style=\"background-color: lightgreen;\">e</span><span style=\"background-color: lightgreen;\">'</span><span style=\"background-color: lightgreen;\"> </span><span style=\"background-color: lightgreen;\">o</span><span style=\"background-color: lightgreen;\">r</span><span style=\"background-color: lightgreen;\"> </span><span style=\"background-color: lightgreen;\">'</span><span style=\"background-color: lightgreen;\">O</span><span style=\"background-color: lightgreen;\">b</span><span style=\"background-color: lightgreen;\">j</span><span style=\"background-color: lightgreen;\">e</span><span style=\"background-color: lightgreen;\">c</span><span style=\"background-color: lightgreen;\">t</span><span style=\"background-color: lightgreen;\">i</span><span style=\"background-color: lightgreen;\">v</span><span style=\"background-color: lightgreen;\">e</span><span style=\"background-color: lightgreen;\">'</span><span style=\"background-color: lightgreen;\">.</span><span style=\"background-color: lightgreen;\"> </span><span style=\"background-color: lightgreen;\">I</span><span style=\"background-color: lightgreen;\">f</span><span style=\"background-color: lightgreen;\"> </span><span style=\"background-color: lightgreen;\">t</span><span style=\"background-color: lightgreen;\">h</span><span style=\"background-color: lightgreen;\">e</span><span style=\"background-color: lightgreen;\"> </span><span style=\"background-color: lightgreen;\">r</span><span style=\"background-color: lightgreen;\">e</span><span style=\"background-color: lightgreen;\">v</span><span style=\"background-color: lightgreen;\">i</span><span style=\"background-color: lightgreen;\">e</span>w pri<span style=\"background-color: lightgreen;\">m</span>a<span style=\"background-color: lightgreen;\">r</span><span style=\"background-color: lightgreen;\">i</span>ly<span style=\"background-color: lightgreen;\"> </span><span style=\"background-color: lightgreen;\">s</span><span style=\"background-color: lightgreen;\">h</span><span style=\"background-color: lightgreen;\">a</span><span style=\"background-color: lightgreen;\">r</span><span style=\"background-color: lightgreen;\">e</span><span style=\"background-color: lightgreen;\">s</span><span style=\"background-color: lightgreen;\"> </span><span style=\"background-color: lightgreen;\">p</span><span style=\"background-color: lightgreen;\">e</span><span style=\"background-color: lightgreen;\">r</span><span style=\"background-color: lightgreen;\">s</span><span style=\"background-color: lightgreen;\">o</span><span style=\"background-color: lightgreen;\">n</span><span style=\"background-color: lightgreen;\">a</span><span style=\"background-color: lightgreen;\">l</span><span style=\"background-color: lightgreen;\"> </span><span style=\"background-color: lightgreen;\">f</span><span style=\"background-color: lightgreen;\">e</span><span style=\"background-color: lightgreen;\">e</span><span style=\"background-color: lightgreen;\">l</span><span style=\"background-color: lightgreen;\">i</span><span style=\"background-color: lightgreen;\">n</span><span style=\"background-color: lightgreen;\">g</span><span style=\"background-color: lightgreen;\">s</span><span style=\"background-color: lightgreen;\">,</span><span style=\"background-color: lightgreen;\"> </span><span style=\"background-color: lightgreen;\">e</span><span style=\"background-color: lightgreen;\">m</span><span style=\"background-color: lightgreen;\">o</span><span style=\"background-color: lightgreen;\">t</span><span style=\"background-color: lightgreen;\">i</span><span style=\"background-color: lightgreen;\">o</span><span style=\"background-color: lightgreen;\">n</span><span style=\"background-color: lightgreen;\">s</span><span style=\"background-color: lightgreen;\">,</span><span style=\"background-color: lightgreen;\"> </span><span style=\"background-color: lightgreen;\">o</span><span style=\"background-color: lightgreen;\">r</span><span style=\"background-color: lightgreen;\"> </span><span style=\"background-color: lightgreen;\">p</span><span style=\"background-color: lightgreen;\">r</span><span style=\"background-color: lightgreen;\">e</span><span style=\"background-color: lightgreen;\">f</span><span style=\"background-color: lightgreen;\">e</span><span style=\"background-color: lightgreen;\">r</span><span style=\"background-color: lightgreen;\">e</span><span style=\"background-color: lightgreen;\">n</span><span style=\"background-color: lightgreen;\">c</span><span style=\"background-color: lightgreen;\">e</span><span style=\"background-color: lightgreen;\">s</span><span style=\"background-color: lightgreen;\">,</span><span style=\"background-color: lightgreen;\"> </span><span style=\"background-color: lightgreen;\">l</span><span style=\"background-color: lightgreen;\">a</span><span style=\"background-color: lightgreen;\">b</span><span style=\"background-color: lightgreen;\">e</span><span style=\"background-color: lightgreen;\">l</span><span style=\"background-color: lightgreen;\"> </span><span style=\"background-color: lightgreen;\">i</span><span style=\"background-color: lightgreen;\">t</span><span style=\"background-color: lightgreen;\"> </span><span style=\"background-color: lightgreen;\">a</span><span style=\"background-color: lightgreen;\">s</span><span style=\"background-color: lightgreen;\"> </span><span style=\"background-color: lightgreen;\">'</span><span style=\"background-color: lightgreen;\">S</span><span style=\"background-color: lightgreen;\">u</span><span style=\"background-color: lightgreen;\">b</span><span style=\"background-color: lightgreen;\">j</span><span style=\"background-color: lightgreen;\">e</span><span style=\"background-color: lightgreen;\">c</span><span style=\"background-color: lightgreen;\">t</span><span style=\"background-color: lightgreen;\">i</span><span style=\"background-color: lightgreen;\">v</span><span style=\"background-color: lightgreen;\">e</span><span style=\"background-color: lightgreen;\">'</span><span style=\"background-color: lightgreen;\">.</span><span style=\"background-color: lightgreen;\"> </span><span style=\"background-color: lightgreen;\">S</span><span style=\"background-color: lightgreen;\">u</span><span style=\"background-color: lightgreen;\">b</span><span style=\"background-color: lightgreen;\">j</span><span style=\"background-color: lightgreen;\">e</span><span style=\"background-color: lightgreen;\">c</span><span style=\"background-color: lightgreen;\">t</span><span style=\"background-color: lightgreen;\">i</span><span style=\"background-color: lightgreen;\">v</span><span style=\"background-color: lightgreen;\">i</span><span style=\"background-color: lightgreen;\">t</span><span style=\"background-color: lightgreen;\">y</span><span style=\"background-color: lightgreen;\"> </span><span style=\"background-color: lightgreen;\">m</span><span style=\"background-color: lightgreen;\">a</span><span style=\"background-color: lightgreen;\">y</span><span style=\"background-color: lightgreen;\"> </span><span style=\"background-color: lightgreen;\">b</span><span style=\"background-color: lightgreen;\">e</span><span style=\"background-color: lightgreen;\"> </span><span style=\"background-color: lightgreen;\">p</span><span style=\"background-color: lightgreen;\">r</span><span style=\"background-color: lightgreen;\">e</span><span style=\"background-color: lightgreen;\">s</span><span style=\"background-color: lightgreen;\">e</span><span style=\"background-color: lightgreen;\">n</span><span style=\"background-color: lightgreen;\">t</span><span style=\"background-color: lightgreen;\">e</span><span style=\"background-color: lightgreen;\">d</span><span style=\"background-color: lightgreen;\"> </span><span style=\"background-color: lightgreen;\">t</span><span style=\"background-color: lightgreen;\">h</span><span style=\"background-color: lightgreen;\">r</span><span style=\"background-color: lightgreen;\">o</span><span style=\"background-color: lightgreen;\">u</span><span style=\"background-color: lightgreen;\">g</span><span style=\"background-color: lightgreen;\">h</span><span style=\"background-color: lightgreen;\"> </span><span style=\"background-color: lightgreen;\">b</span><span style=\"background-color: lightgreen;\">o</span><span style=\"background-color: lightgreen;\">t</span><span style=\"background-color: lightgreen;\">h</span><span style=\"background-color: lightgreen;\"> </span><span style=\"background-color: lightgreen;\">p</span><span style=\"background-color: lightgreen;\">o</span><span style=\"background-color: lightgreen;\">s</span><span style=\"background-color: lightgreen;\">i</span><span style=\"background-color: lightgreen;\">t</span><span style=\"background-color: lightgreen;\">i</span><span style=\"background-color: lightgreen;\">v</span><span style=\"background-color: lightgreen;\">e</span><span style=\"background-color: lightgreen;\"> </span><span style=\"background-color: lightgreen;\">a</span><span style=\"background-color: lightgreen;\">n</span><span style=\"background-color: lightgreen;\">d</span><span style=\"background-color: lightgreen;\"> </span><span style=\"background-color: lightgreen;\">n</span><span style=\"background-color: lightgreen;\">e</span><span style=\"background-color: lightgreen;\">g</span><span style=\"background-color: lightgreen;\">a</span><span style=\"background-color: lightgreen;\">t</span><span style=\"background-color: lightgreen;\">i</span><span style=\"background-color: lightgreen;\">v</span><span style=\"background-color: lightgreen;\">e</span><span style=\"background-color: lightgreen;\"> </span><span style=\"background-color: lightgreen;\">a</span><span style=\"background-color: lightgreen;\">t</span><span style=\"background-color: lightgreen;\">t</span><span style=\"background-color: lightgreen;\">i</span><span style=\"background-color: lightgreen;\">t</span><span style=\"background-color: lightgreen;\">u</span><span style=\"background-color: lightgreen;\">d</span><span style=\"background-color: lightgreen;\">e</span><span style=\"background-color: lightgreen;\">s</span><span style=\"background-color: lightgreen;\">.</span><span style=\"background-color: lightgreen;\"> </span><span style=\"background-color: lightgreen;\">I</span><span style=\"background-color: lightgreen;\">f</span><span style=\"background-color: lightgreen;\"> </span><span style=\"background-color: lightgreen;\">t</span><span style=\"background-color: lightgreen;\">h</span><span style=\"background-color: lightgreen;\">e</span><span style=\"background-color: lightgreen;\"> </span><span style=\"background-color: lightgreen;\">r</span><span style=\"background-color: lightgreen;\">e</span><span style=\"background-color: lightgreen;\">v</span><span style=\"background-color: lightgreen;\">i</span><span style=\"background-color: lightgreen;\">e</span><span style=\"background-color: lightgreen;\">w</span><span style=\"background-color: lightgreen;\">,</span><span style=\"background-color: lightgreen;\"> </span><span style=\"background-color: lightgreen;\">o</span><span style=\"background-color: lightgreen;\">n</span><span style=\"background-color: lightgreen;\"> </span><span style=\"background-color: lightgreen;\">t</span><span style=\"background-color: lightgreen;\">h</span><span style=\"background-color: lightgreen;\">e</span><span style=\"background-color: lightgreen;\"> </span><span style=\"background-color: lightgreen;\">o</span><span style=\"background-color: lightgreen;\">t</span><span style=\"background-color: lightgreen;\">h</span><span style=\"background-color: lightgreen;\">e</span><span style=\"background-color: lightgreen;\">r</span><span style=\"background-color: lightgreen;\"> </span><span style=\"background-color: lightgreen;\">h</span><span style=\"background-color: lightgreen;\">a</span><span style=\"background-color: lightgreen;\">n</span><span style=\"background-color: lightgreen;\">d</span><span style=\"background-color: lightgreen;\">,</span><span style=\"background-color: lightgreen;\"> </span><span style=\"background-color: lightgreen;\">c</span><span style=\"background-color: lightgreen;\">o</span><span style=\"background-color: lightgreen;\">n</span><span style=\"background-color: lightgreen;\">v</span><span style=\"background-color: lightgreen;\">e</span><span style=\"background-color: lightgreen;\">y</span><span style=\"background-color: lightgreen;\">s</span><span style=\"background-color: lightgreen;\"> </span><span style=\"background-color: lightgreen;\">f</span><span style=\"background-color: lightgreen;\">a</span><span style=\"background-color: lightgreen;\">c</span><span style=\"background-color: lightgreen;\">t</span><span style=\"background-color: lightgreen;\">u</span><span style=\"background-color: lightgreen;\">a</span><span style=\"background-color: lightgreen;\">l</span><span style=\"background-color: lightgreen;\"> </span><span style=\"background-color: lightgreen;\">d</span><span style=\"background-color: lightgreen;\">e</span><span style=\"background-color: lightgreen;\">t</span><span style=\"background-color: lightgreen;\">a</span><span style=\"background-color: lightgreen;\">i</span><span style=\"background-color: lightgreen;\">l</span><span style=\"background-color: lightgreen;\">s</span><span style=\"background-color: lightgreen;\"> </span><span style=\"background-color: lightgreen;\">o</span><span style=\"background-color: lightgreen;\">r</span><span style=\"background-color: lightgreen;\"> </span><span style=\"background-color: lightgreen;\">h</span><span style=\"background-color: lightgreen;\">i</span><span style=\"background-color: lightgreen;\">g</span><span style=\"background-color: lightgreen;\">h</span><span style=\"background-color: lightgreen;\">l</span><span style=\"background-color: lightgreen;\">i</span><span style=\"background-color: lightgreen;\">g</span><span style=\"background-color: lightgreen;\">h</span><span style=\"background-color: lightgreen;\">t</span><span style=\"background-color: lightgreen;\">s</span><span style=\"background-color: lightgreen;\"> </span><span style=\"background-color: lightgreen;\">p</span><span style=\"background-color: lightgreen;\">o</span><span style=\"background-color: lightgreen;\">s</span><span style=\"background-color: lightgreen;\">i</span><span style=\"background-color: lightgreen;\">t</span><span style=\"background-color: lightgreen;\">i</span><span style=\"background-color: lightgreen;\">v</span><span style=\"background-color: lightgreen;\">e</span><span style=\"background-color: lightgreen;\"> </span><span style=\"background-color: lightgreen;\">o</span><span style=\"background-color: lightgreen;\">r</span><span style=\"background-color: lightgreen;\"> </span><span style=\"background-color: lightgreen;\">n</span><span style=\"background-color: lightgreen;\">e</span><span style=\"background-color: lightgreen;\">g</span><span style=\"background-color: lightgreen;\">a</span><span style=\"background-color: lightgreen;\">t</span><span style=\"background-color: lightgreen;\">i</span><span style=\"background-color: lightgreen;\">v</span><span style=\"background-color: lightgreen;\">e</span><span style=\"background-color: lightgreen;\"> </span><span style=\"background-color: lightgreen;\">a</span><span style=\"background-color: lightgreen;\">t</span><span style=\"background-color: lightgreen;\">t</span><span style=\"background-color: lightgreen;\">r</span><span style=\"background-color: lightgreen;\">i</span><span style=\"background-color: lightgreen;\">b</span><span style=\"background-color: lightgreen;\">u</span><span style=\"background-color: lightgreen;\">t</span><span style=\"background-color: lightgreen;\">e</span><span style=\"background-color: lightgreen;\">s</span><span style=\"background-color: lightgreen;\"> </span><span style=\"background-color: lightgreen;\">a</span><span style=\"background-color: lightgreen;\">b</span><span style=\"background-color: lightgreen;\">o</span><span style=\"background-color: lightgreen;\">u</span><span style=\"background-color: lightgreen;\">t</span><span style=\"background-color: lightgreen;\"> </span><span style=\"background-color: lightgreen;\">t</span><span style=\"background-color: lightgreen;\">h</span><span style=\"background-color: lightgreen;\">e</span><span style=\"background-color: lightgreen;\"> </span><span style=\"background-color: lightgreen;\">p</span><span style=\"background-color: lightgreen;\">r</span><span style=\"background-color: lightgreen;\">o</span><span style=\"background-color: lightgreen;\">d</span><span style=\"background-color: lightgreen;\">u</span><span style=\"background-color: lightgreen;\">c</span><span style=\"background-color: lightgreen;\">t</span><span style=\"background-color: lightgreen;\">'</span><span style=\"background-color: lightgreen;\">s</span><span style=\"background-color: lightgreen;\"> </span><span style=\"background-color: lightgreen;\">f</span><span style=\"background-color: lightgreen;\">u</span><span style=\"background-color: lightgreen;\">n</span><span style=\"background-color: lightgreen;\">c</span><span style=\"background-color: lightgreen;\">t</span><span style=\"background-color: lightgreen;\">i</span><span style=\"background-color: lightgreen;\">o</span><span style=\"background-color: lightgreen;\">n</span><span style=\"background-color: lightgreen;\">a</span><span style=\"background-color: lightgreen;\">l</span><span style=\"background-color: lightgreen;\">i</span><span style=\"background-color: lightgreen;\">t</span><span style=\"background-color: lightgreen;\">y</span><span style=\"background-color: lightgreen;\"> </span><span style=\"background-color: lightgreen;\">o</span><span style=\"background-color: lightgreen;\">r</span><span style=\"background-color: lightgreen;\"> </span><span style=\"background-color: lightgreen;\">p</span><span style=\"background-color: lightgreen;\">e</span><span style=\"background-color: lightgreen;\">r</span><span style=\"background-color: lightgreen;\">f</span><span style=\"background-color: lightgreen;\">o</span><span style=\"background-color: lightgreen;\">r</span><span style=\"background-color: lightgreen;\">m</span><span style=\"background-color: lightgreen;\">a</span><span style=\"background-color: lightgreen;\">n</span><span style=\"background-color: lightgreen;\">c</span><span style=\"background-color: lightgreen;\">e</span><span style=\"background-color: lightgreen;\">,</span><span style=\"background-color: lightgreen;\"> </span><span style=\"background-color: lightgreen;\">c</span><span style=\"background-color: lightgreen;\">l</span><span style=\"background-color: lightgreen;\">a</span><span style=\"background-color: lightgreen;\">s</span><span style=\"background-color: lightgreen;\">s</span><span style=\"background-color: lightgreen;\">i</span><span style=\"background-color: lightgreen;\">f</span><span style=\"background-color: lightgreen;\">y</span><span style=\"background-color: lightgreen;\"> </span><span style=\"background-color: lightgreen;\">i</span><span style=\"background-color: lightgreen;\">t</span><span style=\"background-color: lightgreen;\"> </span><span style=\"background-color: lightgreen;\">a</span><span style=\"background-color: lightgreen;\">s</span><span style=\"background-color: lightgreen;\"> </span><span style=\"background-color: lightgreen;\">'</span><span style=\"background-color: lightgreen;\">O</span><span style=\"background-color: lightgreen;\">b</span><span style=\"background-color: lightgreen;\">j</span><span style=\"background-color: lightgreen;\">e</span><span style=\"background-color: lightgreen;\">c</span><span style=\"background-color: lightgreen;\">t</span><span style=\"background-color: lightgreen;\">i</span><span style=\"background-color: lightgreen;\">v</span><span style=\"background-color: lightgreen;\">e</span><span style=\"background-color: lightgreen;\">'</span><span style=\"background-color: lightgreen;\">.</span><span style=\"background-color: lightgreen;\"> </span><span style=\"background-color: lightgreen;\">T</span><span style=\"background-color: lightgreen;\">h</span><span style=\"background-color: lightgreen;\">e</span><span style=\"background-color: lightgreen;\">s</span><span style=\"background-color: lightgreen;\">e</span><span style=\"background-color: lightgreen;\"> </span><span style=\"background-color: lightgreen;\">f</span><span style=\"background-color: lightgreen;\">a</span><span style=\"background-color: lightgreen;\">c</span><span style=\"background-color: lightgreen;\">t</span><span style=\"background-color: lightgreen;\">u</span><span style=\"background-color: lightgreen;\">a</span><span style=\"background-color: lightgreen;\">l</span><span style=\"background-color: lightgreen;\"> </span><span style=\"background-color: lightgreen;\">d</span><span style=\"background-color: lightgreen;\">e</span><span style=\"background-color: lightgreen;\">t</span><span style=\"background-color: lightgreen;\">a</span><span style=\"background-color: lightgreen;\">i</span><span style=\"background-color: lightgreen;\">l</span><span style=\"background-color: lightgreen;\">s</span><span style=\"background-color: lightgreen;\"> </span><span style=\"background-color: lightgreen;\">m</span><span style=\"background-color: lightgreen;\">a</span><span style=\"background-color: lightgreen;\">y</span><span style=\"background-color: lightgreen;\"> </span><span style=\"background-color: lightgreen;\">s</span><span style=\"background-color: lightgreen;\">t</span><span style=\"background-color: lightgreen;\">e</span><span style=\"background-color: lightgreen;\">m</span><span style=\"background-color: lightgreen;\"> </span><span style=\"background-color: lightgreen;\">f</span><span style=\"background-color: lightgreen;\">r</span><span style=\"background-color: lightgreen;\">o</span><span style=\"background-color: lightgreen;\">m</span><span style=\"background-color: lightgreen;\"> </span><span style=\"background-color: lightgreen;\">p</span><span style=\"background-color: lightgreen;\">e</span><span style=\"background-color: lightgreen;\">r</span><span style=\"background-color: lightgreen;\">s</span><span style=\"background-color: lightgreen;\">o</span><span style=\"background-color: lightgreen;\">n</span><span style=\"background-color: lightgreen;\">a</span><span style=\"background-color: lightgreen;\">l</span> expe<span style=\"background-color: lightgreen;\">r</span><span style=\"background-color: lightgreen;\">i</span>e<span style=\"background-color: lightgreen;\">n</span><span style=\"background-color: lightgreen;\">c</span><span style=\"background-color: lightgreen;\">e</span>s <span style=\"background-color: lightgreen;\">b</span>u<span style=\"background-color: lightgreen;\">t</span><span style=\"background-color: lightgreen;\"> </span><span style=\"background-color: lightgreen;\">d</span><span style=\"background-color: lightgreen;\">o</span><span style=\"background-color: lightgreen;\"> </span><span style=\"background-color: lightgreen;\">n</span><span style=\"background-color: lightgreen;\">o</span><span style=\"background-color: lightgreen;\">t</span><span style=\"background-color: lightgreen;\"> </span><span style=\"background-color: lightgreen;\">e</span><span style=\"background-color: lightgreen;\">x</span><span style=\"background-color: lightgreen;\">p</span><span style=\"background-color: lightgreen;\">r</span><span style=\"background-color: lightgreen;\">e</span><span style=\"background-color: lightgreen;\">s</span><span style=\"background-color: lightgreen;\">s</span><span style=\"background-color: lightgreen;\"> </span><span style=\"background-color: lightgreen;\">p</span><span style=\"background-color: lightgreen;\">e</span><span style=\"background-color: lightgreen;\">r</span><span style=\"background-color: lightgreen;\">s</span><span style=\"background-color: lightgreen;\">o</span><span style=\"background-color: lightgreen;\">n</span><span style=\"background-color: lightgreen;\">a</span><span style=\"background-color: lightgreen;\">l</span><span style=\"background-color: lightgreen;\"> </span><span style=\"background-color: lightgreen;\">e</span><span style=\"background-color: lightgreen;\">m</span><span style=\"background-color: lightgreen;\">o</span><span style=\"background-color: lightgreen;\">t</span><span style=\"background-color: lightgreen;\">i</span><span style=\"background-color: lightgreen;\">o</span><span style=\"background-color: lightgreen;\">n</span><span style=\"background-color: lightgreen;\">s</span><span style=\"background-color: lightgreen;\"> </span><span style=\"background-color: lightgreen;\">o</span><span style=\"background-color: lightgreen;\">r</span><span style=\"background-color: lightgreen;\"> </span><span style=\"background-color: lightgreen;\">p</span><span style=\"background-color: lightgreen;\">r</span><span style=\"background-color: lightgreen;\">e</span><span style=\"background-color: lightgreen;\">f</span><span style=\"background-color: lightgreen;\">e</span><span style=\"background-color: lightgreen;\">r</span><span style=\"background-color: lightgreen;\">e</span><span style=\"background-color: lightgreen;\">n</span><span style=\"background-color: lightgreen;\">c</span><span style=\"background-color: lightgreen;\">e</span><span style=\"background-color: lightgreen;\">s</span><span style=\"background-color: lightgreen;\">.</span><span style=\"background-color: lightgreen;\"> </span><span style=\"background-color: lightgreen;\">R</span><span style=\"background-color: lightgreen;\">e</span><span style=\"background-color: lightgreen;\">m</span><span style=\"background-color: lightgreen;\">e</span><span style=\"background-color: lightgreen;\">m</span><span style=\"background-color: lightgreen;\">b</span><span style=\"background-color: lightgreen;\">e</span><span style=\"background-color: lightgreen;\">r</span><span style=\"background-color: lightgreen;\">,</span><span style=\"background-color: lightgreen;\"> </span><span style=\"background-color: lightgreen;\">a</span><span style=\"background-color: lightgreen;\"> </span><span style=\"background-color: lightgreen;\">r</span><span style=\"background-color: lightgreen;\">e</span><span style=\"background-color: lightgreen;\">v</span><span style=\"background-color: lightgreen;\">i</span><span style=\"background-color: lightgreen;\">e</span><span style=\"background-color: lightgreen;\">w</span><span style=\"background-color: lightgreen;\"> </span><span style=\"background-color: lightgreen;\">m</span><span style=\"background-color: lightgreen;\">a</span><span style=\"background-color: lightgreen;\">y</span><span style=\"background-color: lightgreen;\"> </span><span style=\"background-color: lightgreen;\">c</span><span style=\"background-color: lightgreen;\">o</span><span style=\"background-color: lightgreen;\">n</span><span style=\"background-color: lightgreen;\">t</span><span style=\"background-color: lightgreen;\">a</span><span style=\"background-color: lightgreen;\">i</span><span style=\"background-color: lightgreen;\">n</span><span style=\"background-color: lightgreen;\"> </span><span style=\"background-color: lightgreen;\">p</span><span style=\"background-color: lightgreen;\">e</span><span style=\"background-color: lightgreen;\">r</span><span style=\"background-color: lightgreen;\">s</span><span style=\"background-color: lightgreen;\">o</span><span style=\"background-color: lightgreen;\">n</span><span style=\"background-color: lightgreen;\">a</span><span style=\"background-color: lightgreen;\">l</span><span style=\"background-color: lightgreen;\"> </span><span style=\"background-color: lightgreen;\">e</span><span style=\"background-color: lightgreen;\">x</span><span style=\"background-color: lightgreen;\">p</span><span style=\"background-color: lightgreen;\">e</span><span style=\"background-color: lightgreen;\">r</span><span style=\"background-color: lightgreen;\">i</span><span style=\"background-color: lightgreen;\">e</span><span style=\"background-color: lightgreen;\">n</span><span style=\"background-color: lightgreen;\">c</span><span style=\"background-color: lightgreen;\">e</span><span style=\"background-color: lightgreen;\">s</span><span style=\"background-color: lightgreen;\"> </span><span style=\"background-color: lightgreen;\">b</span><span style=\"background-color: lightgreen;\">u</span><span style=\"background-color: lightgreen;\">t</span><span style=\"background-color: lightgreen;\"> </span><span style=\"background-color: lightgreen;\">r</span><span style=\"background-color: lightgreen;\">e</span><span style=\"background-color: lightgreen;\">m</span><span style=\"background-color: lightgreen;\">a</span><span style=\"background-color: lightgreen;\">i</span><span style=\"background-color: lightgreen;\">n</span><span style=\"background-color: lightgreen;\"> </span><span style=\"background-color: lightgreen;\">'</span><span style=\"background-color: lightgreen;\">O</span>bjective' <span style=\"background-color: lightgreen;\">i</span><span style=\"background-color: lightgreen;\">f</span> <span style=\"background-color: lightgreen;\">i</span><span style=\"background-color: lightgreen;\">t</span><span style=\"background-color: lightgreen;\"> </span><span style=\"background-color: lightgreen;\">i</span><span style=\"background-color: lightgreen;\">m</span><span style=\"background-color: lightgreen;\">p</span><span style=\"background-color: lightgreen;\">a</span><span style=\"background-color: lightgreen;\">r</span><span style=\"background-color: lightgreen;\">t</span><span style=\"background-color: lightgreen;\">s</span><span style=\"background-color: lightgreen;\"> </span><span style=\"background-color: lightgreen;\">i</span><span style=\"background-color: lightgreen;\">n</span><span style=\"background-color: lightgreen;\">f</span><span style=\"background-color: lightgreen;\">o</span><span style=\"background-color: lightgreen;\">r</span><span style=\"background-color: lightgreen;\">m</span><span style=\"background-color: lightgreen;\">a</span><span style=\"background-color: lightgreen;\">t</span><span style=\"background-color: lightgreen;\">i</span><span style=\"background-color: lightgreen;\">o</span><span style=\"background-color: lightgreen;\">n</span><span style=\"background-color: lightgreen;\"> </span><span style=\"background-color: lightgreen;\">a</span><span style=\"background-color: lightgreen;\">b</span><span style=\"background-color: lightgreen;\">o</span><span style=\"background-color: lightgreen;\">u</span><span style=\"background-color: lightgreen;\">t</span><span style=\"background-color: lightgreen;\"> </span><span style=\"background-color: lightgreen;\">t</span><span style=\"background-color: lightgreen;\">h</span><span style=\"background-color: lightgreen;\">e</span><span style=\"background-color: lightgreen;\"> </span><span style=\"background-color: lightgreen;\">p</span><span style=\"background-color: lightgreen;\">r</span><span style=\"background-color: lightgreen;\">o</span><span style=\"background-color: lightgreen;\">d</span><span style=\"background-color: lightgreen;\">u</span><span style=\"background-color: lightgreen;\">c</span><span style=\"background-color: lightgreen;\">t</span>'<span style=\"background-color: lightgreen;\">s</span><span style=\"background-color: lightgreen;\"> </span><span style=\"background-color: lightgreen;\">p</span><span style=\"background-color: lightgreen;\">e</span><span style=\"background-color: lightgreen;\">r</span><span style=\"background-color: lightgreen;\">f</span><span style=\"background-color: lightgreen;\">o</span><span style=\"background-color: lightgreen;\">r</span><span style=\"background-color: lightgreen;\">m</span><span style=\"background-color: lightgreen;\">a</span><span style=\"background-color: lightgreen;\">n</span><span style=\"background-color: lightgreen;\">c</span><span style=\"background-color: lightgreen;\">e</span><span style=\"background-color: lightgreen;\"> </span><span style=\"background-color: lightgreen;\">o</span><span style=\"background-color: lightgreen;\">r</span><span style=\"background-color: lightgreen;\"> </span><span style=\"background-color: lightgreen;\">f</span><span style=\"background-color: lightgreen;\">u</span><span style=\"background-color: lightgreen;\">n</span><span style=\"background-color: lightgreen;\">c</span><span style=\"background-color: lightgreen;\">t</span><span style=\"background-color: lightgreen;\">i</span><span style=\"background-color: lightgreen;\">o</span><span style=\"background-color: lightgreen;\">n</span><span style=\"background-color: lightgreen;\">a</span><span style=\"background-color: lightgreen;\">l</span><span style=\"background-color: lightgreen;\">i</span><span style=\"background-color: lightgreen;\">t</span><span style=\"background-color: lightgreen;\">y</span><span style=\"background-color: lightgreen;\"> </span><span style=\"background-color: lightgreen;\">w</span><span style=\"background-color: lightgreen;\">i</span><span style=\"background-color: lightgreen;\">t</span><span style=\"background-color: lightgreen;\">h</span><span style=\"background-color: lightgreen;\">o</span><span style=\"background-color: lightgreen;\">u</span><span style=\"background-color: lightgreen;\">t</span><span style=\"background-color: lightgreen;\"> </span><span style=\"background-color: lightgreen;\">e</span><span style=\"background-color: lightgreen;\">x</span><span style=\"background-color: lightgreen;\">p</span><span style=\"background-color: lightgreen;\">r</span><span style=\"background-color: lightgreen;\">e</span><span style=\"background-color: lightgreen;\">s</span><span style=\"background-color: lightgreen;\">s</span><span style=\"background-color: lightgreen;\">i</span><span style=\"background-color: lightgreen;\">n</span><span style=\"background-color: lightgreen;\">g</span><span style=\"background-color: lightgreen;\"> </span><span style=\"background-color: lightgreen;\">p</span><span style=\"background-color: lightgreen;\">e</span><span style=\"background-color: lightgreen;\">r</span><span style=\"background-color: lightgreen;\">s</span><span style=\"background-color: lightgreen;\">o</span><span style=\"background-color: lightgreen;\">n</span><span style=\"background-color: lightgreen;\">a</span><span style=\"background-color: lightgreen;\">l</span><span style=\"background-color: lightgreen;\"> </span><span style=\"background-color: lightgreen;\">f</span><span style=\"background-color: lightgreen;\">e</span><span style=\"background-color: lightgreen;\">e</span><span style=\"background-color: lightgreen;\">l</span><span style=\"background-color: lightgreen;\">i</span><span style=\"background-color: lightgreen;\">n</span><span style=\"background-color: lightgreen;\">g</span><span style=\"background-color: lightgreen;\">s</span><span style=\"background-color: lightgreen;\">,</span><span style=\"background-color: lightgreen;\"> </span><span style=\"background-color: lightgreen;\">p</span><span style=\"background-color: lightgreen;\">r</span><span style=\"background-color: lightgreen;\">e</span><span style=\"background-color: lightgreen;\">f</span><span style=\"background-color: lightgreen;\">e</span><span style=\"background-color: lightgreen;\">r</span><span style=\"background-color: lightgreen;\">e</span><span style=\"background-color: lightgreen;\">n</span><span style=\"background-color: lightgreen;\">c</span><span style=\"background-color: lightgreen;\">e</span><span style=\"background-color: lightgreen;\">s</span><span style=\"background-color: lightgreen;\">,</span><span style=\"background-color: lightgreen;\"> </span><span style=\"background-color: lightgreen;\">o</span><span style=\"background-color: lightgreen;\">r</span><span style=\"background-color: lightgreen;\"> </span><span style=\"background-color: lightgreen;\">a</span><span style=\"background-color: lightgreen;\">s</span><span style=\"background-color: lightgreen;\">s</span><span style=\"background-color: lightgreen;\">e</span><span style=\"background-color: lightgreen;\">r</span><span style=\"background-color: lightgreen;\">t</span><span style=\"background-color: lightgreen;\">i</span><span style=\"background-color: lightgreen;\">o</span><span style=\"background-color: lightgreen;\">n</span><span style=\"background-color: lightgreen;\">s</span><span style=\"background-color: lightgreen;\">.</span><span style=\"background-color: lightgreen;\"> </span>Objective reviews presen<span style=\"background-color: lightgreen;\">t</span> s<span style=\"background-color: lightgreen;\">t</span><span style=\"background-color: lightgreen;\">a</span><span style=\"background-color: lightgreen;\">t</span><span style=\"background-color: lightgreen;\">e</span><span style=\"background-color: lightgreen;\">m</span><span style=\"background-color: lightgreen;\">e</span><span style=\"background-color: lightgreen;\">n</span><span style=\"background-color: lightgreen;\">t</span><span style=\"background-color: lightgreen;\">s</span> <span style=\"background-color: lightgreen;\">t</span><span style=\"background-color: lightgreen;\">h</span><span style=\"background-color: lightgreen;\">a</span><span style=\"background-color: lightgreen;\">t</span><span style=\"background-color: lightgreen;\"> </span>would be gener<span style=\"background-color: lightgreen;\">a</span><span style=\"background-color: lightgreen;\">l</span><span style=\"background-color: lightgreen;\">l</span><span style=\"background-color: lightgreen;\">y</span> <span style=\"background-color: lightgreen;\">t</span>r<span style=\"background-color: lightgreen;\">u</span>e<span style=\"background-color: lightgreen;\">,</span><span style=\"background-color: lightgreen;\"> </span>re<span style=\"background-color: lightgreen;\">g</span><span style=\"background-color: lightgreen;\">a</span><span style=\"background-color: lightgreen;\">r</span><span style=\"background-color: lightgreen;\">d</span><span style=\"background-color: lightgreen;\">l</span>es<span style=\"background-color: lightgreen;\">s</span><span style=\"background-color: lightgreen;\"> </span><span style=\"background-color: lightgreen;\">o</span><span style=\"background-color: lightgreen;\">f</span><span style=\"background-color: lightgreen;\"> </span><span style=\"background-color: lightgreen;\">t</span><span style=\"background-color: lightgreen;\">h</span><span style=\"background-color: lightgreen;\">e</span><span style=\"background-color: lightgreen;\"> </span><span style=\"background-color: lightgreen;\">u</span><span style=\"background-color: lightgreen;\">s</span><span style=\"background-color: lightgreen;\">e</span><span style=\"background-color: lightgreen;\">r</span>.\""
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">Train is done!\n",
       "</pre>\n"
      ],
      "text/plain": [
       "Train is done!\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "ground_truth_signal = agent.learn(learning_iterations=3, accuracy_threshold=0.95)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's see the final instructions:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\"><span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\">Total Agent Skills: </span><span style=\"color: #000080; text-decoration-color: #000080; font-weight: bold\">1</span>\n",
       "\n",
       "<span style=\"color: #008000; text-decoration-color: #008000; font-weight: bold; text-decoration: underline\">subjectivity_detection</span>\n",
       "<span style=\"color: #008000; text-decoration-color: #008000\">\"Analyze the provided product review and classify it as 'Subjective' or 'Objective'. If the review primarily shares</span>\n",
       "<span style=\"color: #008000; text-decoration-color: #008000\">personal feelings, emotions, or preferences, label it as 'Subjective'. Subjectivity may be presented through both </span>\n",
       "<span style=\"color: #008000; text-decoration-color: #008000\">positive and negative attitudes. If the review, on the other hand, conveys factual details or highlights positive </span>\n",
       "<span style=\"color: #008000; text-decoration-color: #008000\">or negative attributes about the product's functionality or performance, classify it as 'Objective'. These factual </span>\n",
       "<span style=\"color: #008000; text-decoration-color: #008000\">details may stem from personal experiences but do not express personal emotions or preferences. Remember, a review </span>\n",
       "<span style=\"color: #008000; text-decoration-color: #008000\">may contain personal experiences but remain 'Objective' if it imparts information about the product's performance </span>\n",
       "<span style=\"color: #008000; text-decoration-color: #008000\">or functionality without expressing personal feelings, preferences, or assertions. Objective reviews present </span>\n",
       "<span style=\"color: #008000; text-decoration-color: #008000\">statements that would be generally true, regardless of the user.\"</span>\n",
       "\n",
       "</pre>\n"
      ],
      "text/plain": [
       "\u001b[1;34mTotal Agent Skills: \u001b[0m\u001b[1;34m1\u001b[0m\n",
       "\n",
       "\u001b[1;4;32msubjectivity_detection\u001b[0m\n",
       "\u001b[32m\"Analyze the provided product review and classify it as 'Subjective' or 'Objective'. If the review primarily shares\u001b[0m\n",
       "\u001b[32mpersonal feelings, emotions, or preferences, label it as 'Subjective'. Subjectivity may be presented through both \u001b[0m\n",
       "\u001b[32mpositive and negative attitudes. If the review, on the other hand, conveys factual details or highlights positive \u001b[0m\n",
       "\u001b[32mor negative attributes about the product's functionality or performance, classify it as 'Objective'. These factual \u001b[0m\n",
       "\u001b[32mdetails may stem from personal experiences but do not express personal emotions or preferences. Remember, a review \u001b[0m\n",
       "\u001b[32mmay contain personal experiences but remain 'Objective' if it imparts information about the product's performance \u001b[0m\n",
       "\u001b[32mor functionality without expressing personal feelings, preferences, or assertions. Objective reviews present \u001b[0m\n",
       "\u001b[32mstatements that would be generally true, regardless of the user.\"\u001b[0m\n",
       "\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "print(agent.skills)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "... and predictions created by the skill:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">Applying skill: subjectivity_detection\n",
       "</pre>\n"
      ],
      "text/plain": [
       "Applying skill: subjectivity_detection\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|█| 5/5 [00:00<00:00, \n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>text</th>\n",
       "      <th>ground_truth</th>\n",
       "      <th>prediction</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>The mic is great.</td>\n",
       "      <td>Subjective</td>\n",
       "      <td>Subjective</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Will order from them again!</td>\n",
       "      <td>Subjective</td>\n",
       "      <td>Subjective</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Not loud enough and doesn't turn on like it sh...</td>\n",
       "      <td>Objective</td>\n",
       "      <td>Subjective</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>The phone doesn't seem to accept anything exce...</td>\n",
       "      <td>Objective</td>\n",
       "      <td>Objective</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>All three broke within two months of use.</td>\n",
       "      <td>Objective</td>\n",
       "      <td>Objective</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                text ground_truth  prediction\n",
       "0                                  The mic is great.   Subjective  Subjective\n",
       "1                        Will order from them again!   Subjective  Subjective\n",
       "2  Not loud enough and doesn't turn on like it sh...    Objective  Subjective\n",
       "3  The phone doesn't seem to accept anything exce...    Objective   Objective\n",
       "4          All three broke within two months of use.    Objective   Objective"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "agent.run()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Applying learned skills to the real data\n",
    "\n",
    "Now as we have our Agent with evolved \"subjectivity detection\" skill, we can apply it to the real dataset without ground truth data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>text</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Doesn't hold charge.</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Excellent bluetooth headset</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>I love this thing!</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>VERY DISAPPOINTED.</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                          text\n",
       "0         Doesn't hold charge.\n",
       "1  Excellent bluetooth headset\n",
       "2           I love this thing!\n",
       "3           VERY DISAPPOINTED."
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_df = pd.DataFrame([\n",
    "    \"Doesn't hold charge.\",\n",
    "    \"Excellent bluetooth headset\",\n",
    "    \"I love this thing!\",\n",
    "    \"VERY DISAPPOINTED.\"\n",
    "], columns=['text'])\n",
    "test_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<pre style=\"white-space:pre;overflow-x:auto;line-height:normal;font-family:Menlo,'DejaVu Sans Mono',consolas,'Courier New',monospace\">Applying skill: subjectivity_detection\n",
       "</pre>\n"
      ],
      "text/plain": [
       "Applying skill: subjectivity_detection\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|█| 4/4 [00:01<00:00, \n"
     ]
    }
   ],
   "source": [
    "predictions = agent.run(test_df)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>text</th>\n",
       "      <th>prediction</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Doesn't hold charge.</td>\n",
       "      <td>Objective</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Excellent bluetooth headset</td>\n",
       "      <td>Objective</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>I love this thing!</td>\n",
       "      <td>Subjective</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>VERY DISAPPOINTED.</td>\n",
       "      <td>Subjective</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                          text  prediction\n",
       "0         Doesn't hold charge.   Objective\n",
       "1  Excellent bluetooth headset   Objective\n",
       "2           I love this thing!  Subjective\n",
       "3           VERY DISAPPOINTED.  Subjective"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "predictions"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "adala",
   "language": "python",
   "name": "adala"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}