fix Maximum input token count exceeds limit aws-samples#224

.github/ISSUE_TEMPLATE/bug_report.md

@@ -0,0 +1,44 @@
+---
+name: Bug report
+about: Create a report to help us improve
+title: Amazon Bedrock Workshop - [Bug]
+labels: bug
+assignees: ''
+
+---
+
+**Describe the bug**
+A clear and concise description of what the bug is. This could be anything from:
+
+1. A typo or text changes that make the lab better.... to
+2. minor annoying code that still works.. to
+3. breaking changes
+
+**To Reproduce**
+Steps to reproduce the behavior (or some version of this):
+1. Go to '...'
+2. Click on '....'
+3. Scroll down to '....'
+4. See error
+
+**Expected behavior**
+A clear and concise description of what you expected to happen.
+
+**Screenshots**
+If applicable, add screenshots to help explain your problem.
+
+**Desktop (please complete the following information):**
+ - OS: [e.g. iOS]
+ - Browser [e.g. chrome, safari]
+ - Version [e.g. 22]
+
+**Smartphone (please complete the following information):**
+ - Device: [e.g. iPhone6]
+ - OS: [e.g. iOS8.1]
+ - Browser [e.g. stock browser, safari]
+ - Version [e.g. 22]
+
+**! Additional context !**
+- Sagemaker Studio Kernel used
+- Instance used
+- commit link to the workshop if you're not using the latest

00_Prerequisites/bedrock_basics.ipynb

@@ -33,6 +33,7 @@
    "execution_count": null,
    "id": "108c611c-7246-45c4-9f1e-76888b5076eb",
    "metadata": {
+    "scrolled": true,
     "tags": []
    },
    "outputs": [],
@@ -66,7 +67,7 @@
   },
   {
    "cell_type": "code",
-   "execution_count": null,
+   "execution_count": 3,
    "id": "ae2b2a05-78a9-40ca-9b5e-121030f9ede1",
    "metadata": {
     "tags": []
@@ -94,14 +95,85 @@
   },
   {
    "cell_type": "code",
-   "execution_count": null,
+   "execution_count": 4,
    "id": "f67b4466-12ff-4975-9811-7a19c6206604",
    "metadata": {
     "tags": []
    },
-   "outputs": [],
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "['amazon.titan-tg1-large',\n",
+       " 'amazon.titan-image-generator-v1:0',\n",
+       " 'amazon.titan-image-generator-v1',\n",
+       " 'amazon.titan-text-premier-v1:0',\n",
+       " 'amazon.titan-embed-g1-text-02',\n",
+       " 'amazon.titan-text-lite-v1:0:4k',\n",
+       " 'amazon.titan-text-lite-v1',\n",
+       " 'amazon.titan-text-express-v1:0:8k',\n",
+       " 'amazon.titan-text-express-v1',\n",
+       " 'amazon.titan-embed-text-v1:2:8k',\n",
+       " 'amazon.titan-embed-text-v1',\n",
+       " 'amazon.titan-embed-text-v2:0:8k',\n",
+       " 'amazon.titan-embed-text-v2:0',\n",
+       " 'amazon.titan-embed-image-v1:0',\n",
+       " 'amazon.titan-embed-image-v1',\n",
+       " 'stability.stable-diffusion-xl-v1:0',\n",
+       " 'stability.stable-diffusion-xl-v1',\n",
+       " 'ai21.j2-grande-instruct',\n",
+       " 'ai21.j2-jumbo-instruct',\n",
+       " 'ai21.j2-mid',\n",
+       " 'ai21.j2-mid-v1',\n",
+       " 'ai21.j2-ultra',\n",
+       " 'ai21.j2-ultra-v1:0:8k',\n",
+       " 'ai21.j2-ultra-v1',\n",
+       " 'anthropic.claude-instant-v1:2:100k',\n",
+       " 'anthropic.claude-instant-v1',\n",
+       " 'anthropic.claude-v2:0:18k',\n",
+       " 'anthropic.claude-v2:0:100k',\n",
+       " 'anthropic.claude-v2:1:18k',\n",
+       " 'anthropic.claude-v2:1:200k',\n",
+       " 'anthropic.claude-v2:1',\n",
+       " 'anthropic.claude-v2',\n",
+       " 'anthropic.claude-3-sonnet-20240229-v1:0:28k',\n",
+       " 'anthropic.claude-3-sonnet-20240229-v1:0:200k',\n",
+       " 'anthropic.claude-3-sonnet-20240229-v1:0',\n",
+       " 'anthropic.claude-3-haiku-20240307-v1:0:48k',\n",
+       " 'anthropic.claude-3-haiku-20240307-v1:0:200k',\n",
+       " 'anthropic.claude-3-haiku-20240307-v1:0',\n",
+       " 'cohere.command-text-v14:7:4k',\n",
+       " 'cohere.command-text-v14',\n",
+       " 'cohere.command-r-v1:0',\n",
+       " 'cohere.command-r-plus-v1:0',\n",
+       " 'cohere.command-light-text-v14:7:4k',\n",
+       " 'cohere.command-light-text-v14',\n",
+       " 'cohere.embed-english-v3:0:512',\n",
+       " 'cohere.embed-english-v3',\n",
+       " 'cohere.embed-multilingual-v3:0:512',\n",
+       " 'cohere.embed-multilingual-v3',\n",
+       " 'meta.llama2-13b-chat-v1:0:4k',\n",
+       " 'meta.llama2-13b-chat-v1',\n",
+       " 'meta.llama2-70b-chat-v1:0:4k',\n",
+       " 'meta.llama2-70b-chat-v1',\n",
+       " 'meta.llama2-13b-v1:0:4k',\n",
+       " 'meta.llama2-13b-v1',\n",
+       " 'meta.llama2-70b-v1:0:4k',\n",
+       " 'meta.llama2-70b-v1',\n",
+       " 'meta.llama3-8b-instruct-v1:0',\n",
+       " 'meta.llama3-70b-instruct-v1:0',\n",
+       " 'mistral.mistral-7b-instruct-v0:2',\n",
+       " 'mistral.mixtral-8x7b-instruct-v0:1',\n",
+       " 'mistral.mistral-large-2402-v1:0']"
+      ]
+     },
+     "execution_count": 4,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
    "source": [
-    "boto3_bedrock.list_foundation_models()\n"
+    "[models['modelId'] for models in boto3_bedrock.list_foundation_models()['modelSummaries']]\n"
    ]
   },
   {
@@ -128,7 +200,7 @@
     "tags": []
    },
    "source": [
-    "### Amazon Titan Large\n",
+    "### Amazon Titan Large and Premier\n",
     "\n",
     "#### Input\n",
     "```json\n",
@@ -326,7 +398,7 @@
     "tags": []
    },
    "source": [
-    "### Amazon Titan Large"
+    "### Amazon Titan Text Premier"
    ]
   },
   {
@@ -365,7 +437,7 @@
     "try:\n",
     "\n",
     "    body = json.dumps({\"inputText\": prompt_data, \"textGenerationConfig\" : {\"topP\":0.95, \"temperature\":0.2}})\n",
-    "    modelId = \"amazon.titan-tg1-large\"\n",
+    "    modelId = \"amazon.titan-tg1-large\" # \n",
     "    accept = \"application/json\"\n",
     "    contentType = \"application/json\"\n",
     "\n",
@@ -522,7 +594,9 @@
    "cell_type": "code",
    "execution_count": null,
    "id": "45072848-000a-4c22-8f08-2647e5c2230e",
-   "metadata": {},
+   "metadata": {
+    "tags": []
+   },
    "outputs": [],
    "source": [
     "import base64\n",
@@ -579,15 +653,8 @@
     "            chunk = event.get('chunk')\n",
     "            if chunk:\n",
     "                chunk_obj = json.loads(chunk.get('bytes').decode())\n",
-    "                if 'outputText' in chunk_obj:\n",
-    "                    text = chunk_obj.get('outputText', None)\n",
-    "                    print(text,end='')\n",
-    "                    if not text :\n",
-    "                        break\n",
-    "                    #text = chunk_obj['outputText']\n",
-    "                    clear_output(wait=True)\n",
-    "                    output.append(text)\n",
-    "                    display_markdown(Markdown(''.join(output)))\n",
+    "                text = chunk_obj['outputText']\n",
+    "                display_markdown(Markdown(print(text, end='')))\n",
     "\n",
     "except botocore.exceptions.ClientError as error:\n",
     "\n",
@@ -636,7 +703,7 @@
    "source": [
     "messages_API_body = {\n",
     "    \"anthropic_version\": \"bedrock-2023-05-31\", \n",
-    "    \"max_tokens\": 512,\n",
+    "    \"max_tokens\": int(500/0.75),\n",
     "    \"messages\": [\n",
     "        {\n",
     "            \"role\": \"user\",\n",
@@ -664,7 +731,7 @@
     "from IPython.display import clear_output, display, display_markdown, Markdown\n",
     "\n",
     "body = json.dumps(messages_API_body)\n",
-    "modelId = \"anthropic.claude-v2\"  # (Change this to try different model versions)\n",
+    "modelId = \"anthropic.claude-instant-v1\"  # (Change this to try different model versions)\n",
     "accept = \"application/json\"\n",
     "contentType = \"application/json\"\n",
     "\n",
@@ -774,7 +841,7 @@
    "outputs": [],
    "source": [
     "body = json.dumps({\"inputText\": prompt_data})\n",
-    "modelId = \"amazon.titan-embed-text-v1\"  # (Change this to try different embedding models)\n",
+    "modelId = \"amazon.titan-embed-g1-text-02\"  # (Change this to try different embedding models)\n",
     "accept = \"application/json\"\n",
     "contentType = \"application/json\"\n",
     "\n",

01_Text_generation/00_text_generation_w_bedrock.ipynb

@@ -105,7 +105,7 @@
    "id": "cc9784e5-5e9d-472d-8ef1-34108ee4968b",
    "metadata": {},
    "source": [
-    "Let's start by using the Amazon Titan Large model. Amazon Titan Large supports a context window of ~4k tokens and accepts the following parameters:\n",
+    "Let's start by using the Amazon Titan Large model. The Amazon Titan family of models support a large context window of up to 32k tokens and accepts the following parameters:\n",
     "- `inputText`: Prompt to the LLM\n",
     "- `textGenerationConfig`: These are the parameters that model will take into account while generating the output."
    ]
@@ -119,15 +119,7 @@
    },
    "outputs": [],
    "source": [
-    "body = json.dumps({\n",
-    "    \"inputText\": prompt_data, \n",
-    "    \"textGenerationConfig\":{\n",
-    "        \"maxTokenCount\":4096,\n",
-    "        \"stopSequences\":[],\n",
-    "        \"temperature\":0,\n",
-    "        \"topP\":0.9\n",
-    "        }\n",
-    "    }) "
+    "body = json.dumps({\"inputText\": prompt_data, \"textGenerationConfig\" : {\"topP\":0.95, \"temperature\":0.1}})"
    ]
   },
   {
@@ -149,7 +141,7 @@
    "id": "088cf6bf-dd73-4710-a0cc-6c11d220c431",
    "metadata": {},
    "source": [
-    "#### Invoke the Amazon Titan Large language model"
+    "#### Invoke the Amazon Titan Text language model"
    ]
   },
   {
@@ -171,7 +163,8 @@
    },
    "outputs": [],
    "source": [
-    "modelId = 'amazon.titan-tg1-large' # change this to use a different version from the model provider\n",
+    "# modelId = 'amazon.titan-text-premier-v1:0' # Make sure Titan text premier is available in the account you are doing this workhsop in before uncommenting!\n",
+    "modelId = \"amazon.titan-tg1-large\"\n",
     "accept = 'application/json'\n",
     "contentType = 'application/json'\n",
     "outputText = \"\\n\"\n",
@@ -221,11 +214,20 @@
     "Bedrock also supports that the output can be streamed as it is generated by the model in form of chunks. Below is an example of invoking the model with streaming option. `invoke_model_with_response_stream` returns a `ResponseStream` which you can read from."
    ]
   },
+  {
+   "cell_type": "markdown",
+   "id": "cc9ff846-9db5-44cd-997c-598f9ff62bee",
+   "metadata": {},
+   "source": [
+    "_You may want to enable scrolling on your output cell below:_ "
+   ]
+  },
   {
    "cell_type": "code",
    "execution_count": null,
    "id": "ad073290",
    "metadata": {
+    "scrolled": true,
     "tags": []
    },
    "outputs": [],
@@ -867,6 +869,36 @@
     "memoryGiB": 1152,
     "name": "ml.p4de.24xlarge",
     "vcpuNum": 96
+   },
+   {
+    "_defaultOrder": 57,
+    "_isFastLaunch": false,
+    "category": "Accelerated computing",
+    "gpuNum": 0,
+    "hideHardwareSpecs": false,
+    "memoryGiB": 32,
+    "name": "ml.trn1.2xlarge",
+    "vcpuNum": 8
+   },
+   {
+    "_defaultOrder": 58,
+    "_isFastLaunch": false,
+    "category": "Accelerated computing",
+    "gpuNum": 0,
+    "hideHardwareSpecs": false,
+    "memoryGiB": 512,
+    "name": "ml.trn1.32xlarge",
+    "vcpuNum": 128
+   },
+   {
+    "_defaultOrder": 59,
+    "_isFastLaunch": false,
+    "category": "Accelerated computing",
+    "gpuNum": 0,
+    "hideHardwareSpecs": false,
+    "memoryGiB": 512,
+    "name": "ml.trn1n.32xlarge",
+    "vcpuNum": 128
    }
   ],
   "instance_type": "ml.t3.medium",