1_llama_parse_upload_s3.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "64be232e-ae9d-487d-b27a-1321554e21ba",
   "metadata": {},
   "source": [
    "# 1 - Ingest with LlamaParse into S3 for KB"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d255267b-253c-4965-998f-c38a3a168339",
   "metadata": {},
   "source": [
    "In this notebook, we use LlamaParse to pre-process complex documents and stage them in S3 for Amazon Bedrock Knowledge Base (KB)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "24ff9ba5-4187-4eb5-bc05-20a39bb3cd2b",
   "metadata": {},
   "source": [
    "### LlamaParse is a document parser optimized for RAG over complex documents\n",
    "- ✅ Extracts tables / charts\n",
    "- ✅ Input natural language parsing instructions\n",
    "- ✅ JSON mode\n",
    "- ✅ Image Extraction\n",
    "- ✅ Support for ~10+ document types (.pdf, .pptx, .docx, .xml)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "445f5647-56b4-40a2-81d8-6c2e50bb4a83",
   "metadata": {},
   "source": [
    "### Installation"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c173cb8e-fe1e-4c43-8a08-f93e8e733c21",
   "metadata": {},
   "source": [
    "Install llama-index (core framework) and llama-parse (LlamaParse client)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "id": "e126c2e2-2da7-4777-9a7f-43862063ba69",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Requirement already satisfied: llama-index in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (0.10.30)\n",
      "Requirement already satisfied: llama-index-agent-openai<0.3.0,>=0.1.4 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index) (0.2.2)\n",
      "Requirement already satisfied: llama-index-cli<0.2.0,>=0.1.2 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index) (0.1.12)\n",
      "Requirement already satisfied: llama-index-core<0.11.0,>=0.10.30 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index) (0.10.30)\n",
      "Requirement already satisfied: llama-index-embeddings-openai<0.2.0,>=0.1.5 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index) (0.1.7)\n",
      "Requirement already satisfied: llama-index-indices-managed-llama-cloud<0.2.0,>=0.1.2 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index) (0.1.5)\n",
      "Requirement already satisfied: llama-index-legacy<0.10.0,>=0.9.48 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index) (0.9.48)\n",
      "Requirement already satisfied: llama-index-llms-openai<0.2.0,>=0.1.13 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index) (0.1.15)\n",
      "Requirement already satisfied: llama-index-multi-modal-llms-openai<0.2.0,>=0.1.3 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index) (0.1.5)\n",
      "Requirement already satisfied: llama-index-program-openai<0.2.0,>=0.1.3 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index) (0.1.5)\n",
      "Requirement already satisfied: llama-index-question-gen-openai<0.2.0,>=0.1.2 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index) (0.1.3)\n",
      "Requirement already satisfied: llama-index-readers-file<0.2.0,>=0.1.4 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index) (0.1.19)\n",
      "Requirement already satisfied: llama-index-readers-llama-parse<0.2.0,>=0.1.2 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index) (0.1.4)\n",
      "Requirement already satisfied: openai>=1.14.0 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index-agent-openai<0.3.0,>=0.1.4->llama-index) (1.17.0)\n",
      "Requirement already satisfied: PyYAML>=6.0.1 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index-core<0.11.0,>=0.10.30->llama-index) (6.0.1)\n",
      "Requirement already satisfied: SQLAlchemy[asyncio]>=1.4.49 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index-core<0.11.0,>=0.10.30->llama-index) (2.0.29)\n",
      "Requirement already satisfied: aiohttp<4.0.0,>=3.8.6 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index-core<0.11.0,>=0.10.30->llama-index) (3.9.4)\n",
      "Requirement already satisfied: dataclasses-json in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index-core<0.11.0,>=0.10.30->llama-index) (0.6.4)\n",
      "Requirement already satisfied: deprecated>=1.2.9.3 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index-core<0.11.0,>=0.10.30->llama-index) (1.2.14)\n",
      "Requirement already satisfied: dirtyjson<2.0.0,>=1.0.8 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index-core<0.11.0,>=0.10.30->llama-index) (1.0.8)\n",
      "Requirement already satisfied: fsspec>=2023.5.0 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index-core<0.11.0,>=0.10.30->llama-index) (2024.3.1)\n",
      "Requirement already satisfied: httpx in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index-core<0.11.0,>=0.10.30->llama-index) (0.27.0)\n",
      "Requirement already satisfied: llamaindex-py-client<0.2.0,>=0.1.18 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index-core<0.11.0,>=0.10.30->llama-index) (0.1.18)\n",
      "Requirement already satisfied: nest-asyncio<2.0.0,>=1.5.8 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index-core<0.11.0,>=0.10.30->llama-index) (1.6.0)\n",
      "Requirement already satisfied: networkx>=3.0 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index-core<0.11.0,>=0.10.30->llama-index) (3.3)\n",
      "Requirement already satisfied: nltk<4.0.0,>=3.8.1 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index-core<0.11.0,>=0.10.30->llama-index) (3.8.1)\n",
      "Requirement already satisfied: numpy in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index-core<0.11.0,>=0.10.30->llama-index) (1.26.4)\n",
      "Requirement already satisfied: pandas in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index-core<0.11.0,>=0.10.30->llama-index) (2.2.2)\n",
      "Requirement already satisfied: pillow>=9.0.0 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index-core<0.11.0,>=0.10.30->llama-index) (10.3.0)\n",
      "Requirement already satisfied: requests>=2.31.0 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index-core<0.11.0,>=0.10.30->llama-index) (2.31.0)\n",
      "Requirement already satisfied: tenacity<9.0.0,>=8.2.0 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index-core<0.11.0,>=0.10.30->llama-index) (8.2.3)\n",
      "Requirement already satisfied: tiktoken>=0.3.3 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index-core<0.11.0,>=0.10.30->llama-index) (0.6.0)\n",
      "Requirement already satisfied: tqdm<5.0.0,>=4.66.1 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index-core<0.11.0,>=0.10.30->llama-index) (4.66.2)\n",
      "Requirement already satisfied: typing-extensions>=4.5.0 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index-core<0.11.0,>=0.10.30->llama-index) (4.11.0)\n",
      "Requirement already satisfied: typing-inspect>=0.8.0 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index-core<0.11.0,>=0.10.30->llama-index) (0.9.0)\n",
      "Requirement already satisfied: wrapt in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index-core<0.11.0,>=0.10.30->llama-index) (1.16.0)\n",
      "Requirement already satisfied: beautifulsoup4<5.0.0,>=4.12.3 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index-readers-file<0.2.0,>=0.1.4->llama-index) (4.12.3)\n",
      "Requirement already satisfied: pypdf<5.0.0,>=4.0.1 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index-readers-file<0.2.0,>=0.1.4->llama-index) (4.2.0)\n",
      "Requirement already satisfied: striprtf<0.0.27,>=0.0.26 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index-readers-file<0.2.0,>=0.1.4->llama-index) (0.0.26)\n",
      "Requirement already satisfied: llama-parse<0.5.0,>=0.4.0 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index-readers-llama-parse<0.2.0,>=0.1.2->llama-index) (0.4.0)\n",
      "Requirement already satisfied: aiosignal>=1.1.2 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from aiohttp<4.0.0,>=3.8.6->llama-index-core<0.11.0,>=0.10.30->llama-index) (1.3.1)\n",
      "Requirement already satisfied: attrs>=17.3.0 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from aiohttp<4.0.0,>=3.8.6->llama-index-core<0.11.0,>=0.10.30->llama-index) (23.2.0)\n",
      "Requirement already satisfied: frozenlist>=1.1.1 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from aiohttp<4.0.0,>=3.8.6->llama-index-core<0.11.0,>=0.10.30->llama-index) (1.4.1)\n",
      "Requirement already satisfied: multidict<7.0,>=4.5 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from aiohttp<4.0.0,>=3.8.6->llama-index-core<0.11.0,>=0.10.30->llama-index) (6.0.5)\n",
      "Requirement already satisfied: yarl<2.0,>=1.0 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from aiohttp<4.0.0,>=3.8.6->llama-index-core<0.11.0,>=0.10.30->llama-index) (1.9.4)\n",
      "Requirement already satisfied: soupsieve>1.2 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from beautifulsoup4<5.0.0,>=4.12.3->llama-index-readers-file<0.2.0,>=0.1.4->llama-index) (2.5)\n",
      "Requirement already satisfied: pydantic>=1.10 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llamaindex-py-client<0.2.0,>=0.1.18->llama-index-core<0.11.0,>=0.10.30->llama-index) (2.7.0)\n",
      "Requirement already satisfied: anyio in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from httpx->llama-index-core<0.11.0,>=0.10.30->llama-index) (4.3.0)\n",
      "Requirement already satisfied: certifi in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from httpx->llama-index-core<0.11.0,>=0.10.30->llama-index) (2024.2.2)\n",
      "Requirement already satisfied: httpcore==1.* in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from httpx->llama-index-core<0.11.0,>=0.10.30->llama-index) (1.0.5)\n",
      "Requirement already satisfied: idna in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from httpx->llama-index-core<0.11.0,>=0.10.30->llama-index) (3.7)\n",
      "Requirement already satisfied: sniffio in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from httpx->llama-index-core<0.11.0,>=0.10.30->llama-index) (1.3.1)\n",
      "Requirement already satisfied: h11<0.15,>=0.13 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from httpcore==1.*->httpx->llama-index-core<0.11.0,>=0.10.30->llama-index) (0.14.0)\n",
      "Requirement already satisfied: click in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from nltk<4.0.0,>=3.8.1->llama-index-core<0.11.0,>=0.10.30->llama-index) (8.1.7)\n",
      "Requirement already satisfied: joblib in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from nltk<4.0.0,>=3.8.1->llama-index-core<0.11.0,>=0.10.30->llama-index) (1.4.0)\n",
      "Requirement already satisfied: regex>=2021.8.3 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from nltk<4.0.0,>=3.8.1->llama-index-core<0.11.0,>=0.10.30->llama-index) (2023.12.25)\n",
      "Requirement already satisfied: distro<2,>=1.7.0 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from openai>=1.14.0->llama-index-agent-openai<0.3.0,>=0.1.4->llama-index) (1.9.0)\n",
      "Requirement already satisfied: charset-normalizer<4,>=2 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from requests>=2.31.0->llama-index-core<0.11.0,>=0.10.30->llama-index) (3.3.2)\n",
      "Requirement already satisfied: urllib3<3,>=1.21.1 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from requests>=2.31.0->llama-index-core<0.11.0,>=0.10.30->llama-index) (1.26.18)\n",
      "Requirement already satisfied: greenlet!=0.4.17 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from SQLAlchemy[asyncio]>=1.4.49->llama-index-core<0.11.0,>=0.10.30->llama-index) (3.0.3)\n",
      "Requirement already satisfied: mypy-extensions>=0.3.0 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from typing-inspect>=0.8.0->llama-index-core<0.11.0,>=0.10.30->llama-index) (1.0.0)\n",
      "Requirement already satisfied: marshmallow<4.0.0,>=3.18.0 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from dataclasses-json->llama-index-core<0.11.0,>=0.10.30->llama-index) (3.21.1)\n",
      "Requirement already satisfied: python-dateutil>=2.8.2 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from pandas->llama-index-core<0.11.0,>=0.10.30->llama-index) (2.9.0.post0)\n",
      "Requirement already satisfied: pytz>=2020.1 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from pandas->llama-index-core<0.11.0,>=0.10.30->llama-index) (2024.1)\n",
      "Requirement already satisfied: tzdata>=2022.7 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from pandas->llama-index-core<0.11.0,>=0.10.30->llama-index) (2024.1)\n",
      "Requirement already satisfied: packaging>=17.0 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from marshmallow<4.0.0,>=3.18.0->dataclasses-json->llama-index-core<0.11.0,>=0.10.30->llama-index) (24.0)\n",
      "Requirement already satisfied: annotated-types>=0.4.0 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from pydantic>=1.10->llamaindex-py-client<0.2.0,>=0.1.18->llama-index-core<0.11.0,>=0.10.30->llama-index) (0.6.0)\n",
      "Requirement already satisfied: pydantic-core==2.18.1 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from pydantic>=1.10->llamaindex-py-client<0.2.0,>=0.1.18->llama-index-core<0.11.0,>=0.10.30->llama-index) (2.18.1)\n",
      "Requirement already satisfied: six>=1.5 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from python-dateutil>=2.8.2->pandas->llama-index-core<0.11.0,>=0.10.30->llama-index) (1.16.0)\n",
      "\n",
      "\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m A new release of pip available: \u001b[0m\u001b[31;49m22.3.1\u001b[0m\u001b[39;49m -> \u001b[0m\u001b[32;49m24.0\u001b[0m\n",
      "\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m To update, run: \u001b[0m\u001b[32;49mpip install --upgrade pip\u001b[0m\n",
      "Note: you may need to restart the kernel to use updated packages.\n",
      "Requirement already satisfied: llama-parse in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (0.4.0)\n",
      "Requirement already satisfied: llama-index-core>=0.10.7 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-parse) (0.10.30)\n",
      "Requirement already satisfied: PyYAML>=6.0.1 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index-core>=0.10.7->llama-parse) (6.0.1)\n",
      "Requirement already satisfied: SQLAlchemy[asyncio]>=1.4.49 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index-core>=0.10.7->llama-parse) (2.0.29)\n",
      "Requirement already satisfied: aiohttp<4.0.0,>=3.8.6 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index-core>=0.10.7->llama-parse) (3.9.4)\n",
      "Requirement already satisfied: dataclasses-json in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index-core>=0.10.7->llama-parse) (0.6.4)\n",
      "Requirement already satisfied: deprecated>=1.2.9.3 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index-core>=0.10.7->llama-parse) (1.2.14)\n",
      "Requirement already satisfied: dirtyjson<2.0.0,>=1.0.8 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index-core>=0.10.7->llama-parse) (1.0.8)\n",
      "Requirement already satisfied: fsspec>=2023.5.0 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index-core>=0.10.7->llama-parse) (2024.3.1)\n",
      "Requirement already satisfied: httpx in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index-core>=0.10.7->llama-parse) (0.27.0)\n",
      "Requirement already satisfied: llamaindex-py-client<0.2.0,>=0.1.18 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index-core>=0.10.7->llama-parse) (0.1.18)\n",
      "Requirement already satisfied: nest-asyncio<2.0.0,>=1.5.8 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index-core>=0.10.7->llama-parse) (1.6.0)\n",
      "Requirement already satisfied: networkx>=3.0 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index-core>=0.10.7->llama-parse) (3.3)\n",
      "Requirement already satisfied: nltk<4.0.0,>=3.8.1 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index-core>=0.10.7->llama-parse) (3.8.1)\n",
      "Requirement already satisfied: numpy in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index-core>=0.10.7->llama-parse) (1.26.4)\n",
      "Requirement already satisfied: openai>=1.1.0 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index-core>=0.10.7->llama-parse) (1.17.0)\n",
      "Requirement already satisfied: pandas in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index-core>=0.10.7->llama-parse) (2.2.2)\n",
      "Requirement already satisfied: pillow>=9.0.0 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index-core>=0.10.7->llama-parse) (10.3.0)\n",
      "Requirement already satisfied: requests>=2.31.0 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index-core>=0.10.7->llama-parse) (2.31.0)\n",
      "Requirement already satisfied: tenacity<9.0.0,>=8.2.0 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index-core>=0.10.7->llama-parse) (8.2.3)\n",
      "Requirement already satisfied: tiktoken>=0.3.3 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index-core>=0.10.7->llama-parse) (0.6.0)\n",
      "Requirement already satisfied: tqdm<5.0.0,>=4.66.1 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index-core>=0.10.7->llama-parse) (4.66.2)\n",
      "Requirement already satisfied: typing-extensions>=4.5.0 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index-core>=0.10.7->llama-parse) (4.11.0)\n",
      "Requirement already satisfied: typing-inspect>=0.8.0 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index-core>=0.10.7->llama-parse) (0.9.0)\n",
      "Requirement already satisfied: wrapt in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llama-index-core>=0.10.7->llama-parse) (1.16.0)\n",
      "Requirement already satisfied: aiosignal>=1.1.2 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from aiohttp<4.0.0,>=3.8.6->llama-index-core>=0.10.7->llama-parse) (1.3.1)\n",
      "Requirement already satisfied: attrs>=17.3.0 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from aiohttp<4.0.0,>=3.8.6->llama-index-core>=0.10.7->llama-parse) (23.2.0)\n",
      "Requirement already satisfied: frozenlist>=1.1.1 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from aiohttp<4.0.0,>=3.8.6->llama-index-core>=0.10.7->llama-parse) (1.4.1)\n",
      "Requirement already satisfied: multidict<7.0,>=4.5 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from aiohttp<4.0.0,>=3.8.6->llama-index-core>=0.10.7->llama-parse) (6.0.5)\n",
      "Requirement already satisfied: yarl<2.0,>=1.0 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from aiohttp<4.0.0,>=3.8.6->llama-index-core>=0.10.7->llama-parse) (1.9.4)\n",
      "Requirement already satisfied: pydantic>=1.10 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from llamaindex-py-client<0.2.0,>=0.1.18->llama-index-core>=0.10.7->llama-parse) (2.7.0)\n",
      "Requirement already satisfied: anyio in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from httpx->llama-index-core>=0.10.7->llama-parse) (4.3.0)\n",
      "Requirement already satisfied: certifi in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from httpx->llama-index-core>=0.10.7->llama-parse) (2024.2.2)\n",
      "Requirement already satisfied: httpcore==1.* in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from httpx->llama-index-core>=0.10.7->llama-parse) (1.0.5)\n",
      "Requirement already satisfied: idna in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from httpx->llama-index-core>=0.10.7->llama-parse) (3.7)\n",
      "Requirement already satisfied: sniffio in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from httpx->llama-index-core>=0.10.7->llama-parse) (1.3.1)\n",
      "Requirement already satisfied: h11<0.15,>=0.13 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from httpcore==1.*->httpx->llama-index-core>=0.10.7->llama-parse) (0.14.0)\n",
      "Requirement already satisfied: click in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from nltk<4.0.0,>=3.8.1->llama-index-core>=0.10.7->llama-parse) (8.1.7)\n",
      "Requirement already satisfied: joblib in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from nltk<4.0.0,>=3.8.1->llama-index-core>=0.10.7->llama-parse) (1.4.0)\n",
      "Requirement already satisfied: regex>=2021.8.3 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from nltk<4.0.0,>=3.8.1->llama-index-core>=0.10.7->llama-parse) (2023.12.25)\n",
      "Requirement already satisfied: distro<2,>=1.7.0 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from openai>=1.1.0->llama-index-core>=0.10.7->llama-parse) (1.9.0)\n",
      "Requirement already satisfied: charset-normalizer<4,>=2 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from requests>=2.31.0->llama-index-core>=0.10.7->llama-parse) (3.3.2)\n",
      "Requirement already satisfied: urllib3<3,>=1.21.1 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from requests>=2.31.0->llama-index-core>=0.10.7->llama-parse) (1.26.18)\n",
      "Requirement already satisfied: greenlet!=0.4.17 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from SQLAlchemy[asyncio]>=1.4.49->llama-index-core>=0.10.7->llama-parse) (3.0.3)\n",
      "Requirement already satisfied: mypy-extensions>=0.3.0 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from typing-inspect>=0.8.0->llama-index-core>=0.10.7->llama-parse) (1.0.0)\n",
      "Requirement already satisfied: marshmallow<4.0.0,>=3.18.0 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from dataclasses-json->llama-index-core>=0.10.7->llama-parse) (3.21.1)\n",
      "Requirement already satisfied: python-dateutil>=2.8.2 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from pandas->llama-index-core>=0.10.7->llama-parse) (2.9.0.post0)\n",
      "Requirement already satisfied: pytz>=2020.1 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from pandas->llama-index-core>=0.10.7->llama-parse) (2024.1)\n",
      "Requirement already satisfied: tzdata>=2022.7 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from pandas->llama-index-core>=0.10.7->llama-parse) (2024.1)\n",
      "Requirement already satisfied: packaging>=17.0 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from marshmallow<4.0.0,>=3.18.0->dataclasses-json->llama-index-core>=0.10.7->llama-parse) (24.0)\n",
      "Requirement already satisfied: annotated-types>=0.4.0 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from pydantic>=1.10->llamaindex-py-client<0.2.0,>=0.1.18->llama-index-core>=0.10.7->llama-parse) (0.6.0)\n",
      "Requirement already satisfied: pydantic-core==2.18.1 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from pydantic>=1.10->llamaindex-py-client<0.2.0,>=0.1.18->llama-index-core>=0.10.7->llama-parse) (2.18.1)\n",
      "Requirement already satisfied: six>=1.5 in /Users/suo/dev/odr-poc/.env/lib/python3.11/site-packages (from python-dateutil>=2.8.2->pandas->llama-index-core>=0.10.7->llama-parse) (1.16.0)\n",
      "\n",
      "\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m A new release of pip available: \u001b[0m\u001b[31;49m22.3.1\u001b[0m\u001b[39;49m -> \u001b[0m\u001b[32;49m24.0\u001b[0m\n",
      "\u001b[1m[\u001b[0m\u001b[34;49mnotice\u001b[0m\u001b[1;39;49m]\u001b[0m\u001b[39;49m To update, run: \u001b[0m\u001b[32;49mpip install --upgrade pip\u001b[0m\n",
      "Note: you may need to restart the kernel to use updated packages.\n"
     ]
    }
   ],
   "source": [
    "%pip install llama-index\n",
    "%pip install llama-parse"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6f3ba3bc-78b8-46c5-8964-725a46b4d2d9",
   "metadata": {},
   "source": [
    "### Setup and imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "b1ce7212-1ec3-472f-bfe6-20f2d515e66b",
   "metadata": {},
   "outputs": [],
   "source": [
    "import nest_asyncio\n",
    "\n",
    "nest_asyncio.apply()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a195eac4-081d-49a7-9bde-b60169e222d0",
   "metadata": {},
   "source": [
    "Get API key from http://cloud.llamaindex.ai/ and configure via env variable"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "f796a6f0-83f2-42ee-b6e4-542f28821657",
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "os.environ['LLAMA_CLOUD_API_KEY'] = 'llx-...'"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6294c909-dfd2-4398-8ac5-7fdeb6dfbedd",
   "metadata": {},
   "source": [
    "### Download data"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cc0d4ac0-2a16-4b2b-8348-4b39cd76fd7b",
   "metadata": {},
   "source": [
    "For this demo, we will build a simple knowledge base with 2 10K filings for uber and lyft."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "id": "02ab751f-2e88-4be8-9aff-bf84c23098c3",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--2024-04-18 13:31:02--  https://raw.githubusercontent.com/run-llama/llama_index/main/docs/docs/examples/data/10k/lyft_2021.pdf\n",
      "Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 2606:50c0:8001::154, 2606:50c0:8002::154, 2606:50c0:8003::154, ...\n",
      "Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|2606:50c0:8001::154|:443... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 1440303 (1.4M) [application/octet-stream]\n",
      "Saving to: ‘./data/10k/lyft_2021.pdf’\n",
      "\n",
      "./data/10k/lyft_202 100%[===================>]   1.37M  --.-KB/s    in 0.07s   \n",
      "\n",
      "2024-04-18 13:31:02 (18.6 MB/s) - ‘./data/10k/lyft_2021.pdf’ saved [1440303/1440303]\n",
      "\n",
      "--2024-04-18 13:31:02--  https://raw.githubusercontent.com/run-llama/llama_index/main/docs/docs/examples/data/10k/uber_2021.pdf\n",
      "Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 2606:50c0:8001::154, 2606:50c0:8002::154, 2606:50c0:8003::154, ...\n",
      "Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|2606:50c0:8001::154|:443... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 1880483 (1.8M) [application/octet-stream]\n",
      "Saving to: ‘./data/10k/uber_2021.pdf’\n",
      "\n",
      "./data/10k/uber_202 100%[===================>]   1.79M  --.-KB/s    in 0.1s    \n",
      "\n",
      "2024-04-18 13:31:02 (12.4 MB/s) - ‘./data/10k/uber_2021.pdf’ saved [1880483/1880483]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "!mkdir -p './data/10k/'\n",
    "!wget 'https://raw.githubusercontent.com/run-llama/llama_index/main/docs/docs/examples/data/10k/lyft_2021.pdf' -O './data/10k/lyft_2021.pdf'\n",
    "!wget 'https://raw.githubusercontent.com/run-llama/llama_index/main/docs/docs/examples/data/10k/uber_2021.pdf' -O './data/10k/uber_2021.pdf'"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e1da7ba1-215d-401a-8cd1-32442bfb64f9",
   "metadata": {},
   "source": [
    "### Load and parse data with LlamaParse"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "589e2fc0-7ffc-4de3-a297-275f476e8126",
   "metadata": {},
   "outputs": [],
   "source": [
    "from llama_parse import LlamaParse\n",
    "from llama_index.core import SimpleDirectoryReader\n",
    "\n",
    "parser = LlamaParse(\n",
    "    api_key=os.environ.get('LLAMA_CLOUD_API_KEY'),  # set via api_key param or in your env as LLAMA_CLOUD_API_KEY\n",
    "    result_type=\"markdown\",  # \"markdown\" and \"text\" are available\n",
    "    num_workers=4,  # if multiple files passed, split in `num_workers` API calls\n",
    "    verbose=True,\n",
    "    language=\"en\",  # Optionally you can define a language, default=en\n",
    ")\n",
    "\n",
    "file_extractor = {\".pdf\": parser}\n",
    "reader = SimpleDirectoryReader(\n",
    "    input_dir='data/10k/',\n",
    "    file_extractor=file_extractor\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "75a3d4f9-b6ad-4128-bb63-72e49bc349d3",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Started parsing the file under job_id 705de5a7-2782-40e8-a5cf-813bb198a9fe\n",
      "Started parsing the file under job_id f35906d9-4b66-4d31-99d8-6cdb087c8934\n"
     ]
    }
   ],
   "source": [
    "documents = reader.load_data()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "86231a57-5595-4cf7-8fda-0bda763305f9",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'file_path': '/Users/suo/dev/rag-bedrock/data/10k/lyft_2021.pdf',\n",
       " 'file_name': 'lyft_2021.pdf',\n",
       " 'file_type': 'application/pdf',\n",
       " 'file_size': 1440303,\n",
       " 'creation_date': '2024-04-18',\n",
       " 'last_modified_date': '2024-04-18'}"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "documents[0].metadata"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ae1ff75a-8bea-4457-901d-025504c08c32",
   "metadata": {},
   "source": [
    "### Upload data as markdown and metadata file "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eafe0255-9139-4338-93ec-441c9dd3248a",
   "metadata": {},
   "source": [
    "Now, upload and stage the parsed result markdown files for ingestion in to Amazon Bedrock KB.\n",
    "\n",
    "We also create metadata json files (the format is specific to Amazong Bedrock KB)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "9f7b34db-36f7-432f-ba88-3bb4baf292ab",
   "metadata": {},
   "outputs": [],
   "source": [
    "import boto3 \n",
    "import botocore\n",
    "\n",
    "# Create an S3 client\n",
    "s3 = boto3.client('s3')\n",
    "\n",
    "# Specify the bucket\n",
    "bucket_name = 'bedrock-kb-10ks'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "d833bf9b-8834-4f12-bc2d-840e7788c8b3",
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "\n",
    "def create_bucket(bucket_name):\n",
    "    try:\n",
    "        s3.head_bucket(Bucket=bucket_name)\n",
    "        print(f\"Bucket '{bucket_name}' already exists.\")\n",
    "    except botocore.exceptions.ClientError as e:\n",
    "        if e.response['Error']['Code'] == '404':\n",
    "            s3.create_bucket(Bucket=bucket_name)\n",
    "            print(f\"Bucket '{bucket_name}' created successfully.\")\n",
    "        else:\n",
    "            print(f\"Error creating bucket: {str(e)}\")\n",
    "            raise    \n",
    "\n",
    "def upload_document(document, bucket_name):\n",
    "    try:\n",
    "        object_key = document.metadata['file_path']\n",
    "        \n",
    "        # Upload the text as a markdown file\n",
    "        s3.put_object(\n",
    "            Body=document.text.encode('utf-8'),\n",
    "            Bucket=bucket_name,\n",
    "            Key=f\"{object_key}.md\",\n",
    "            ContentType='text/markdown'\n",
    "        )\n",
    "        print(f\"Text uploaded to S3 as '{object_key}.md'\")\n",
    "\n",
    "        # Format the metadata in the desired structure\n",
    "        formatted_metadata = {\n",
    "            \"metadataAttributes\": document.metadata\n",
    "        }\n",
    "        \n",
    "        # Upload the metadata as a JSON file\n",
    "        metadata_json = json.dumps(formatted_metadata, indent=4)\n",
    "        s3.put_object(\n",
    "            Body=metadata_json.encode('utf-8'),\n",
    "            Bucket=bucket_name,\n",
    "            Key=f\"{object_key}.md.metadata.json\",\n",
    "            ContentType='application/json'\n",
    "        )\n",
    "        print(f\"Metadata uploaded to S3 as '{object_key}.md.metadata.json'\")\n",
    "    \n",
    "    except Exception as e:\n",
    "        print(f\"Error uploading document: {str(e)}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "ce181b28-3318-42d7-b8ce-15a6c659bee4",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Text uploaded to S3 as '/Users/suo/dev/rag-bedrock/data/10k/lyft_2021.pdf.md'\n",
      "Metadata uploaded to S3 as '/Users/suo/dev/rag-bedrock/data/10k/lyft_2021.pdf.md.metadata.json'\n",
      "Text uploaded to S3 as '/Users/suo/dev/rag-bedrock/data/10k/uber_2021.pdf.md'\n",
      "Metadata uploaded to S3 as '/Users/suo/dev/rag-bedrock/data/10k/uber_2021.pdf.md.metadata.json'\n"
     ]
    }
   ],
   "source": [
    "for doc in documents:\n",
    "    upload_document(doc, bucket_name)  "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "43c3893e-03d2-4429-9eaf-98d1bb787e51",
   "metadata": {},
   "source": [
    "### Next: Create Amazon Bedrock KB"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9c163e38-5711-4d60-b7f8-d4729d997037",
   "metadata": {},
   "source": [
    "Now, you can create an Amazon Bedrock KB either:\n",
    "1. via the AWS managment console, or\n",
    "2. programmatically following this guide: https://github.com/aws-samples/amazon-bedrock-workshop/blob/main/02_KnowledgeBases_and_RAG/0_create_ingest_documents_test_kb.ipynb"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9278cf5e-057d-42ca-8d62-0ad388a0d7f2",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}