[Question]: How to pass document parameter table data?

[iriye]

Question Validation

• I have searched both the documentation and discord for an answer.

Question

Is it possible for the "documents" parameter to be in a table data format when creating a VectorStore.from_documents? I use tapas model that table question answring model,

[dosubot[bot]]

🤖

Hello,

Thank you for your question. Currently, the "documents" parameter in the VectorStore.from_documents method cannot be in a table data format. The "documents" parameter is expected to be a sequence of Document objects, where each Document object contains text and metadata. Here's a snippet of the relevant code:

def from_documents(
    cls: Type[IndexType],
    documents: Sequence[Document],
    storage_context: Optional[StorageContext] = None,
    service_context: Optional[ServiceContext] = None,
    show_progress: bool = False,
    **kwargs: Any,
) -> IndexType:
    """Build a Vectara index from a sequence of documents."""
    nodes = [
        TextNode(text=document.text, metadata=document.metadata)
        for document in documents
    ]
    return cls(
        nodes=nodes,
        show_progress=show_progress,
        **kwargs,
    )

In this method, each Document object is converted into a TextNode object, which is used to build the Vectara index. Therefore, the "documents" parameter cannot be in a table data format.

As for the structure of the Document object in the LlamaIndex codebase, it is represented by the DocumentSummaryIndex class, which is a subclass of BaseIndex with IndexDocumentSummary as its index structure. You can find more details about the DocumentSummaryIndex class and the IndexDocumentSummary class in the LlamaIndex codebase here and here respectively.

I hope this answers your question. If you have any more questions or need further clarification, feel free to ask.

Best regards,
Dosu

Sources

• llama_index/indices/managed/vectara/base.py
• llama_index/indices/document_summary/base.py
• llama_index/data_structs/document_summary.py

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[iriye]

How can I create a vector store with tabular data? The model is working as a Table-Question Answering model ?

I can give you example code blocks:

model_name = "google/tapas-base-finetuned-wtq"
tokenizer = TapasTokenizer.from_pretrained(model_name)
model = TapasForQuestionAnswering.from_pretrained(model_name, local_files_only=False)

index = VectorStoreIndex.from_documents(documents, service_context=service_context)

How can create documents tabular format or change code block create another voectore store using tabular data

[logan-markewich]

I'm not sure about the issue here?

I think you could support this creating the nodes directly with the "table text" that you would normally give tapas?

from llama_index.schema import TextNode

node = TextNode(text="table text")
index = VectorStoreIndex([node])