[Question]: Does SQLTableRetrieverQueryEngine support streaming?

[rolandgvc]

Question Validation

• I have searched both the documentation and discord for an answer.

Question

Here is my current code

llm = ChatOpenAI(temperature=0.1, model="gpt-4", streaming=True)
service_context = ServiceContext.from_defaults(llm=llm)

query_engine = SQLTableRetrieverQueryEngine(
    sql_database, obj_index.as_retriever(similarity_top_k=5), service_context=service_context
)

however, I can't get the StreamingResponse output out of query_engine.query. Any ideas how I could get the streaming functionality working here?

[dosubot[bot]]

🤖

Hi there,

Based on the information provided, it seems like you're trying to use the streaming parameter with the SQLTableRetrieverQueryEngine class. However, according to the context provided, the SQLTableRetrieverQueryEngine class does not have a streaming parameter in its constructor.

The streaming parameter is available in the from_args class method of the RetrieverQueryEngine class. Here's how you can use it:

query_engine = RetrieverQueryEngine.from_args(
    retriever=my_retriever,
    streaming=True,
    ...
)

After setting up the streaming parameter, you can call the query method which will now return a StreamingResponse object:

streaming_response = query_engine.query(
    "What did the author do growing up?", 
)

You can then obtain a Generator from the streaming response and iterate over the tokens as they arrive:

for text in streaming_response.response_gen:
    # do something with text as they arrive.

Or, if you just want to print the text as they arrive:

streaming_response.print_response_stream() 

Please note that the LLM you use must support streaming. Currently, streaming is supported by OpenAI, HuggingFaceLLM, and most LangChain LLMs (via LangChainLLM).

If you're trying to use the streaming parameter with a different class, could you please provide more information about that class? It would be helpful to know where it's defined and how it's supposed to be used.

I hope this helps! Let me know if you have any other questions.

Sources

• Does chat engine support streaming response?
• [Question]: Want to ask how to support the streaming of multi-document question
• docs/core_modules/query_modules/query_engine/streaming.md
• llama_index/query_engine/retriever_query_engine.py

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[logan-markewich]

@rolandgvc hmm, looks like it's not possible at the moment.

Ideally, the code would be using an actual response synthesizer here, and not predict() directly
https://github.com/jerryjliu/llama_index/blob/94912afd2c15fc10b7eddbe7f95cba8555fd1fc6/llama_index/indices/struct_store/sql_query.py#L181

Would need a PR to fix

[rolandgvc]

Thanks @logan-markewich . Do you mind creating the PR, I'm not very familiar with the inner workings of the classes. Would this enable streaming right away or does it need additional changes?

[birudeghi]

Hi, is this being actively worked on? I have similar problem with enabling streaming with SQLTableRetrieverQueryEngine

[logan-markewich]

@birudeghi yea not being actively worked on yet. As mentioned above, ideally it would use a response synthesizer. I highly encourage a contribution if you need it now 🙏🏻