-
Notifications
You must be signed in to change notification settings - Fork 2.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Bug]: Async engine hangs with 0.4.* releases #4789
Comments
Quote from issue templates. |
vllm_trace_function.log |
When you initialize the async engine I think it expects to be in an running event loop, not sure why though. If you change the code to diff --git a/original.py b/repro.py
index 37ede83..8881b6f 100644
--- a/original.py
+++ b/repro.py
@@ -32,15 +32,15 @@ async def generate(engine: AsyncLLMEngine, prompts: list[str]) -> list[str]:
return list(outputs)
-def main():
+async def main():
args = parse_args()
engine = AsyncLLMEngine.from_engine_args(AsyncEngineArgs.from_cli_args(args))
prompts = ["I've never been to", "I would like to see"]
- outputs = asyncio.run(generate(engine, prompts))
+ outputs = await generate(engine, prompts)
for output in outputs:
print(output)
print("FINISHED")
if __name__ == '__main__':
- main()
+ asyncio.run(main()) The prints should succeed. However, the script still will not exit. So there's still a bug. |
Saw the same issue. I tested and the culprit commit should be #3015. It's not clear to me what's the root cause though. |
So the problem is that vllm/vllm/engine/async_llm_engine.py Line 509 in 97b0300
The bug is fixed in Python 3.12 but will not be backported. It's very easy to workaround it in Python 3.11 with |
Your current environment
🐛 Describe the bug
The code below worked fine with vllm==0.3.3 (I could see the generated output printed to the console).
However, when I try to use it with 0.4.0, 0.4.1 or 0.4.2 it hangs.
The last logs before iterrupting the execution are
And after finally pressing Ctrl+C I get
Is there something wrong with my code or is it a bug in
vllm
?The text was updated successfully, but these errors were encountered: