[Bug]: Using the new GPT-4o as the LLM model makes Pythagora sometimes spew out and endless loop which stops the next agent from being able to do anything

[MathieuDawes]

Version

VisualStudio Code extension

Operating System

MacOS

What happened?

When I run Pythagora using GPT-4o it produces very long json outputs and keeps repeating the code for each step over and over. Whereas when it's using gpt-4-turbo it doesn't repeat the code unnecessarily in the json instructions.

[RKonnerth-A]

I have the same Bug, or is it a feature for creating a vector knowledge base for the project ?

[senko]

Thanks for reporting this @MathieuDawes and @RKonnerth-A.

We've had a lot of problems with GPT 4o, in our use cases it produced much worse results than gpt-4-turbo. We also compared gpt-4-turbo with gpt-4-turbo-preview and found out that gpt-4-turbo-preview works best.

Currently it's unclear whether that's to any optimizations OpenAI did to make gpt-4o run faster/cheaper (as we're at the bleeding/expensive edge of AI capabilities, any such change might have an outsize impact on us), or whether our promtps are too fine-tuned (pun intended!) for gpt-4-turbo-preview, or a combination of the two.

We're currently doing a deeper analysis but for now recommend using gpt-4-turbo-preview for the best results.