Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

example: add notebook processing reasoning trace from sqlite artifact #2526

Merged
merged 46 commits into from
May 29, 2024

Conversation

lalo
Copy link
Member

@lalo lalo commented Apr 26, 2024

Why are these changes needed?

The goal of this notebook is to generate this type of UI [currently work in progress] - emphasis on the next/back buttons:

currentui:

  • first text box has the scenario.py event section
  • second text box has the first element in this section with an image
  • third text box has the remaining event_names after the previous [second box] event

current version:

image

To explore traces with the relevant screenshots obtained from the sqlite artifact generated by autogen runs.

  • link merged events with the UI
  • improve textbox UI - ideally this 'box' will contain the event that has the image and all the following events without images. Clicking next will show the next event with image and then the next chunk of image-less events. This way we can keep track of the events emitted after certain screenshot states.

Related issue number

Checks

@lalo lalo marked this pull request as draft April 26, 2024 12:25
@lalo lalo requested review from gagb and afourney April 26, 2024 12:27
@lalo lalo requested a review from jackgerrits April 26, 2024 22:05
@afourney afourney marked this pull request as ready for review April 27, 2024 04:31
@lalo lalo changed the title wip: add notebook processing reasoning trace from sqlite artifact example: add notebook processing reasoning trace from sqlite artifact May 17, 2024
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This file currently lives inside GAIA, maybe move it to WebArena or Autogenbench?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

moved to webarena/

"\n",
"final = merged_and_nested\n",
"\n",
"cookies_events = [event for event in final if event[\"event_name\"] == \"oookies\"]\n",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm getting KeyError: 'event_name' from this line

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fixed thanks

"# Brief explanation\n",
"\n",
"The logs collected by scenario.py in https://github.com/microsoft/autogen/blob/ct_webarena/samples/tools/autogenbench/scenarios/WebArena/Templates/TwoAgents/scenario.py#L50\n",
"\n",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

not sure if this line is out of date, line 50 of scenario.py is pointing to the task not log_event? Maybe just remove the link?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

outdated, yes

Copy link
Member

@afourney afourney left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good. I was able to get the notebook up and running in WSL2, without any issues (beyond changing the log path).

@lalo lalo merged commit 3ce7fd0 into microsoft:ct_webarena May 29, 2024
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants