Handling of tool errors (e.g. Database) #432

mattlindsey · 2023-12-28T03:08:26Z

Example below shows where a Tool raises an error, but the Assistant doesn't recognize it and happily gives wrong answer. Maybe the Assistant can at least acknowledge that an error happened, and in the specific case below if the Assistant gets access to a 'schema' function/tool, it should work better.

mattlindsey · 2023-12-28T03:48:06Z

Would just making sure any error message is put on the Thread help? I don't see it anywhere above except the log message.

Python langchain has elaborate configuration for 'Handling Tool Errors,' but is that unnecessary complexity?

mattlindsey · 2023-12-28T04:24:56Z

This looks better:

 #<Langchain::Message:0x0000000105a35ab0
  @content=
   "PG::UndefinedColumn: ERROR:  column \"name\" does not exist\nLINE 1: SELECT COUNT(*) FROM users WHERE LENGTH(name) > 5\n                                                ^\n",
  @role="tool",
  @tool_call_id="call_yKmQDXW3OpJM28m8JO7IowAy",
  @tool_calls=[]>,
 #<Langchain::Message:0x0000000105beed98
  @content=
   "I apologize for the confusion. It seems that the column \"name\" does not exist in the users table. Could you please provide the correct column name for the user's name?",
  @role="assistant",
  @tool_call_id=nil,
  @tool_calls=[]>]
#<Langchain::Message:0x0000000103eeab48 @content="Actually I meant the first_name column.", @role="user", @tool_call_id=nil, @tool_calls=[]>,
 #<Langchain::Message:0x000000010593f070
  @content="",
  @role="assistant",
  @tool_call_id=nil,
  @tool_calls=
   [{"id"=>"call_99rBiKnqGwnWC2B6Q708lmHE",
     "type"=>"function",
     "function"=>{"name"=>"database", "arguments"=>"{\n  \"input\": \"SELECT COUNT(*) FROM users WHERE LENGTH(first_name) > 5\"\n}"}}]>,
 #<Langchain::Message:0x000000010593eda0 @content="[{:count=>66}]", @role="tool", @tool_call_id="call_99rBiKnqGwnWC2B6Q708lmHE", @tool_calls=[]>,
 #<Langchain::Message:0x0000000105936ce0
  @content="There are 66 users in the users table whose first_name has a length greater than 5.",
  @role="assistant",
  @tool_call_id=nil,
  @tool_calls=[]>]

Return error below in database.rb. Maybe in other tools too?

    #
    # Evaluates a sql expression
    #
    # @param input [String] sql expression
    # @return [Array] results
    #
    def execute(input:)
      Langchain.logger.info("Executing \"#{input}\"", for: self.class)

      db[input].to_a
    rescue Sequel::DatabaseError => e
      Langchain.logger.error(e.message, for: self.class)
      e.message     ###### ADDED THIS #######
    end

mattlindsey mentioned this issue Dec 28, 2023

Return any database error from the tool #433

Closed

This was referenced Apr 24, 2024

Minor db tool improvements #589

Closed

Minor database tool improvements #591

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Handling of tool errors (e.g. Database) #432

Handling of tool errors (e.g. Database) #432

mattlindsey commented Dec 28, 2023

mattlindsey commented Dec 28, 2023 •

edited

mattlindsey commented Dec 28, 2023 •

edited

Handling of tool errors (e.g. Database) #432

Handling of tool errors (e.g. Database) #432

Comments

mattlindsey commented Dec 28, 2023

mattlindsey commented Dec 28, 2023 • edited

mattlindsey commented Dec 28, 2023 • edited

mattlindsey commented Dec 28, 2023 •

edited

mattlindsey commented Dec 28, 2023 •

edited