Qwen-vl API request error

[yzhao666]

Hey MS UFO team,

Thanks for your excellent work and recent updates for supporting Qwen API.
I'm testing the code on my win11 using the Qwen-vl-plus api, with outlook activated and fully maximized on my desktop in order to be taken a screenshotbut, but constantly receiving the 'Error making API request' issue.
After debugging, I found that although I can receive text response from Qwen-vl-plus, the content doesn't contain 'Observation' part (according to line86-92 of script 'ufo\llm\qwen.py' and the example in 'ufo\app_agent_example.yaml' ), thus raising an exception.
Could you please help identify how to fix the problem? Thanks in advance!
My input and output details are as follows：

(```
UFO) PS D:\Projects\GithubProjects\UFO-v0.1.1> python -m ufo --task 'write email'

Welcome to use UFO🛸, A UI-focused Agent for Windows OS Interaction.

---

| | | || | / _
| | | || | | | | |
| || || | | || |
_/ |_| __/

Please enter your request to be completed🛸:
My name is Zac. Please send a email to jack@outlook.com to thanks his contribution on the open source.
Round 1, Step 1: Selecting an application.
Error making API request: To send an email via code, you can use various libraries such as smtplib in Python or JavaMail API in Java.

Here's how you could do it using these tools:

Python:

import smtplib

# Set up your credentials and message content here
sender_email = 'your-email@example.com'
receiver_email = 'jack@outlook.com'

message_content = """\
Subject: Thanks for contributing!

Dear Jack,

Thank you very much for your valuable contributions to our open-source project! We appreciate your hard work and dedication.

Best regards,
[Your Name]"""

# Establish connection with SMTP server
with smtplib.SMTP('smtp.example.com', 587) as smtp_server:
    # Start TLS encryption if available
    smtp_server.starttls()

    # Login with your credentials
    smtp_server.login(sender_email, 'password')

    # Send mail
    smtp_server.sendmail(
        sender_email,
        receiver_email,
        message_content.encode()
    )

Java:

import com.sun.mail.smtp.SMTPTransport;
import javax.activation.DataHandler;
import javax.mail.Message;
import javax.mail.MessagingException;
import javax.mail.Session;
import javax.mail.Transport;
import javax.mail.internet.AddressException;
import javax.mail.internet.InternetAddress;
import javax.mail.internet.MimeMessage;

// ...

try {
    // Create session object
    Session session = Session.getDefaultInstance(props);

    // Get address of recipient from command line arguments
    String dest = args[0];

    try {
        InternetAddress internetDest =
            new InternetAddress(dest);
        Message msg = new MimeMessage(session);
        msg.setFrom(new InternetAddress("youremail@example.com"));
        msg.addRecipient(Message.RecipientType.TO, internetDest);
        msg.setSubject("Thanks for contributing!");
        msg.setText("Dear Jack,\n\n" +
                "Thank you very much for your valuable contributions to our open-source project!\nWe appreciate your hard work and dedication.\n\n" +
                "Best regards," + Environment.NewLine +
                "[Your Name]");
        Transport transport = session.getTransport();
        transport.connect(); // Connects to host specified by properties above
        transport.sendMessage(msg, msg.getAllRecipients());
        System.out.println("Email sent successfully.");
    } catch (MessagingException e) {
        throw new RuntimeException(e);
    }
} finally { ... }

Please replace 'your-email@example.com', ['your-name'] and 'password' with appropriate values before running this script. Also note that sending emails through plain text might not be secure depending on the email service provider. For more security-conscious applications, consider switching to encrypted protocols like SSL/TLS or even HTTPS.
{"status_code": 200, "request_id": "9ff2692f-705c-9724-aba7-304117b95ffc", "code": "", "message": "", "output": {"text": null, "finish_reason": null, "choices": [{"finish_reason": "stop", "message": {"role": "assistant", "content": [{"text": "To send an email via code, you can use various libraries such as smtplib in Python or JavaMail API in Java.\n\nHere's how you could do it using these tools:\n\nPython:\npython\nimport smtplib\n\n# Set up your credentials and message content here\nsender_email = 'your-email@example.com'\nreceiver_email = 'jack@outlook.com'\n\nmessage_content = \"\"\"\\\nSubject: Thanks for contributing!\n\nDear Jack,\n\nThank you very much for your valuable contributions to our open-source project! We appreciate your hard work and dedication.\n\nBest regards,\n[Your Name]\"\"\"\n\n# Establish connection with SMTP server\nwith smtplib.SMTP('smtp.example.com', 587) as smtp_server:\n # Start TLS encryption if available\n smtp_server.starttls()\n\n # Login with your credentials\n smtp_server.login(sender_email, 'password')\n\n # Send mail\n smtp_server.sendmail(\n sender_email,\n receiver_email,\n message_content.encode()\n )\n\n\nJava:\njava\nimport com.sun.mail.smtp.SMTPTransport;\nimport javax.activation.DataHandler;\nimport javax.mail.Message;\nimport javax.mail.MessagingException;\nimport javax.mail.Session;\nimport javax.mail.Transport;\nimport javax.mail.internet.AddressException;\nimport javax.mail.internet.InternetAddress;\nimport javax.mail.internet.MimeMessage;\n\n// ...\n\ntry {\n // Create session object\n Session session = Session.getDefaultInstance(props);\n\n // Get address of recipient from command line arguments\n String dest = args[0];\n\n try {\n InternetAddress internetDest =\n new InternetAddress(dest);\n Message msg = new MimeMessage(session);\n msg.setFrom(new InternetAddress(\"youremail@example.com\"));\n msg.addRecipient(Message.RecipientType.TO, internetDest);\n msg.setSubject(\"Thanks for contributing!\");\n msg.setText(\"Dear Jack,\\n\\n\" +\n \"Thank you very much for your valuable contributions to our open-source project!\\nWe appreciate your hard work and dedication.\\n\\n\" +\n \"Best regards,\" + Environment.NewLine +\n \"[Your Name]\");\n Transport transport = session.getTransport();\n transport.connect(); // Connects to host specified by properties above\n transport.sendMessage(msg, msg.getAllRecipients());\n System.out.println(\"Email sent successfully.\");\n } catch (MessagingException e) {\n throw new RuntimeException(e); \n }\n} finally { ... }\n\nPlease replace 'your-email@example.com', ['your-name'] and 'password' with appropriate values before running this script. Also note that sending emails through plain text might not be secure depending on the email service provider. For more security-conscious applications, consider switching to encrypted protocols like SSL/TLS or even HTTPS."}]}}]}, "usage": {"input_tokens": 1241, "output_tokens": 552, "image_tokens": 180}}
Error making API request: To send an email via code, you can use various libraries such as smtplib and email. Here's an example of how you might do this in Python:

import smtplib

# Set up the sender and recipient
sender = 'your_email@example.com'
recipient = 'jack@outlook.com'

# Compose the message body
message_body = """
    Subject: Thanks for your contribution!

    Hi Jack,

    Thank you very much for contributing to our open-source project! We really appreciate your help.

    Best regards,
    Zac"""

# Create a secure SSL connection using port 465
with smtplib.SMTP_SSL('smtp.outlook.com', 465) as server:

    # Send the email with the composed message
    server.login(sender, 'your_password')
    server.sendmail(
        sender,
        recipient,
        message_body
    )

Note that you will need to replace 'your_email@example.com', 'your_password', and other placeholders with your actual email address and password. Also be sure to import any necessary modules or dependencies at the top of your script if they're not already imported.
{"status_code": 200, "request_id": "12b3347f-2964-91e0-985a-d348867b4d3f", "code": "", "message": "", "output": {"text": null, "finish_reason": null, "choices": [{"finish_reason": "stop", "message": {"role": "assistant", "content": [{"text": "To send an email via code, you can use various libraries such as smtplib and email. Here's an example of how you might do this in Python:\npython\nimport smtplib\n\n# Set up the sender and recipient\nsender = 'your_email@example.com'\nrecipient = 'jack@outlook.com'\n\n# Compose the message body\nmessage_body = \"\"\"\n Subject: Thanks for your contribution!\n\n Hi Jack,\n\n Thank you very much for contributing to our open-source project! We really appreciate your help.\n\n Best regards,\n Zac\"\"\"\n\n# Create a secure SSL connection using port 465\nwith smtplib.SMTP_SSL('smtp.outlook.com', 465) as server:\n\n # Send the email with the composed message\n server.login(sender, 'your_password')\n server.sendmail(\n sender,\n recipient,\n message_body\n )\n\n\nNote that you will need to replace 'your_email@example.com', 'your_password', and other placeholders with your actual email address and password. Also be sure to import any necessary modules or dependencies at the top of your script if they're not already imported."}]}}]}, "usage": {"input_tokens": 1241, "output_tokens": 242, "image_tokens": 180}}
Error making API request: To send an email via code, you can use various libraries such as smtplib in Python or JavaMail API in Java. Here's how you could do it using these tools:

Python:

import smtplib

def send_email():
    # Set up the sender and recipient addresses
    fromaddr = 'your-email@example.com'
    toaddrs   = ['jack@example.com']

    # Create the message content
    msg = """\
Subject: Thanks!

Dear Jack,

Thank you very much for your contributions to our open-source project! We appreciate your hard work.

Best regards,
The Project Team"""

    try:
        # Connect to the SMTP server and send the mail
        smtpObj = smtplib.SMTP('smtp.example.com')
        smtpObj.sendmail(fromaddr, toaddrs, msg)
        print("Email sent successfully!")
    except Exception as e:
        print(f"Error occurred while sending email: {str(e)}")

send_email()

Java:

import javax.mail.*;
import java.util.Properties;

public class SendEmail {
    public static void main(String args[]) throws MessagingException {

        // Get system properties
        Properties props = System.getProperties();

        // Setup mail server
        String host = "smtp.example.com";
        props.put("mail.smtp.host", host);

        // Get the Session object
        Session session = Session.getDefaultInstance(props);

        try {
            // Create a default MimeMessage object.
            Message message = new MimeMessage(session);
            message.setFrom(new InternetAddress("from-email@example.com"));
            message.addRecipient(Message.RecipientType.TO, new InternetAddress("jack@example.com"));

            // Set the subject and the body of the message
            message.setSubject("Thanks!");
            message.setText("Dear Jack,\n\nThank you very much for your contributions to our open-source project!\nWe appreciate your hard work.\n\nBest regards," +
                    "\nThe Project Team");

            // Send the actual email with the message through the transport service
            Transport.send(message);
            System.out.println("Email sent successfully!");

        } catch (MessagingException mex) {
            throw new RuntimeException(mex);
        }
    }
}

Please replace your-email@example.com, jack@example.com, and smtp.example.com with appropriate values before running this script. Also note that some parts may vary depending on which programming language/library you choose to implement this functionality.
{"status_code": 200, "request_id": "d58465d5-11ba-9009-93fc-bb682810236c", "code": "", "message": "", "output": {"text": null, "finish_reason": null, "choices": [{"finish_reason": "stop", "message": {"role": "assistant", "content": [{"text": "To send an email via code, you can use various libraries such as smtplib in Python or JavaMail API in Java. Here's how you could do it using these tools:\n\nPython:\npython\nimport smtplib\n\ndef send_email():\n # Set up the sender and recipient addresses\n fromaddr = 'your-email@example.com'\n toaddrs = ['jack@example.com']\n\n # Create the message content\n msg = \"\"\"\\\nSubject: Thanks!\n\nDear Jack,\n\nThank you very much for your contributions to our open-source project! We appreciate your hard work.\n\nBest regards,\nThe Project Team\"\"\"\n\n try:\n # Connect to the SMTP server and send the mail\n smtpObj = smtplib.SMTP('smtp.example.com')\n smtpObj.sendmail(fromaddr, toaddrs, msg)\n print(\"Email sent successfully!\")\n except Exception as e:\n print(f\"Error occurred while sending email: {str(e)}\")\n\nsend_email()\n\nJava:\njava\nimport javax.mail.*;\nimport java.util.Properties;\n\npublic class SendEmail {\n public static void main(String args[]) throws MessagingException {\n\n // Get system properties\n Properties props = System.getProperties();\n\n // Setup mail server\n String host = \"smtp.example.com\";\n props.put(\"mail.smtp.host\", host);\n\n // Get the Session object\n Session session = Session.getDefaultInstance(props);\n\n try {\n // Create a default MimeMessage object.\n Message message = new MimeMessage(session);\n message.setFrom(new InternetAddress(\"from-email@example.com\"));\n message.addRecipient(Message.RecipientType.TO, new InternetAddress(\"jack@example.com\"));\n\n // Set the subject and the body of the message\n message.setSubject(\"Thanks!\");\n message.setText(\"Dear Jack,\\n\\nThank you very much for your contributions to our open-source project!\\nWe appreciate your hard work.\\n\\nBest regards,\" +\n \"\\nThe Project Team\");\n\n // Send the actual email with the message through the transport service\n Transport.send(message);\n System.out.println(\"Email sent successfully!\");\n\n } catch (MessagingException mex) {\n throw new RuntimeException(mex);\n }\n }\n}\n\n\nPlease replace your-email@example.com, jack@example.com, and smtp.example.com with appropriate values before running this script. Also note that some parts may vary depending on which programming language/library you choose to implement this functionality."}]}}]}, "usage": {"input_tokens": 1241, "output_tokens": 496, "image_tokens": 180}}
Error occurs when calling LLM: Traceback (most recent call last):
File "D:\Projects\GithubProjects\UFO-v0.1.1\ufo\module\processors\processor.py", line 165, in get_response
self._response, self._cost = self.host_agent.get_response(
File "D:\Projects\GithubProjects\UFO-v0.1.1\ufo\agent\basic.py", line 278, in get_response
response_string, cost = llm_call.get_completion(
File "D:\Projects\GithubProjects\UFO-v0.1.1\ufo\llm\llm_call.py", line 33, in get_completion
return responses[0], cost
IndexError: list index out of range

Traceback (most recent call last):
File "D:\ProgramFiles\anaconda3\envs\UFO\lib\runpy.py", line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
File "D:\ProgramFiles\anaconda3\envs\UFO\lib\runpy.py", line 86, in run_code
exec(code, run_globals)
File "D:\Projects\GithubProjects\UFO-v0.1.1\ufo_main.py", line 7, in
ufo.main()
File "D:\Projects\GithubProjects\UFO-v0.1.1\ufo\ufo.py", line 55, in main
clients.run_all()
File "D:\Projects\GithubProjects\UFO-v0.1.1\ufo\module\client.py", line 52, in run_all
client.run()
File "D:\Projects\GithubProjects\UFO-v0.1.1\ufo\module\client.py", line 28, in run
self.session.handle()
File "D:\Projects\GithubProjects\UFO-v0.1.1\ufo\module\basic.py", line 459, in handle
self._state.handle(self)
File "D:\Projects\GithubProjects\UFO-v0.1.1\ufo\module\state.py", line 194, in handle
session.round_hostagent_execution()
File "D:\Projects\GithubProjects\UFO-v0.1.1\ufo\module\session.py", line 224, in round_hostagent_execution
self.application = self.app_window.window_text()
AttributeError: 'NoneType' object has no attribute 'window_text'

[vyokky]

Got it. Will look into it and get back to you.

[yzhao666]

Got it. Will look into it and get back to you.

Many thanks for your quick feedback!!!

[Mac0q]

Can you show us the configuration so we can reproduce it?

[yzhao666]

Can you show us the configuration so we can reproduce it?

Sure. Thanks for your quick reply.
By the way yesterday I met annother error with the code you released . After adding 'api_type' and 'prices' in 'llm\qwen.py' as follows, that problem solved.

class QwenService(BaseService):
    def __init__(self, config, agent_type: str):
        self.config_llm = config[agent_type]
        self.config = config
        **self.api_type = self.config_llm["API_TYPE"].lower()**
        self.max_retry = self.config["MAX_RETRY"]
        **self.prices = self.config["PRICES"]**
        self.timeout = self.config["TIMEOUT"]
        dashscope.api_key = self.config_llm["API_KEY"]
        self.tmp_dir = None

The api request error still exist:

My 'config.yaml':

I didn't change anything else in the config.yaml file.

[Update]:

1. I've checked the description in Qwen-vl developer reference:
   https://help.aliyun.com/zh/dashscope/developer-reference/tongyi-qianwen-vl-plus-api?spm=a2c4g.11186623.0.i0
   It says that "Important: VL model currently does not recommend customizing system role"
   

Is this why qwen-vl can not follow the response examples listed in the system role's prompt message？

2. I've switched to the lite version config in 'ufo/config/config_dev.yaml' file:
   
   Meet the 'open_app_guideline' and 'open_app_comment' key error: solved by copying the open_app_guidline and open_app_comment content from 'base/host_agent.yaml' file to 'lite/host_agent.yaml '. But the "Error making API request' problem" not solved yet.

3. Good news: I've switched to 'qwen-vl-max' and increased 'MAX_TOKENS' to 6000 in 'config.yaml' file, finally get some reasonable results.
   Bad news: but due to the format of Qwen response is not consistent, always encounter different problems during 'parse_qwen_response’ process.
   --config
   
   --result1: without '\n' after 'Obeservation'、‘Thoughts’、etc.:
   

--result2: content of the 'Plan' in the following figure is not in the same line with 'Plan', thus cannot be splited

Now I understand why you say "The lite version of the prompt is not fully optimized. To achieve better results, it is recommended that users adjust the prompt according to performance!!!" in the model_worker/readme file. I'll try it. Thanks!

[yzhao666]

Hi @Mac0q ，
Thanks for quickly fixing the previous bugs so that I have the chance to taste more ablility of UFO.

Now I am stucked in the "WIN32COM is not supported" problem as follows:

In line 44 of 'ufo\automator\app_apis\factory.py', The 'win_com_client_mapping' list only cantains 'WINWORD' key, but no keys for other apps like EXCEL, POWERPNT, etc. It will not work when the task is not word-only. It seems that you only released WIN32 API service for WORD for now. Would you release API support for other apps?

As for the "Error making API request: Range of input length should be [1, 6000]" error, qwen-vl only support 6k context, so even if the WIN32 API issue were fixed, it seems that I have to use GPT-4V to test the complete functionality right?

[vyokky]

Hi @yzhao666 , "WinCOM is not supported" is only an warning, so it will not crash your program. I think in your case, the error is because the input token out of range so it does not generate the response. You may need to reduce the prompt size for QWEN.

I think Qwen is still weak for this task, we will try it optimize the prompt to make it doable, but GPT-4V is for sure the best choice.

[yzhao666]

Hi @yzhao666 , "WinCOM is not supported" is only an warning, so it will not crash your program. I think in your case, the error is because the input token out of range so it does not generate the response. You may need to reduce the prompt size for QWEN.

I think Qwen is still weak for this task, we will try it optimize the prompt to make it doable, but GPT-4V is for sure the best choice.

Hi @vyokky , thanks for your reply.
OK i see. Yes indeed the warning will not crash my program.
I thought the warning would not make my programe use the correct api to take action. Good to hear that it is not a problem.

So I will try to use GPT-4V instead.

Thank you and your team's effort again for making this great project :)