Automate web and desktop apps with computer use - Microsoft Copilot Studio (original) (raw)

Computer use is a tool in Copilot Studio that lets your agent interact with and automate tasks on a Windows computer. It works with websites and desktop apps by selecting buttons, choosing menus, and entering text into fields on the screen. Describe in natural language what you want computer use to do, and it performs the task on a computer you set up by using a virtual mouse and keyboard. By using computer use, agents can complete tasks even when there's no API to connect directly to the system. If a person can use an app or website, computer use can too. You can use computer use for tasks like automated data entry, invoice processing, and data extraction.

Computer use is powered by Computer-Using Agents (CUA), an AI model that combines vision capabilities with advanced reasoning to interact with graphical user interfaces (GUIs). Because it's AI-powered, it adapts to interface changes. For example, when buttons or screens change, the tool continues working without breaking your flow. It's simple to use. Just describe what you want in natural language. You don't need to write code.

Watch this video to learn how the computer use tool lets an agent interact with a web application.

Add computer use to your agent

In Copilot Studio, add computer use as a tool by following these steps:

1. Go to the Tools page for your agent and select Add tool.
2. In Add tool, select New tool.
3. Select Computer use.
4. Provide the instructions that describe the task the tool should perform on the computer. Instruction templates are available to help you get started. Follow best practices for instructions for computer use.
5. Select Add and configure. On the configuration page, configure these four fields:
   • Name: Enter the display name for the computer use tool. This name helps you differentiate it from other tools you add to your agent.
   • Description: Enter a short description of what this tool does and when to use it. This text lets your agent know when to use this tool.
   • Model: Select the model used to execute the computer use tool. The following models are available:

     Provider	Model	Tier	Status
     OpenAI	Computer-Using Agent (CUA)	Standard	Generally available
     Anthropic	Claude Sonnet 4.5	Standard	Generally available
     Anthropic	Claude Sonnet 4.6	Standard	Experimental
     Anthropic	Claude Opus 4.6	Premium	Experimental

   • Instructions: List the steps the tool should perform, including URLs and application names. Follow best practices for instructions for computer use.
6. Optionally, review and adjust the other fields available on the configuration page.
7. Select Save.

Configure optional settings

The configuration page includes optional fields beyond the four required fields (Name, Description, Model, Instructions). Review and configure these options as needed for your scenario.

Inputs

Use inputs to define dynamic values that change each time computer use runs. For example, if you want to fill out a form with a different value on every run, create an input for that field. At execution time, computer use combines your instructions with the input values to complete the task.

Machine

Select the target machine that the agent uses to run computer use. Learn more in Configure where computer use runs.

• Select Refresh to update the machine list.
• Select Manage machines to open the machine management page in Power Automate.
• Select See machine details to open the machine details page in Power Automate.

Connection

Connection used for this tool. Update or create a new connection to change the credentials used.

Credentials to use

Specify how computer use authenticates during execution:

• Maker-provided credentials (default): Use the maker's credentials. This option is suitable for autonomous agents.
  Warning
  If you share an agent with this setting, anyone using it can act with the original author's access on the configured machine.
• End user credentials: Use the credentials of the person interacting with the agent. Each user must have access credentials to the machine.

Human supervision

Specify who should be contacted via email (Outlook) if the computer-use agent detects potentially harmful instructions that could alter model behavior. Each agent run and its activity is tied to the user who initiated it. If you choose a reviewer other than the person running the computer-use agent, they likely don't see the activity because they didn't initiate the run. Therefore, they can't properly verify or act on the request. Also, ensure the recipient is authorized and has the necessary context to handle such requests. The response time limit defines how long the request remains active. After this period, the request expires, and the computer-use run stops if no response is received. Learn more in Human supervision.

Stored credentials

Define the credentials that computer use uses to sign in to websites and applications. During execution, if a sign-in prompt appears, computer use securely uses any credentials you defined in this section for that site or application. Passwords are either stored in Power Platform internal storage (no configuration required) or in an Azure Key Vault that you provide. Learn more in Create a key vault using the Azure portal.

• Internal storage option: By using this option, you can configure credentials in the tool without any preconfiguration. Power Platform encrypts and stores secrets internally. For each credential, provide the following details:
  • Type: select between Website and Desktop app depending on which surface you plan to use this credential
  • Username: the username you use to sign in to the target website or application.
  • Password: the password value used to sign in to the target website or application.
    Note
    Password fields are supported on all websites and most Windows applications (WinForms, WPF, UWP, WinUI, Win32), which covers most customer scenarios. Some app types, such as Electron, Java, Unity, games, command-line interfaces, Citrix, or other virtualized environments, might not be supported.
  • Login domain (websites) or desktop app name (desktop applications):
    * Websites: Enter the main website address (for example, example.com). You automatically include all pages on that website. You can also use wildcards (*) for subdomains. Examples: www.contoso.com, *.contoso.com, contoso.com
    * Desktop applications: Enter the application product name or process name. To find it, press Ctrl+Shift+Esc to open Task Manager. Then check the Processes tab. Examples: Microsoft Edge, msedge, Notepad.
• Azure Key Vault option: By using this option, you can configure credentials in the tool by first entering the subscription ID, resource group name, and key vault name. You can find all this information on the Overview page for your key vault.
  • To use Azure Key Vault secrets with Power Platform:
    * The Azure subscription that has the vault must have the PowerPlatform resource provider registered.
    * The user who creates the environment variable must have appropriate permissions to the Azure Key Vault resource.
  • If you didn't already, follow the steps in Configure Azure Key Vault and then provide the following details:
    * Username: the username you use to sign in to the target website or application.
    * Azure secret name: the name of the secret in the key vault that stores the password for the website or application
    * Login domain (websites) or desktop app name (desktop applications):
    * Websites: Enter the main website address (for example, example.com). You automatically include all pages on that website. You can also use wildcards (*) for subdomains. Examples: www.contoso.com, *.contoso.com, contoso.com
    * Desktop applications: Enter the application product name or process name. To find it, press Ctrl+Shift+Esc to open Task Manager. Then check the Processes tab. Examples: Microsoft Edge, msedge, Notepad.

Access control

By default, computer use can operate on any website or application. If you want to restrict this access, enable access control to define the specific URLs and desktop applications that computer use should be limited to. You can configure both websites and applications:

• Websites: Enter the main website address (for example, example.com). You automatically include all pages on that website. You can also use wildcards (*) for subdomains. Examples: www.contoso.com, *.contoso.com, contoso.com
• Desktop applications: Enter the application product name or process name. To find it, press Ctrl+Shift+Esc to open Task Manager. Then check the Processes tab. Examples: Microsoft Edge, msedge, Notepad.

Note

Access control only prevents the model from taking actions on websites or applications that aren't in the allow list. It doesn't stop the model from opening them. For example, if only microsoft.com and Microsoft Edge are in the allow list, the model can still use the Microsoft Edge search bar to open Bing. However, once Bing is opened, any attempt to interact with it fails because it isn't in the allow list.

Test computer use

Testing computer use is a key step in the authoring journey. After you enter a name, description, and instructions, and save the tool, select Test to start the test experience.

After a short loading period, the test experience appears:

• The left panel shows your instructions and a step-by-step log of the tool's reasoning and actions.
• The right panel shows a preview of the actions on the machine you set up for computer use.

When the task finishes, a Test completed message appears. While the test is in progress, you can select Stop testing to immediately stop all actions on the machine.

If the result isn't what you expect, go to the configuration page and refine your instructions. Add more detail to improve accuracy. Follow best practices for writing effective instructions.

Publish an agent with computer use

Set up computer use, and then publish your agent. How your agent runs depends on the scenario. It can be autonomous or conversational:

• Autonomous agents run automatically and perform tasks in the background.
• Conversational agents let users interact through channels like Microsoft Teams.

Computer use works best for autonomous agents, performing tasks in the background without user interaction.

You can also apply computer use in conversational experiences, but keep these considerations in mind:

• If you select User authentication as the authentication setting, each user interacting with the agent in a conversation needs valid credentials for the machine used by computer use.
• When the tool runs, it shares reasoning messages and screenshots of the machine's activity in the chat.

Best practices

To stay productive and safe in today's digital environments, follow best practices for computer use, especially when writing clear instructions and securing machines.

Best practices for securing machines

When setting up machines for computer use that lets AI perform tasks by using natural language, consider these security recommendations:

Best practices for instructions for computer use

The instructions you write determine how well computer use works. Specific, detailed instructions help computer use complete tasks accurately. Think of it as explaining a task to a colleague. Clear, step-by-step guidance helps ensure success.

Tips for writing effective instructions:

• Be specific about websites and applications. Always include the full URL of any website and the exact name of any application the tool should use. Example: Open https://www.microsoft.com and go to 'Company news'.
• Clearly state relevant actions. If you want something done, say it explicitly—especially for actions like submitting a form or sending an email. Example: Once you fill in the form, select Submit. No need to ask for permission.
• Break down complex interactions. For areas where the UI might be more complex to navigate, explain each step in detail. Example: Select the More icon in the top right corner. A dropdown appears. Once it opens, select the last item in the list.
• Use step-by-step formatting for longer tasks. Long instructions are easier to follow when you format them as a list.

Sample instructions

Explore these sample instructions to try out computer use or use them as a reference for writing your own.

Use computer use to extract data from websites or applications. Then, pass that data to your agent or to other tools in the agent. Describe what information you want to extract directly in the instructions for computer use.

If you want to use the extracted data with another tool (for example, sending it by email), specify that need in the agent instructions and make sure you add both tools to the agent. Example: Use computer use to extract the client information for the financial portfolio dashboard and email the extracted data. In this case, your agent must include both the computer use tool and the email tool.

Instruct computer use to extract specific values and return them as plain text. For example, the following prompt extracts the portfolio manager and portfolio value for the client, Fourth Coffee:

1. Go to https://computerusedemos.blob.core.windows.net/web/Portfolio/index.html.
1. Find the row for Fourth Coffee and record the Portfolio Manager name and the current Portfolio Value exactly as shown.
1. Return those two values as the final output.

The agent returns only the two requested values as text.

You can also ask computer use to return extracted information in JSON format. This format is useful when you want to pass structured data to another tool. The following example extracts all portfolio rows related to Contoso and returns them as properly formatted JSON:

Navigate to https://computerusedemos.blob.core.windows.net/web/Portfolio/index.html, retrieve the portfolio details for all Contoso entities, and return the results as a valid JSON object.

Structure the output so that:
*    Each top-level key is the client name
*    Each value contains the client's portfolio ID, portfolio value, portfolio manager, and last updated date (format: YYYY-MM-DD)
*    Return only the JSON, with no additional text.

This approach ensures the output is structured.

Licensing

Computer use bills usage through the Agent action feature with a billing rate of five Copilot Credits. Learn more in Billing rate and management.

Each computer use run relies on an AI model that executes a sequence of steps. A step might involve one or more low-level actions, such as clicking, typing, or navigating. Each step consumes 5 Copilot Credits, or 15 Copilot Credits when using a premium model.

The following models are available for computer use:

For example, if you configure computer use to fill out a web-based time-sheet form, the run might perform the following steps:

1. Launch the browser and go to the time-sheet portal.
2. Select Create new time sheet.
3. Fill in the Start Time, End Time, and Project Code fields.
4. Select Submit.

In this example, the computer use run executes four steps. If it uses a standard model, this process consumes a total of 20 Copilot Credits. If it uses a premium model, it consumes a total of 60 Copilot Credits.

Do you have feedback about computer use? Let us know at computeruse-feedback@microsoft.com.

• FAQ for the computer use tool
• Add standalone computer use tools to agents and agent flows (preview)