Copilot Vision Is Watching You But That Might Not Be a Bad Thing

Microsoft talked a lot about Copilot Vision at its ‘Windows 50 Years’ event a couple of weeks ago. In case you’re not familiar with the technology, Vision allows Copilot to see exactly what you can see on your screen. If you’re thinking “what could possibly go wrong with that?” – considering the controversy over Windows Recall, I wouldn’t blame you.

But Copilot Vision has been planned better from the get-go. You have to opt in every time you want Copilot’s help, it’s not watching every single thing that you’re doing. You have to actively turn it on and then you only get to select a specific window on your screen that it can see. There are other systems that are comparable to Copilot Vision. For instance, Google AI studio has similar functionality.

Copilot Vision could change the way you interact with your PC

Copilot Vision is a technology that has the potential to change the way you work with Windows. Imagine a future where you could tell it what to do using natural language rather than using a keyboard and a mouse. Imagine you’ve got an Excel spreadsheet. You need to perform a particular task but you’re not sure how to do it. You could turn on Copilot Vision and ask it to help you solve the problem. It will tell you where to click and guide you through to a solution. 

It could even replace training, how-to articles, and video tutorials. It’s like having an assistant standing behind you telling you what to do and how to do it. There might even be a point where it just does the task for you rather than it telling you to click here, click there.

There are many potential ways this technology could help you make decisions faster and get stuff done by allowing it to see exactly what you can see. In some of the demos Microsoft gave, they showed Copilot Vision looking at a website where you could ask it, using natural language, specific questions to find a particular product on Amazon. You could ask it to find the right section of a website, look at all the different options, and choose which is best for a particular scenario.

Copilot Vision free for Microsoft Edge users in the U.S.

Copilot Vision is a paid option for Copilot Pro subscribers today. It’s not part of Copilot for Microsoft 365 at this stage as it’s likely Microsoft is testing Copilot Vision out with consumers first, perfecting  it, and then releasing it to commercial customers at a later point.

Microsoft did announce this week that they’re making Copilot Vision free for Microsoft Edge users in the United States. And that it’s only going to work with a limited number of websites, Amazon being one of them.

Search in Copilot for Windows now available for Windows Insiders

Windows Insiders are getting access to file search in Copilot for Windows. Search is one of the areas where AI can help boost productivity. We know that Microsoft is adding semantic search to Windows and the ability to search using natural language. But if you’re a Windows Insider, the Copilot for Windows app now allows you to search your device.

Search is an opt-in feature. There are two opt-ins:

1. You can allow it to search your files.
2. And you can also allow it to read your files.

If you search for a file and it returns a result, you can ask it questions about your files (providing you allow read access), a feature that is available today in Copilot for Microsoft 365 that I’ve found to be helpful.

We can expect search in Copilot for Windows to roll out to the stable channel at some point over the next few months.

It could be confusing to have search available in so many different places in Windows but providing it works the same way in Windows Search and Copilot for Windows, I don’t think it will be an issue. The problems start when there’s a lack of consistency between how search works across different entry points.