Microsoft wants to bring generative AI to the forefront of Windows – and the PCs it runs on.
During a pair of keynotes at the annual Build developer conference this week, the company unveiled a new line of Windows machines it's calling Copilot+ PCs, plus generative AI-powered features like Recall, which will let users find apps, files and other content they have in viewed the past. Copilot, Microsoft's brand of generative AI, will soon be much more deeply integrated into the Windows 11 experience. And new Microsoft Surface devices are on the way.
We have collected all the important announcements from Monday and Tuesday here.
Volumetric apps
Microsoft is bringing Windows Volumetric Apps – essentially spatially aware, interactive VR apps – to Meta Quest headsets. Through a partnership with Meta, Microsoft says it will deliver Windows 365 and local PC connectivity to Quest headsets, allowing developers to extend their apps into 3D space.
During Tuesday's keynote, Microsoft showed off a digitally exploded 3D view of an Xbox controller from the perspective of a Meta Quest 3 headset – a digital object that the wearer could manipulate with their hands. “We are deepening our partnership with Meta to make Windows a best-in-class experience on Quest devices,” Pavan Davuluri, CVP of Windows and Devices at Microsoft, said during the demo.
Developers can do that sign up for a preview today to access Microsoft's new Volumetric API.
Copilot+ PCs
Copilot+ PCs are Microsoft's take on AI-first, flagship Windows hardware. They all contain special chips called NPUs to power AI experiences like Recall. And they come with at least 16 GB of RAM, paired with SSD storage.
The first Copilot+ PCs will feature Qualcomm's Snapdragon Chipmakers Intel and AMD are also committed to building processors for Copilot+ devices in collaboration with a range of manufacturers, including Acer, Asus, Dell, HP, Lenovo and Samsung.
Copilot+ PCs start at $999, and some are available for pre-order starting today.
Surface Pro and Surface Laptop
Microsoft's newly unveiled Surface devices, the Surface Laptop and Surface Pro, focus on performance and battery.
The latest Surface Laptop – available with a 13.8- or 15-inch screen – has been redesigned with 'modern lines' and thinner screen bezels. It lasts up to 22 hours on a charge and is up to 86% faster than the Surface Laptop 5, the company says. It also supports Wi-Fi 7 and has a touchpad with haptic feedback.
As for the new Surface Pro, Microsoft says it's up to 90% faster than the previous-generation Surface Pro (the Surface Pro 9), and will feature a new OLED with HDR display, Wi-Fi 7 (and optional 5G ) and an improved ultra-wide front camera. Additionally, the detachable keyboard – which is reinforced with additional carbon fiber – now has haptic feedback.
To remind
Windows 11's upcoming Recall feature can “remember” apps and content a user visited on their PC weeks or even months ago, helping them, for example, find a Discord chat in which they discussed the clothes they wanted to buy. Users can use Recall's timeline to “scroll back” to see what they've worked on in the recent past and zoom into files such as PowerPoint presentations to surface information that may be relevant to their searches.
Microsoft says Recall can create associations between colors, images and more, allowing users to search for virtually anything on their PC using natural language (not unlike startup Rewind's technology); developers can improve Recall by adding contextual information to their apps. And Microsoft claims that all user data associated with Recall is kept private and on-device – and more importantly, not used to train AI models.
Here is more from Microsoft: “Your snapshots are yours; they remain local on your PC. You can delete individual snapshots, adjust and delete time ranges in Settings, or pause at any time, right from the notification area icon on your taskbar. You can also prevent apps and websites from ever being saved.”
Image editing and live translations
There's now more AI in Windows than ever, and some of it exclusive to the new Copilot+ PCs.
A new feature called Super Resolution can restore old photos by upscaling them automatically. And Copilot can now analyze images to give users ideas for creative compositions. Through a feature called Cocreator, users can generate images and also ask the AI model to track what they draw to change or restyle the image.
Elsewhere, Live Captions with live translations translates all audio passing through a PC (whether from YouTube or a local file) into the language of the user's choice. Live translations initially support approximately 40 languages, including English, Spanish, Mandarin and Russian.
A separate but related new feature in Microsoft Edge offers real-time video translation on sites like LinkedIn, YouTube, Coursera, Reuters, CNBC, Bloomberg, and more. The feature, which will be available in the near future, will translate spoken content live via both dubbing and subtitles.
Team Copilot and expansions
Team Copilot is the latest extension of Microsoft's growing Copilot suite of generative AI technology. It integrates with Teams, the company's video conferencing app, to help manage meeting agendas and create notes that everyone in a meeting can co-author. And it extends to Loop and Planner, Microsoft's collaboration and scheduling platforms, to create and assign tasks, track deadlines, and notify team members when their input is needed.
In somewhat related Copilot news, Microsoft has launched (in private preview) Copilot Extensions, allowing developers to extend GitHub's code generation tool GitHub Copilot with third-party apps and skills. Launch partners include DataStax, Docker and LambdaTest; extensions will be available on the GitHub Marketplace, but developers can also create their own private extensions to integrate with their internal systems and APIs.
Windows Copilot runtime
Capabilities like Recall and Super Resolution are powered by the Windows Copilot Runtime, a collection of ~40 generative AI models that are part of what Microsoft describes as “a new layer” of Windows. Together with the Semantic Index, a vector-based system local to an individual Copilot+ PC, the Windows Copilot Runtime allows generative AI-powered apps (including third-party apps) to run without necessarily requiring an internet connection.
“[The runtime] consists of ready-made AI APIs such as Studio Effects, Live Captions translations, OCR, User Activity Recall and [more]which will be available to developers in June,” Davuluri said on Tuesday.
Microsoft says CapCut, the popular video editor from TikTok owner ByteDance, will use the Windows Copilot Runtime and its new Windows Copilot Library, a set of APIs and AI development tools, to accelerate its AI features. And Meta will add the aforementioned Studio Effects to WhatsApp to provide features like background blur and eye contact during video calls.
Improved bot builders
Azure AI Studio, the toolset within Microsoft's Azure OpenAI Service that lets customers combine an AI model and build an app that “reasons” on that data, will soon allow developers to create apps using pay-as -you-go' inference APIs – the APIs that allow developers to access and refine generative AI models hosted on Azure infrastructure. Microsoft calls this 'model-as-a-service' and will start with models from Nixtla and Core42.
In the adjacent Copilot Studio product suite, Microsoft is launching Copilot agents, which the company describes as AI bots that can “independently orchestrate tasks tailored to specific roles and functions.” (Copilot Studio provides tools to connect Copilot for Microsoft 365, the AI-powered “copilot” in apps like Excel and Word, to third-party data.) Using memory and context knowledge, Copilot agents can navigate through different navigate types of business workflows, learn from user feedback, and ask for help when they encounter situations they don't know how to handle.
Snapdragon developer kit
There's a new dev kit from Qualcomm aimed at developers building apps for Copilot+ PCs with arm chips
The $899.99 Snapdragon Dev Kit for Windows (which, by the way, measures about the same width, height, and length as Apple's Mac Mini) includes Qualcomm's Snapdragon IO. The Dev Kit supports Wi-Fi 7 and Bluetooth 5.4, and can control up to three 4K monitors simultaneously via the various USB-C and HDMI ports.
Phi-3
Microsoft has announced a new addition to its Phi generative AI model family, Phi-3-vision, which can perform common visual analysis and reasoning tasks such as answering questions about graphs and images. The model can read both text and images and is efficient enough to run on a mobile device.
Phi-3-vision is available in preview, while the previously announced counterparts to the text-only model – Phi-3-mini, Phi-3-small and Phi-3-medium – are now generally available.
Collaboration with Khan Academy
Microsoft has partnered with Khan Academy to donate access to cloud computing infrastructure, allowing Khan Academy to offer teachers in the US free access to Khan Academy's AI-powered tools. The two companies will also work together to explore opportunities to improve AI apps for math tutoring through generative AI, Microsoft announced Tuesday.
We're launching an AI newsletter! To register here to receive it in your inbox from June 5.
