Cortana on Xbox

September 12 2014

Cortana is Microsoft’s personal digital assistant. In July 2014 Xbox assembled a cross-discipline team to look at how we could bring her to the console. I was the lead designer.

Cortana as shipped with Windows and Phone had an emphasize on productivity. The leadership was not super enthusiastic about the product at the time; they saw Cortana integration as a tax to pay to be part of the Windows family. From the start the team focused on identifying opportunities in marrying Cortana and the entertainment space. We met daily, filled a wall with sketches and stickers, looking for the magic story.

Cortana Early Sketches

Soon we scoped down to two areas of interest:

To better understand the experience, I started building a prototype in WPF. Speech recognition was handled using Bing speech API. The conversation flows were scripted, with some flexibility for natural understanding. The core interactive model formed during this exploration was the three forms of Cortana: Tiny Cortana or Cortini (a small, non-controller-interactive, assistive overlay); Cortana snap app (a secondary experience that provides content space and controller interactivity); Cortana full screen app (an immersive experience). She transforms automatically based on both the conversation (e.g. needing simple confirmation vs. list of search results) and the context (e.g. playing a game vs. idling in home).

The prototype stirred huge interests from the senior management. Cortana was listed as one of the user stories in the upcoming dashboard refresh, and rescheduled to 2015 from 2017. We were left a small window to complete the design.

Cortana was an universal Windows app that could execute on Xbox. Its current experience was built upon extensive user research and data analysis, so the Cortana team rightfully had strong convictions about how it should be. The areas where it needed much work were:

Starting November, 2014 the Xbox Shell Speech team worked closely with designers and engineers from Windows and the Cortana service team. We listed every flow that we’d like to cover. I wireframed every state of each flow. They would push back based on technical constraints or experience inconsistencies with Cortana on other platforms.

Cortana Search Flow

In May, 2015 the launch plan was signed off. From the design side I brought in content publishing, visual and motion to finalize the appearance of Cortana. The feature underwent several delays due to strategic planning. It rolled out to beta users in December, 2015 and is now scheduled for release in mid 2016.

Cortini Visual Redlines

Cortana Visual Redlines

user experience console natural user interface speech prototype C#