Continuum XL

October 15 2015

Keyboard vs Controller Cortana

The Story

Xbox may have lost the current generation console war to PlayStation. But Microsoft owns the most prevalent platform for gamers: Windows. For both the size of its user base, and its dominance in high-end and competitive gaming. Xbox, or Windows Gaming, is dying to get a bite of that much larger pie. We want PC gamers to buy games from the Windows Store, and play on Xbox Live.

Windows 10 came with support for game streaming. If you own an Xbox on the same network, you can already use it from any PC. Naturally, there comes the idea to push it further: what if the PC is the Xbox? What if the game is being streamed from the cloud? What if I can get the same gaming experience any time, anywhere, on any Windows device? One scenario is about a college student who can only afford one high-quality device. He uses his laptop for study as well as gaming. Another tells of a traveling business woman. She casts her tablet to the hotel TV and joins her nightly raid squad.

One of the experience gaps here is that Windows does not have a system level “10-foot” story. This was how Continuum XL started.

Making Sense

When I came on board, an incubation team had already introduced their solution: when a controller is plugged in, Windows desktop disappears and the Xbox dashboard shows up. There are two questions that they face whenever they present:

There was a lot of confusion and struggle among the people who worked on this project. After lengthy discussions I suggested that instead of a single mode switch (Windows vs Xbox), the problem space really concerned three:

Three Mode Switches

Gaming covers all input modalities, and a TV is definitely not necessary. The Xbox dashboard only covers where all three overlap. If we bring the Xbox UI to Windows as-is, we will miss big opportunities. I declared my mission to be designing a consistent yet adaptive gaming experience across all user context and input modalities.

Patrick’s Answer

During the summer 2015 Continuum XL was made the challenge for our team’s UX design intern, Patrick Little. I worked as the lead of the track and Patrick’s mentor.

As we started storyboarding, we identified three possible directions for the implementation.

Implementation Models: Companion, Sandbox & Shell

We moved forward with the shell model and mocked up a controller-enabled Windows - using the task bar as the signature UI. If the user presses the Xenon button, a controller version of the task bar appears. If the user moves the cursor to the bottom of the screen, a keyboard/mouse version of the task bar appears. The “mode switch” is implicitly accomplished by enaging in one input modality or another.

Keyboard Mouse Task Bar vs Controller Task Bar Keyboard Mouse Start Menu vs Controller Start Menu Keyboard Mouse Social vs Controller Social

I built a prototype to demonstrate the interaction. The prototype runs in an overlay above the Windows desktop. I implemented the functionalities of launching, task switching, notifications, windows management, etc. It employed a great number of hacks with Win32 API and simulated keyboard shortcuts. It was one of the most complex prototypes I had done, mostly because I wanted it to feel realistic by controlling the actual desktop.

Journey On

One of the criticism of the above design was that it deviated from the design patterns of both Windows and Xbox. Some of our effort is around how we find connection between the two shell’s UI, and whether we can bring them closer together.

This is an ongoing exploration. I’m working closely with designers from the Windows team to explore a spectrum of ideas.