Muse: Mixed Reality Platform for Writers



Sponsored by Microsoft Research for their 2017 Design Expo

Creative storytelling platform enabling writers to explore inspirational worlds through mixed reality.


Timeline: 9 Weeks
My Role: Ideation, Video Prototyping


Microsoft Design Expo is an annual challenge among leading design schools to create a user experience prototype that solves a real-world problem. The 2017 theme was "Intentional Design for Positive Cultural Impact in Mixed Reality."

team members

Allison Chan
Olivia Thom
Margaret Tung
Maggie Wang
Madison Zeller

Muse is a storytelling engine that helps writers find inspiration by projecting 3D virtual scenes onto their environment. Users tap physical orbs representing the storytelling elements character, genre, and setting, and then verbally assign them tags that evoke the ensuing virtual visualizations.

Writers can issue further verbal commands to influence the holograms and the ways they interact with each other and the physical world. Upon finding inspiration, users can shrink the holographic world down to a miniature version that they can reference while they write.

Exploration & Scoping


Our journey began with an investigation into art therapy, creativity, and the underpinnings of imagination. We decided to narrow our focus on writers as an audience, but with the understanding and hope that our concept might benefit other artists or creators as well.

How might we use mixed reality to help writers find inspiration?



We read articles on the elements of storytelling, reflected on examples of storytelling done well in various forms (e.g., film, narrative, screenwriting), and interviewed artists and writers to gain insight on the creative process.

The takeaways:

It takes time for an image to ferment and see it line up with other images that seem like they could fit together as a story.

Elements of Storytelling

Successful stories are made up of five key elements:
Character, Setting, Genre, Plot, Conflict

"And then..." is boring. “But therefore...” is interesting. 

Write what you know.

I’d like to be in more environments that let me sit down and write something down and send it to myself to continue later.

The Creative Process

There is beauty in the seemingly mundane.

Moving around can stimulate ideas.

Writing is a process, and requires breaks and reflection.



We brainstormed various solutions and defined the concept for Muse, using storyboards and narrative to communicate our ideas and gain feedback. We then iterated and fleshed out key interaction moments.


Artwork: Allison and Maggie / Animation: Margie / Narrative: Olivia and Madison


Defining the interactions

Activating the system through voice command.

Activating the system through voice command.

Concept ideas for representing "scaled-down" holographic worlds.

Concept ideas for representing "scaled-down" holographic worlds.

A mixed reality scene in someone's living room.

A mixed reality scene in someone's living room.

While the storyboard featured early concept art to convey the magical and whimsical aspects of our idea, we had to buckle down and focus on what the concrete interactions with the system would look like. We wanted to give writers more autonomy in the process through the use of voice commands, allowing them to set the framework for and later interact with the holographic environment.

We pivoted away from possessions (such as the elephant statuette) as anchors, and instead decided on a set of three physical orbs as gateways into the system (see below). We wanted to highlight the benefits of mixed reality (versus virtual reality) by keeping the familiar home setting as a base, and by enabling the holographic world to be scaled down for easy observation.





Users assign tags to physical orbs representing the character, genre, and setting elements of a story. The two remaining storytelling elements, plot and conflict, are left to the writers’ imagination.

Mixed Reality

The imaginative world that unfolds integrates with users’ surroundings. Users stay grounded in a familiar setting while holograms bring fresh perspectives and inspiration.



Verbal commands can alter and influence the holographic elements evoked by the anchors, allowing users to shape, interact with, and dynamically change the mixed reality scenarios.


Through a simple hand gesture (drawing a bounding box on a surface), users can shrink their virtual environment and position it next to them as a reference while they write. This flexibility allows writers to take their holographic world with them to serve as their muse.

Video Prototyping


To showcase our vision and the nature of the 3D hologram design, we produced a video prototype. It features a writer experiencing a creative block who activates the Muse system in order to gain inspiration. In his mixed reality, a story is born.


My Contributions:

I wrote the video's narrative story, serving as one example of what Muse can help inspire:


Cal loved climbing trees so that he could look down at the world below him. But one day, he decided to look up. He climbed the trees up and up until he reached outer space. He had traveled so far and so long, only to realize the world above was much like the one below, the glow of the sun mirroring that of the campfire, the glistening stars reflecting the sparkles in the lake. Cal had been looking up for such a long time that he forgot how long he had been climbing. Finally, he looked down, and what he saw was like nothing he had seen before... 



what i would do differently...

  • Allot more time for learning the tools: Cinema4D (for 3D modeling) and Adobe After Effects (for video animation and compositing) are complex software, and in hindsight, I would have started my exploration with them in Week 1.
  • More primary research: I would interview a wider range of writers (creative, non-fiction) to gain a broader perspective on the writing process.

next steps...

  • Build out the UI. This is tricky since the MR landscape is still developing and requires a lot of imagination. I would want to create more visuals (still and/or animated) showcasing user interactions with the system, as our video mainly focuses on the overall aesthetic concept.
  • Scale to other creative users (e.g., visual artists, musicians).