How To... Work Within the Constraints of Augmented Reality

"

At Nexus Studios we've been working on Augmented Reality projects for a while, some of our previous work in the field includes the Gruffalo Spotter for Magic Light / Forestry Commission England, the first AR cover for The New Yorker and the 1600 app for President Obama's administration.  

This summer Apple announced ARKit, an annoucement swiftly followed by Google's annoucement of ARCore, visual odometry-based solutions that meant that markers weren't necessary anymore, and suddenly we had access to a huge audience of potential users of Augmented Reality Apps. AR suddenly became a mass market medium. 

As content creators, we were super excited by this, but new mediums require new ways of thinking. Lots of new ways of thinking in fact. I spoke about this at yesterday's VR summit but here I'm focussing more closely on something I briefly touched on there, the problems inherent in attempting to believably augment reality. Most of these problems come from 'reality' itself. The real world is a challenging canvas for our ideas. 

One of the important things to consider with any AR experience is that the interaction isn't just between user and software, in an AR app there also needs to be a degree of interaction between digital content and the physical world, and this is tricky to do in a way that is believable to the users, we still have a lot to figure out. 

In many ways it reminds me of the early days of the web, the 'brochure web' days, when people created websites by uploaded existing print brochures, we're in the stage where people are taking existing 3D assets and plonking them into the world though AR without really understanding the contours of the medium.

This is a still from Apple's ARKit keynote at WWDC this summer. It's a dinosaur on a basketball court, and it's sort of impressive but it also isn't. This dinosaur doesn't look like it's really there at all. For example, the lighting and shadows on the dinosaur point in completely opposite directions to the basketball players.

I point this out only to show how difficult this stuff is.

That dinosaur is an excellent model and the tracking in ARKit is superb, but ARKit alone can't tell you what's happening in the scene you're augmenting. 

The illusions that we create in this medium are incredibly brittle, yet they too easily fall apart under scrutiny. The frameworks do an extraordinary amount of heavy lifting for us, but as illusionists we still need to put the work in. 

We've been working with ARKit since it became available and a few weeks ago Nexus released Hotstepper, our self-initiated attempt at putting some of the reality back into Augmented Reality. Hotstepper is your AR-wayfinding guide; you put in where you want to go and you are guided enthusiastically to your destination by a dancing, half naked, overweight man (below).

It's a deliberately silly app, but it really works as an app that takes the real world into account. 

There's a number of things that we're proud of but I'll only highlight one of them. We're especially proud of how the lighting reflects the environment you're in. ARKit provides you with an ambient color temperature for your physically-based models, but it can't really do much more than that on it's own. 

Hotstepper is location aware so because we know where you are on earth, we can look up the weather where you are. We know the position of the sun in relation to you and we can cast our shadows accordingly. 

Admittedly, we haven't solved every problem in AR yet - my personal favourite ever JIRA ticket is that ARKit can't yet tell the difference between solid ground and a calm sea.  

I'm really looking eager to see how we, as illusionists, can build on this platform and rise to the challenges of creating for this medium.