GazeMaps as a Directorial tool in VR:

Youtube has been pioneering video based VR more than any other platform. As of now, it’s safe to say they are way ahead of other VR video players such as Vimeo, Jwplayer and Brightcove. Both, in audience-facing and back-end features.

While some of the other players don’t yet have proper support for stereoscopic video VR, Youtube has forged ahead with spatial audio formats and seamless switching of their player depending on platform used.

In the back-end, it now gets even more exciting with Youtube offering for free, their heatmap analytics tool. As an indie filmmaker (who also has to do marketing) this is an invaluable tool to showcase real “eyeballs” engagement to potential sponsors.

Let’s start with using this tool from a storytelling and Director’s point of view. Before releasing a VR film to the general public, this is the workflow I’d follow from now on:

  • Upload small test sections or a rough cut of the film to Youtube and set the film url to private or unlisted in youtube’s settings.
  • Invite my “screener audience” by giving them the URL  – I marvel at how patterns have changed from physically going to a cinema screening.
  • Wait for Youtube to have enough views to build the Heatmap for the film.
  • Go back to the edit table and reposition cuts, “dwell time” and Point-of-interest. 

…then release the film, satisfied in the knowledge that the VR film will be twice as effective with all the big data crunching and distilling that’s performed. To be fair – there might already be offline gaze mapping tools out there for quicker data gathering.

We’ll dissect my 2015 graphic novel themed VR film, Dirrogate, to gain some insights:

To follow along with the time-stamps, use the pause button on the youtube player, above. Times stamps mentioned below are from the Youtube player timeline (not the embedded timestamps inside the video)

00:00 – 00:30

I’ve realized in VR, people need enough time to acclimatize to a new scene – The establishing shot – in a VR film could safely be twice as long as in a made-for-Imax movie, as the gaze map shows.

 00:30 – 1:15 

It’s only after about 30 seconds into the video that people start looking around the environment. As we approach the 1 minute mark, it’s safe to assume that because the visuals are not changing… people might start to get bored and are looking around more frequently. The heatmap shows this. However – based on the heat index (red-yellow) it looks like a good number of people really don’t move their heads around much at all.

For this establishing scene, in hindsight now, I’d go back and shorten it as I feel it’s starting to drag. An interesting point to consider is – what if the buildings had people moving in the windows – what if it were a live action drone shot, through a city? I’d wager:

  • There’d be red-yellow hotspots all over the place – Which we could conclude might be from multiple viewings of the same scene! – Imagine what this does for brand placement (Bladerunner billboards anyone?)
  • No risk of boredom kicking in if the establishing scene was over a minute and a half long.
  • A lot of Directors are thinking about the possibilities the medium offers over a traditional smaller FOV “framed” narrative.

01:25 – 01:30

Watch how the heatmap converges on the woman as she appears on the balcony. Back in early 2015, there was no support for head-tracked ambisonic audio, otherwise true directional audio would be the storytelling tool that Directors would use when coming in blind, from a cut in VR. I used simple panning of the voice from right to left ear during the course of the dialog.

01:30 – 02:40

Notice how the heatmap rarely diverges from the centre? My theory is this:

In VR, audiences still want to sit back and passively absorb a story – but should they wish to, they want have at their disposal, the ability to look over their shoulders; to cement their presence in that location!

Light, makes for a great directorial cue. A tool to usher audience attention. The man on the bed is (a) a person versus just an object and (b) is lit. These aspects work toward directing audience attention, gaze and retaining their interest.

I base this theory on the sporadic quick spots on the heatmap showing up in these scenes. It also can be inferred that people (actors) will take higher precedence over anything else in a scene.

Pause the previous video & cue up the video above, to 02:45 mark on this one for convenience.

02:45 – 02:55

This scene is important and illustrates two directorial insights:

  • People will look around, if motivated to do so.
  • The actor walking in an arc is deliberate staging – using the 360 canvas to good effect.

In VR, you earn the privilege of seeing the actor lost in thought, up close, and he doesn’t know you’re there! It’s not the same intimacy even on an Imax screen.

  • VR gives you a wide canvas to paint your story on and in this case – the audience turning around to see the main character, lends itself well to creating that feeling of “presence” – of being in the movie, in that bedroom.

How I wish back in early 2015 I had access to budgets, live actors and high quality stereoscopic cameras (It’s worth mentioning – even back then, Dirrogate was produced in stereoscopic 360 with compositing being checked one frame at a time due to no mature tool-sets being available)

03:18 – 03:28

People will be people, and you can see the *ahem* line-of-sight, shifting subtly. It also again shows that the human form, when seen in VR is the most compelling experience of all.

03:31 – 03:38

Again, shows audiences will rarely look around, and:

  • We need more ‘dwell time‘ when coming in from a cut.
  • Unless the scene is of long duration – it’s better to have branding or product placement within the front field of view.
  • At some point people will look over their shoulders, and if there’s anything of importance it’s best to use audio cues to guide their attention.
  • If a scene is of long duration, reward audiences with easter eggs, and eye candy when they turn around (An old coca cola ad from the ’70s playing on a billboard etc) This can lead to multiple viewings of the film.

05:30 – 05:42

Shows how in a blackout the audience is disoriented and looks around – ideal for a jump scare perhaps, or a reminder to get on with the story. The heatmap quickly converges back on the main subjects, and more importantly stays there, even though they are not moving. The audience barely looks around the scene except for quick darting glances – helping them establish they are in the apartment.

Note: The scene is stereoscopic 360, so audiences don’t feel like furniture is 40 feet tall. ‘Spatiality’ of the scene is maintained, and it adds to establishing presence.

07:03 – 07:08

We see how audiences do a quick ‘once over’ before the gaze (heatmap) settles in, where it does. Can marketers infer or profile the audience this way? Perhaps.

When producing video based VR experiences, Youtube’s heatmap is turning out to be a much needed tool in crafting compelling narratives in Virtual Reality.

Suggestion to Youtube: Make a downloadable heatmap overlayed video option available to content producers to work with, offline. That would be great.