Introducing AR face filters

Face filters are well-known everywhere now — Instagram, Facebook, Snapchat... They started being popular 5 years ago, and almost every photo app has this feature today.

Fun fact is that 6 years ago, when there was no such thing as face filters, a similar concept created and designed by me and implemented by my team, won an international Intel AR challenge. 

This was quite an interesting journey.

I always wanted to work with new emerging technologies and got a great opportunity to design user experience for augmented reality video camera. In summer 2014 Intel announced RealSense Challenge — worldwide contest with a prize of $1.000.000, and more than 1000 teams all over the world competing for it.

The competition was sponsored and organized by Intel to showcase the capabilities of its RealSense technology which combines hardware, software, and 3D-camera to allow users to interact with their devices and environment via natural and human interfaces such as hand movements, facial expression, and speech.

IDEA

The first competition stage was called "Idea".

Participants were to suggest a concept of an application using RealSense 3D videocamera that can track facial expression, voice, gestures, and distance.

I wanted to appeal to people's emotions — with a concept of AR videochat add-on. It tracks users’ emotions, gestures, and words during the live call and changes the background behind the user to playful animations.

DEVELOPMENT

My concept was noticed by the jury, and along with other 1000 participants it passed to the next stage of the RealSense Challenge — "Development".

I was leading design and managing a team of enthusiastic engineers who gathered around my idea to build a working demo of “Emma” app.

For MVP we selected following factors to react: 

• 6 basic emotions (Joy, Sadness, Anger, Surprise, Fear, Disgust),

• 2 gestures (Thumbs up and Thumbs down), and

• 2 voice commands (Wow, Okay).

DETAILED VISUAL

I created different animations for each tracked item based the position of user’s head.

SKYPE INTEGRATION

We managed to integrate application with Skype, and the face masks became available during the real-time video Skype chat.

The main product constraint was that application worked only on the devices with integrated Intel Realsense 3D-camera or on machines with Windows 9 OS and plugged in Intel Realsense USB web-camera, which limited our target audience a lot.

USER TESTING

I tested application with over 100+ people.

It appeared that every person is unique in the way they express emotions. Some people have vivid mimic, and it’s easy to track their facial expressions, while others are less expressive — and that’s totally normal. Knowing this, it was clear that it would be hard to use app during the live time videocall.

On the other note, people expressed great interest to observing themselves with AR filters and playing with different faces. That proved that this concept might become popular in the nearest future – and so it did! Snapchat, MSQRD, Instagram, Messenger and other players started to widely use face filters that depend on mimic (open mouse, head movement, etc).

RESULTS

Judges were evaluating intuitive design and great visual (20% of the final score), leverage of technology capabilities (20%), meeting user expectations and reliability (20%), readiness for market (20%), and creativity/innovation (20%).

“Emma” and our team won the 2nd place competing with 1,300 international projects.

After the win, our project got great publicity. We took part in multiple exhibitions (including Microsoft, who wanted to continue collaborating with us on Skype integration).

I was asked to be a speaker on various UX conferences, sharing my learning on designing for AR.

Finally, I received O-1 US visa that stands for “extraordinary abilities in UX” and is given to recipients of internationally recognized awards.

That was a great journey that turned out from an opportunity to practice UX skills to bringing me where I am now.