LDV Vision Summit ’18: 10 questions to an AR research scientist : Facebook

With the fifth edition of the LDV Vision summit right around the corner ( May 23-24, discount tickets available on the right =>), we approached some of the speakers to get a hint of what they are working on and what they will be speaking about. First of the series is Krishnan Ramnath, Research Science Manager and Technical Lead Mobile at Facebook whose work  for the social media platform is pioneering how most humans will experience AR in the years to come:

– A little about you, what is your background?

I joined Facebook in 2016 as a Research Scientist in the Camera team. Currently, I am a Research Science Manager and Technical Lead, driving our Augmented Reality efforts on mobile. Prior to Facebook, I spent 6 years at Microsoft Research working on various applications of Computer Vision such as burst photo analysis, automatic storytelling and image captioning. I also held Research Scientist positions at Amazon and ObjectVideo, after graduating with a Masters in Robotics from Carnegie Mellon University.

My research is primarily motivated by use cases that provide interesting media experiences for our consumers. I derive a lot of personal satisfaction when I see computer vision technology adding value to people’s lives. It has been an amazing journey since my undergraduate studies when I first worked on face recognition, to now with deep learning and augmented reality applications. I am excited about further technical breakthroughs to come that will drive innovative products.

– Building AR on Facebook seems like a tremendous challenge it seems. In short, you have to adapt the existing platform for building, reading and experiencing AR. What is your approach?

Building AR on a social media platform that focuses on messaging and sharing mechanisms requires a lot of innovation. However, I do believe that at Facebook that we have a unique opportunity with the platform and the tools that we already have. Mobile devices will continue to be a predominant avenue for experiencing AR until specialized devices primarily built for AR consumption see widespread adoption and become ubiquitous. However, this is still a few years out so we need to focus on building enhanced AR experiences on mobile devices.

People come to the Facebook app every day to connect and share with others. We add a lot of value by integrating AR experiences as more creative means of expression into people’s existing workflows on our platform. We also believe in giving 3rd party artists and creators the power to define new experiences and we do this by enabling tools such as AR Studio. Through this, we are able to provide a wide range of experiences on our platform to our community. We also focus on building algorithms that can run on a broad array of devices to enable AR. We will continue to be the largest AR platform by focusing on providing value in everyday interaction experiences.

– You have to consider a variety of viewing devices with major differences in hardware. How do you work around such challenges? Do you work with camera manufacturers?

The key to deploying AR experiences on a wide variety of hardware is to dynamically adapt the algorithms’ execution based on different hardware capabilities. The considerations while designing these algorithms are three-fold: first is modeling camera characteristics such as rolling shutter artifacts, lens distortion, and motion blur; second is fusing different sensing modalities such as camera and inertial sensors; and finally, adapting to the device capabilities related to memory, processing power, and storage. We knew from the get-go that a one-size-fits-all approach would result in compromised quality, so our solution was to design a framework that allows us to run multiple algorithms with different performance characteristics and gracefully switch between them online to account for device capability variations. Optimizing across the different axes such as processing power, memory, and quality is a challenging task, and we provide the highest quality experience based on these parameters through this process.

– Who is your main target audience when building AR on Facebook, Consumer, developers, advertisers? Are all their needs the same?

Our primary goal is to build for our community. Our use cases are around helping build more meaningful relationships through creative tools for self-expression. But we realize that the way to empower novel experiences is to open up the platform to creators, and not limit the creation process to just ourselves. We have a strong creator network that is using AR Studio and building amazing experiences for our community. Through our creation and upload process, we have a pipeline that helps with fast delivery of these 3rd party AR effects to consumer devices.

Our goal is to help improve these tools and enable these new capabilities that ultimately ends up in driving value. We also showcase experiences with brand partners, which achieves three goals: 1) we provide novel experiences for our community 2) we show our creators how to use our latest tools and enhancements 3) we help drive brand value for our partners by increasing engagement and mindshare on their brands. An example of this is what we did with our movie partnerships for Ready Player One and A Wrinkle in Time for the target tracking capability in AR Studio.

– Do you see AR taking over 360 or Videos in popularity on Facebook?

Our goal at Facebook is to bring the world closer together and help form meaningful relationships. We want to enable novel capabilities and tools whereby the friction in this process is reduced and I see AR as a new medium for enabling these experiences. I believe that AR is still in its nascent stages and will probably take time to surpass existing usage on videos, but I am hopeful for a future where interacting with AR becomes as natural a behavior as flipping through photos or watching a video.

– How do you use AI to help in the AR experience?

I do believe that AI can help enhance the capabilities that we are building across the board and streamline many of the existing pipelines that we have today. AR experiences require you to analyze, understand the world around you and render interesting experiences on top of them. A lot of our recognition algorithms are powered by the latest machine learning algorithms which helps us understand faces, scenes, and objects to accurately overlay AR. High fidelity AR experiences are possible due to our investments in AI techniques for scene understanding and pattern recognition.

– You recently opened the AR studio on Facebook to everyone. Can you tell us more about it?

AR Studio is our customizable suite of creative tools for Mac that enable building custom effects for Facebook camera. One of our goals with AR Studio is to put the power of augmented reality in the hands of all creators. We want artists, developers, and brands to be able to build and share amazing AR experiences. Over the last few months, we’ve expanded the AR Studio toolset to build and share AR experiences with the Facebook community. By opening AR Studio to all creators, we’re taking steps toward making AR more a part of everyday life. We’ve been continuously advancing the capabilities of the platform with improved face tracking, new graphics capabilities, better scripting and World Effects. This is only the beginning, and we can’t wait to see how creators bring art to life through AR Studio.

– How is the adoption in the first few months? Have you seen some amazing usage that you could share with us?

We are happy to say that we have had tremendous interest and adoption of our creative tools by creators across the world. We have an enthusiastic and active creator community that use our tools daily and provide us with valuable feedback which we use to help make these tools better. We hope by increasing the reach of the platform and enhancing the value we deliver, we can further increase the adoption of our tools.

– What are your sources of inspiration in the AR space currently? Who do you think is paving the way?

I truly believe that at Facebook we are paving the way forward in many cases when it comes to AR. We are tackling interesting challenges whether it comes to democratizing the creation process and building algorithms that run on a wide variety of devices with incredible reach. Our investments in Oculus will pave the way for future mixed reality experiences where people can consume AR and VR on novel consumer devices. I also believe in the charter of enhancing the personal connections and bringing people closer through AR experiences that Facebook is uniquely equipped to deliver.

Having said that I also think the industry as a whole is helping drive AR adoption. Snapchat has very cool AR filters and effects. ARKit by Apple and ARCore by Google are helping developers build AR on their respective platforms which further drive adoption. I am excited about the collective progress we are making in this space.

– What are you looking forward the most while attending the LDV Vision Summit?

I really look forward to attending the LDV Vision Summit because it is a unique blend of industry and VC folks, creators, computer vision researchers, and engineers all in one venue. I like the format of the conference as well in terms of short talks and plenty of opportunities to connect and mingle. Last year we had a lot of focus on self-driving car technology and we heard a lot in that space. This year, I am interested to see more about deep learning, AI and augmented reality.

– What would you like to see Facebook AR offer that technology cannot yet deliver?

I believe in a future where AR is available all the time and is ubiquitous. While the exact wearable hardware configuration that delivers these capabilities successfully is to be determined, I believe that we would be experiencing and interacting with AR content seamlessly in the near future. I hope to see varied applications across different domains that are all created for and delivered in Augmented Reality.

 

Kaptur is a proud media sponsor of the LDV Vision Summit. You can get an exclusive 40% discount by using the code “KAPTUR”.