AR needs a visual API for the world

Vineet Devaiah -

What are we building at Teliportme ?

As you know TeliportMe today is the largest & most engaged social network for Immersive experiences, 9 million users across 160+ countries have contributed to build the most comprehensive user generated VR content database of the world. However, I have never done a good job of explaining our larger long term vision of building the visual API for the world. Our all consuming vision is to power the next generation of Augmented Reality devices that we believe will be the best form computation can take in the next 20years.

Its long been believed by science fiction and all the top computer scientist visionaries that there will come a day where computation becomes Minority reportisque. This means you will be able to interact in the real world and make contextual decisions quickly and accurately.

Critical to this contextual decision making in the real world is the requirement for information that is quick to capture & process; typing and voice input which have dominated computation till today are cumbersome and clunky. Visual information on the other hand contains significantly more data, can be processed many times faster and provides a non-intrusive user interface which is the reason why it is the primary sense that is used by the brain to make decisions.

More importantly VR/AR done right will make our real daily world more fun and exciting — we are already addicted to music, games, FB, snapchat etc. to keep ourselves occupied (ADHD !!); this behavior will only get worse and the addiction more self consuming thanks to games that encouraged this behavior (hello!! Clash of Clans!, Farmville, PokeMon Go). The below video by Keiichi Matsuda blurs real and virtual worlds in fascinating ways that will make you wonder how long will it be till you can live a world like that.

As you can see, the real world is mapped realtime using contextual information to what the person can see, this kind of mapping needs

  1. An existing database to match with (both internal or external environments)

  2. An Artificial Intelligence that can learn/predict real world textures trained on this massive real world database that continuously updates itself.

Just like a majority of the smartphone applications are built over a maps API, AR applications will be built over a visual API of the world. TeliportMe will build that visual API.

Building this real world visual API is far from trivial. All past internet/web/technology platforms that have existed in the virtual world consisted of information that at some point of time was added by humans. The same goes for 2D maps, cartography has existed since has man existed in some form or the other and bringing it to the web was not particularly a herculean task as most data existed in various distributed sources.

Unfortunately 3D data of the real world has never existed and you cannot write a bot/spider to crawl the real world to collect and update this information. Google ran a car across the world capturing immersive 360 content and created Google Streetview (though Lightfield data would have been more effective). Uber which acquired all assets of Microsofts’ mapping team and the head of Googles’ mapping team is looking to invest half a billion dollars to build a better maps product is using the same strategy. Unfortunately cars do not go to majority of places that humans spend most of their time in ( houses, offices, restaurants, malls, nature ). Today despite massive efforts only a small % of the worlds roads are mapped; while significant enough for a starting database it is far from sufficient or the right strategy. Current efforts by companies to mimic this by training a non-geotagged object database is a joke, creating 3D point clouds is nice for some places but this problem is bigger and solution more nuanced.

The best solution is to be able to combine all possible sources of real world 3D/360 image data in an efficient (cost & time) manner with an strong community of users who are constantly editing this database ( ala wikipedia ) in realtime. I believe that true consumer grade AR is still atleast a decade away. Most advanced AR/Mixed Reality devices (Magic/Meta/HoloLens) claim to work well ONLY in closed previously mapped out spaces because they are only a simple projection in a real world on a screen in front of your eyes. I strongly believe that this will change rapidly and like most technology shifts it will happen in the blink of an eye and we all will be Pokemon Go’ing our lifes.

The TeliportMe story and vision started in early 2011 when the smartphone revolution was taking over the entire computation industry; it was getting more certain that computation was ready to go mobile i.e out of your house. It was the idea that personal computation goes from an instructional, obstructive & deliberate system to a more contextual, seamless & automatic system. As someone who was part of the PC boom and then the smartphone avalanche; I wanted to build something that could push this computation wave ahead and success for me was the ability to be an important part of this technology revolution.

Below is a excerpt from an interview I had with Rediff in late 2012 and probably the only publication that published our vision of what we wanted to build.


(vision — 2012 )

Actually Adrianne Jeffries also covered us in 2012 in Verge and she also mentioned about our grand Augmented Reality plans in this article


(vision - 2012)

TeliportMe has always had a master plan .

  1. Create a strong community of billions of users across the world i.e highly engaged in capturing and sharing immersive imagery of the world using their smartphones. This community will also be centrally focused on improving empathy within people across cultures through immersive imagery.

  2. Build the worlds best product that enables publishers/creators to publish VR content across multiple platforms. Give them 100% control over their content.

  3. Structure data from both sources combined with visual imagery from open source projects, government institutions, failed mapping companies so as to create a workable pointcloud of the world running in congruence with a powerful AI over it to approximate missed out spaces.

  4. Build a profitable and impactful business around travel transactions, user subscriptions and SAAS products.

Step 1 has been a success, we are number 1 with Panorama 360 and we need to continue to foster and improve that community, bring more and more users to our community and work on integrations with different capture devices. But the next 2–3 years will be focused on step 2 and becoming number 1 in that space; VR is becoming mainstream and its the right time to unleash the barrage of products we have in our pipeline.

Thanks to everyone who helped me proofread and edit this post. If you want to find more about our company go to www.teliportme.com or better you can download our app Panorama360 for Android or iOS.