Pokemon Go created a 3D map of the world – but for what?
You may have seen the recent headlines about how a company called Niantic Spatial is using a database of real-world locations that was originally compiled by players of the mobile game Pokemon Go — a game that launched about a decade ago and quickly became an obsession for tens of millions of people, young and old. Niantic Spatial says it plans to use the 3D models and locations to make it easier for delivery robots to find specific addresses in order to drop off pizza or groceries or whatever they are doing. The company was spun off from Niantic last year, after it sold Pokemon Go and its other games to a company called Scopely, which is ultimately owned by the sovereign investment fund of Saudi Arabia. Pokemon Go and other Niantic games will still run on Niantic Spatial's data, but the world they helped build will be used for something else now.
Niantic Spatial, not surprisingly, is pretty excited about how all this data has helped the company build what it calls a Visual Position System, using more than a million location models as the core of its Spatial Platform. The company is also pitching its tools to companies that want to create virtual reality or augmented reality experiences that can be hosted online or beamed into VR and AR glasses from Meta and others. However, some current and former Pokemon Go players don't seem to love the idea that they and their smartphones were dragooned into a massive surveillance program that mapped real-world locations without their knowledge. "500 million people played pokemon go, scanned every street, building, and corner on earth [and] thought they were catching pikachu," said one user on X. "Niantic was building a 30 billion image AI map of the world now powering delivery robots... you were the product the whole time."
The company's website says that in addition to the database of more than 30 billion images and locations from Pokemon Go, it is also doing its own real-time three-dimensional data capture using drones and other technologies, so that its customers can generate models of their stores or warehouses or mining operations and what have you. In many cases, Niantic says, traditional global positioning systems don't provide enough granularity or detail within large cities for delivery robots to find specific physical addresses, but its location models can do so in almost every single major city in the world. "AI that understands the physical world," Niantic says. "We are building a living model of the world that people and machines can talk to." The Large Geospatial Model (LGM) is built on "real world data from ground and overhead sensors."
Coincidentally enough, there is a growing sense in the artificial intelligence sphere that while ingesting and then regurgitating massive quantities of text — which in a sense is all that traditional large-language models like ChatGPT and Claude do — is enough for some purposes, such as writing code or romance novels, it is not enough for understanding the world and how human beings operate within it on a physical level. This is a song that LLM critic Gary Marcus has been singing for some time, in a somewhat annoying but still correct way, and it seems that others have started coming to similar conclusions. Yann LeCun, for example — the Turing Award winner who helped create the field of modern AI, along with Geoffrey Hinton and Yoshua Bengio — recently left Meta, where he was chief data scientist, to start a new company called AMI Labs, which is focused on building real-world models. It recently closed a $1 billion financing round.
LeCun says that most human reasoning is grounded in the physical world, not in language, and that world models are necessary to develop true human-level intelligence. "The idea that you’re going to extend the capabilities of LLMs to the point that they’re going to have human-level intelligence is complete nonsense," he told Wired. AMI (pronounced like the French word for friend) said in a release that it plans to build a new kind of AIs that "understand the world, have persistent memory, can reason and plan, and are controllable and safe." LeCun says the new company intends to work with companies in manufacturing, biomedical, robotics, and other industries that have lots of data that could be used to train such models. For example, he told Wired that AMI could build a realistic world model of an aircraft engine and work with the manufacturer to help them optimize for efficiency, minimize emissions, or ensure reliability.
Note: In case you are a first-time reader, or you forgot that you signed up for this newsletter, this is The Torment Nexus. Thanks for reading! You can find out more about me and this newsletter in this post. This newsletter survives solely on your contributions, so please sign up for a paying subscription or visit my Patreon, which you can find here. I also publish a daily email newsletter of odd or interesting links called When The Going Gets Weird, which is here.
Google Earth and the CIA
If Pokemon Go! players had been more familiar with Niantic founder John Hanke's full curriculum vitae, they might not have been surprised by what their data wound up creating. The birthplace of what ultimately became Niantic was a company called Keyhole, which Hanke and a partner created in 1999, in the dying days of the first internet bubble. According to a book about those events titled Never Lost Again: The Google Mapping Revolution That Changed Everything, by former Google staffer Bill Kilday, Hanke showed up at his house one evening with a Dell PC and a monitor that showed a rotating Earth. That was cool enough, but what happened next was even better: Hanke asked Kilday for his address, and when he entered it the blue marble stopped rotating, spun the other direction, then the virtual camera zoomed in and showed a satellite view of Kilday's house from a couple of hundred feet up. It was like looking through Superman's eyes as he flies around the planet and zooms in to hover over your house.
Keyhole had a great technology, which it called EarthViewer, but there was no real business case yet, and so the company was in danger of going bankrupt (after the demo, Kilday's girlfriend reportedly said "but what do you do with it?) Luckily for Hanke, the CIA came along: the agency's investment arm, known as In-Q-Tel, invested some money in Keyhole, and started using the app internally to help researchers and agents track hostile entities, and study enemy installations and troop movements from a distance. Perhaps the CIA's interest was triggered by the fact that the agency's spy satellite program in the 1960s was also called Keyhole. Hanke also said that what became Keyhole was inspired by Neal Stephenson's book Snow Crash, in which the Central Intelligence Corporation created a virtual-reality program called Planet Earth to "keep track of all the maps, weather data, architectural plans, and satellite surveillance stuff."
In-Q-Tel made the investment in Keyhole alongside the National Geospatial-Intelligence Agency, a major intelligence organisation with 14,500 employees and a $5-billion budget, whose job was to deliver satellite-based intelligence to the CIA and the Pentagon. When the US invaded Iraq in 2003, Keyhole's EarthViewer became even more useful, not just for the CIA and the U.S. Army when it came to planning the invasion itself, but for the news media in explaining what was happening: CNN started using it to generate Superman-style video clips that zoomed in on various spots where bombs were going off, etc. That got Keyhole enough attention that it started working on a funding round, and then Google came along and offered to buy the whole company. Hanke said yes, and Keyhole became Google Earth. Here's how Kilday described it:
"It happened in a classic Google way. Sergey Brin, one of Google's co-founders, got a link to the Keyhole software. He was supposed to be in a product review meeting for Picasa, the photo app. But he was so mesmerized by Keyhole that he completely derailed the meeting. He hijacked the projector and just started flying around the globe, showing everyone their houses. So it's like Google was this giant, creative playground, and Sergey just found a new, shiny toy he had to have, and he had the money to buy it. He reportedly just turned to the room and said, "We should buy this company." And that was that."
If you're like me, you remember when Keyhole first came out, and how you spent hours just zooming in on different places — your house, where you grew up, cities you had never been to. After it was relaunched as Google Earth, I remember the newsroom I was working in ground to a halt for hours because so many people were playing around with it (the same thing happened with Google Streetview, as everyone tried to locate their own house). The fact that it seemed like a game was no accident: Hanke said he created Keyhole in part because he was hoping to use the 3D mapping element to build a game — find a specific location, etc. like a scavenger hunt. “Think about the games that you play: how many of them start by unfolding a game board — some kind of map?” he asked The Guardian in 2023. “The two just go together.” Before Keyhole, Hanke and friends created Meridian 59, one of the first massively multiplayer online games.
Location down to the centimeter
After being in charge of Google Earth and then Google map-related innovations like Streetview, Hanke said he felt the itch to get back to games, so he formed Niantic and came up with a real-world exploration game called Field Trip. Ingress followed soon after — a kind of alternate-reality game that I remember playing, where you got text messages and emails and you had to walk or travel to specific coordinates to get updates or find more information. “We wanted a game where you are seeing the world as you see it, but your mind would be able to add that extra layer of mystery," Hanke told The Guardian. That eventually gave way to Pokemon Go, which almost overnight became one of the biggest games — mobile or otherwise — in modern history: “Five hundred million people installed that app in 60 days,” Brian McClendon, CTO at Niantic Spatial told MIT's Tech Review. The game still had 100 million players in 2024.
It's obvious from seeing Hanke's comments about why he created Keyhole and EarthViewer and Streetview and Pokemon Go that he really loves combining real-world data with a kind of game element. And Pokemon Go was extremely fun! But now, that massive database of 30 billion location points and buildings and streets and so on has become a gold mine — and not just for Niantic Spatial, or for world-based AI software like Yann LeCun wants to build, or for 3D modelling so delivery robots can find their way around New York and drop off your shawarma at the right apartment. As I read the description of the company's abilities on its website, I tried to imagine what a senior-level official at In-Q-Tel or at the CIA, or the FBI, or Homeland Security might think as they were reading it. I think they would be as excited as when the CIA saw Keyhole, if not more.
Niantic says using its new world model, it can pinpoint a user's location on a map to within a few centimeters, based on the buildings and other surroundings. “We have a million-plus locations around the world where we can locate you precisely,” says McClendon. “We know where you’re standing within several centimeters of accuracy and, most importantly, where you’re looking.” It turns out that when those millions of Pokemon players were wandering around in search of Pikachu, the game was encouraging them to take photos of their surroundings from different angles, and each of these photos recorded not just a specific image or GPS location, but also all of the other data from the phone's gyroscope and other sensors — elevation, orientation, speed of travel, etc. In other words, hundreds of millions of people playing a game enabled data collection about the real world on a scale that had never even been contemplated before.
I can almost hear ICE agents salivating at the thought. Am I being paranoid? Perhaps. But we already know that these agencies are using whatever means at their disposal to track down anyone they believe might be either a) an immigrant or b) an antifa "agitator." They are using data from hundreds of thousands of license-plate cameras, doorbell cameras, facial-recognition cameras at airports and elsewhere, combined with databases of social-media users and whatever they can get through Palantir's "Surveillance OS," which combines federal databases of all kinds. The FBI has admitted that in addition to all of this, it is also buying commercially available data on American citizens from a number of sources, and according to recent court filings the agency believes that laws against collecting such information — or only doing so with a warrant — don't apply if the information being analyzed is publicly available.
I'm sure John Hanke has no intention of just handing over to Homeland Security or the CIA or FBI access to his locations database. But the Trump administration doesn't seem shy about forcing companies to play ball, or fining them or otherwise penalizing them if they don't do as they're told — like designating Anthropic a supply-chain risk because the company refused to allow its AI to be used to either pilot autonomous killer drones or to surveil American citizens without a warrant. How badly does it want access to 30 billion location parameters, along with data about which way a user was facing, and what speed they were travelling at? A virtual world with that kind of granularity would make Google Earth look like child's play. Hanke may have wanted to create a game, but he could wind up being part of a game he doesn't like — and so could his users.
Got any thoughts or comments? Feel free to either leave them here, or post them on Substack or on my website, or you can also reach me on Twitter, Threads, BlueSky or Mastodon. And thanks for being a reader.
