System Bits: Dec. 3

Tagging is a way of organizing information and today’s System Bits looks at two aspects of tagging. The first is an algorithm associated with social media and the second about how the brain geotags information.

New Algorithm Finds You, Even in Untagged Photos
You cannot hide on the Internet anymore. Anything posted online can be found, although in the past photos that were untagged may have been missed. Not anymore, thanks to a new algorithm designed at the University of Toronto.

Developed by Parham Aarabi, a professor in The Edward S. Rogers Sr. Department of Electrical & Computer Engineering, and his former Master’s student, Ron Appel, the search tool uses tag locations to quantify relationships between individuals, even those not tagged in any given photo.

Imagine you and your mother are pictured together building a sandcastle at the beach. You’re both tagged in the photo quite close together. In the next photo, you and your father are eating watermelon. You’re both tagged. Because of your close ‘tagging’ relationship with both your mother in the first picture and your father in the second, the algorithm can determine that a relationship exists between those two and quantify how strong it may be.

In a third photo, you fly a kite with both parents, but only your mother is tagged. Given the strength of your ‘tagging’ relationship with your parents, when you search for photos of your father the algorithm can return the untagged photo because of the very high likelihood he’s pictured.

Source: University of Toronto.

“Two things are happening: We understand relationships, and we can search images better,” says Professor Aarabi.

The nimble algorithm, called relational social image search, achieves high reliability without using computationally intensive object- or facial-recognition software.

“If you want to search a trillion photos, normally that takes at least a trillion operations. It’s based on the number of photos you have,” says Aarabi. “Facebook has almost half a trillion photos, but a billion users—it’s almost a 500 order of magnitude difference. Our algorithm is simply based on the number of tags, not on the number of photos, which makes it more efficient to search than standard approaches.”

Currently the algorithm’s interface is primarily for research, but Aarabi aims to see it incorporated on the back-end of large image databases or social networks. “I envision the interface would be exactly like you use Facebook search—for users, nothing would change. They would just get better results,” says Aarabi.

While testing the algorithm, Aarabi and Appel discovered an unforeseen application: a new way to generate maps. They tagged a few photographs of buildings around the University of Toronto and ran them through the system with a bunch of untagged campus photos. “The result we got was of almost a pseudo-map of the campus from all these photos we had taken, which was very interesting,” says Aarabi.

Memories Are ‘Geotagged’ With Spatial Information
Having a pseudo map of the campus could enable a video game in which people navigate through a virtual town delivering objects to specific locations. Using this approach, a team of neuroscientists from the University of Pennsylvania and Freiburg University have discovered how brain cells encode spatial information “geotags” for specific memories and are activated immediately before those memories are recalled.

Their work shows how spatial information is incorporated into memories and why remembering an experience can quickly bring to mind other events that happened in the same place.

“These findings provide the first direct neural evidence for the idea that the human memory system tags memories with information about where and when they were formed and that the act of recall involves the reinstatement of these tags,” said Michael Kahana, professor of psychology in Penn’s School of Arts and Sciences.

A still from the game participants played. They made deliveries to stores, then were asked to recall what they had delivered. (Credit: University of Pennsylvania)

Kahana and his colleagues have long conducted research with epilepsy patients who have electrodes implanted in their brains as part of their treatment. The electrodes directly capture electrical activity from throughout the brain while the patients participate in experiments from their hospital beds.

As with earlier spatial memory experiments conducted by Kahana’s group, this study involved playing a simple video game on a bedside computer. The game in this experiment involved making deliveries to stores in a virtual city. The participants were first given a period where they were allowed to freely explore the city and learn the stores’ locations. When the game began, participants were only instructed where their next stop was, without being told what they were delivering. After they reached their destination, the game would reveal the item that had been delivered, and then give the participant their next stop.

After 13 deliveries, the screen went blank and participants were asked to remember and name as many of the items they had delivered in the order they came to mind.
This allowed the researchers to correlate the neural activation associated with the formation of spatial memories (the locations of the stores) and the recall of episodic memories: (the list of items that had been delivered).

“A challenge in studying memory in naturalistic settings is that we cannot create a realistic experience where the experimenter retains control over and can measure every aspect of what the participant does and sees. Virtual reality solves that problem,” Kahana said. “Having these patients play our games allows us to record every action they take in the game and to measure the responses of neurons both during spatial navigation and then later during verbal recall.”

By asking participants to recall the items they delivered instead of the stores they visited, the researchers could test whether their spatial memory systems were being activated even when episodic memories were being accessed. The map-like nature of the neurons associated with spatial memory made this comparison possible.

Using the brain recordings generated while the participants navigated the city, the researchers were able to develop a neural map that corresponded to the city’s layout. As participants passed by a particular store, the researchers correlated their spatial memory of that location with the pattern of place cell activation recorded. To avoid confounding the episodic memories of the items delivered with the spatial memory of a store’s location, the researchers excluded trips that were directly to or from that store when placing it on the neural map.

With maps of place cell activations in hand, the researchers were able to cross-reference each participant’s spatial memories as they accessed their episodic memories of the delivered items. The researchers found that the neurons associated with a particular region of the map activated immediately before a participant named the item that was delivered to a store in that region.