Apple has silently introduced a feature called Enhanced Visual Search that's toggled on by default and works by sending information to the company's server.
Enhanced Visual Search in Photos allows you to search for photos using landmarks or points of interest. Your device privately matches places in your photos to a global index Apple maintains on our servers. We apply homomorphic encryption and differential privacy, and use an OHTTP relay that hides IP address. This prevents Apple from learning about the information in your photos.
Enhanced Visual Search lets your phone identify landmarks in your photos and search for those pictures using the name of the landmarks. | Image Credit - Apple
The Enhanced Visual Search setting was rolled out with Apple's latest operating system versions for Mac (macOS 15) and iPhones (iOS 18), which were released on September 16.
Enhanced Visual Search is enabled by default on iPhones and Macs running the latest OS. | Image Credit - Lapcat Software.
However, Apple first mentioned Enhanced Visual Search only on October 24, and that too discreetly. The feature allows your iPhone to identify photos of places, or most specifically landmarks, in your photo library.
Enhanced Visual Search for photos, which allows a user to search their photo library for specific locations, like landmarks and points of interest, is an illustrative example of a useful feature powered by combining ML with HE and private server lookups. Using PNNS, a user’s device privately queries a global index of popular landmarks and points of interest maintained by Apple to find approximate matches for places depicted in their photo library.
Apple, October 2024
The Verge explains that the feature will identify a specific location when you swipe up on a photo you have taken of a building and tap "Look Up Landmark." A card will pop up with the name of the landmark and a link to a related article.
To determine the name of any given landmark, your iPhone will use a global index of well-known landmarks and points of interest kept by Apple to find matches for places in your album.
An on-device machine learning (ML) model analyzes a photo and decide if there is a region of interest (ROI) that may include a landmark. If the answer is affirmative, an encrypted request is sent to Apple's server, where it carries out a bunch of processes, such as applying homomorphic encryption and differential privacy, and sends back candidate landmarks to the iPhone. An on-device reranking model then takes over and predicts the best candidate.
Recommended Stories
The photo's metadata is then updated with the landmark label, allowing you to easily find it by searching for its name on your iPhone.
While there can be many use cases for the feature and many may find it useful, it shouldn't have been enabled by default, considering it sends information related to your photo to Apple.
Making matters worse is the fact that Apple never publicly announced the feature. Also, in case you are wondering, the feature is not a part of Apple Intelligence. I found the setting on my iPhone 14 Pro (and disabled it right away), which cannot run Apple Intelligence.
Create a free account and join our vibrant community
Register to enjoy the full PhoneArena experience. Here’s what you get with your PhoneArena account:
Anam Hamid is a computer scientist turned tech journalist who has a keen interest in the tech world, with a particular focus on smartphones and tablets. She has previously written for Android Headlines and has also been a ghostwriter for several tech and car publications. Anam is not a tech hoarder and believes in using her gadgets for as long as possible. She is concerned about smartphone addiction and its impact on future generations, but she also appreciates the convenience that phones have brought into our lives. Anam is excited about technological advancements like folding screens and under-display sensors, and she often wonders about the future of technology. She values the overall experience of a device more than its individual specs and admires companies that deliver durable, high-quality products. In her free time, Anam enjoys reading, scrolling through Reddit and Instagram, and occasionally refreshing her programming skills through tutorials.
Recommended Stories
Loading Comments...
COMMENT
All comments need to comply with our
Community Guidelines
Phonearena comments rules
A discussion is a place, where people can voice their opinion, no matter if it
is positive, neutral or negative. However, when posting, one must stay true to the topic, and not just share some
random thoughts, which are not directly related to the matter.
Things that are NOT allowed:
Off-topic talk - you must stick to the subject of discussion
Offensive, hate speech - if you want to say something, say it politely
Spam/Advertisements - these posts are deleted
Multiple accounts - one person can have only one account
Impersonations and offensive nicknames - these accounts get banned
Moderation is done by humans. We try to be as objective as possible and moderate with zero bias. If you think a
post should be moderated - please, report it.
Have a question about the rules or why you have been moderated/limited/banned? Please,
contact us.
FCC OKs Cingular\'s purchase of AT&T Wireless
Things that are NOT allowed: