Facial recognition search is here — and it is really scary
I have looked into facial recognition for consumers, and the results are impressive. What used to be in the domain of fiction or only accessible to law-enforcement has now fully entered the consumer domain. This article is structured into three parts:
1. Face search — Finding online pictures of the same face
2. Face search as a feature of Google and Facebook products
3. Future considerations
- General outlook
- Impact of deep fakes
- Face recognition by governments and major internet companies
1. Face search
The promise of face search is that you can enter a particular image containing a face, and get other publicly available images with the same face. I looked at a few different providers, but the most advanced one seems to be PimEyes.
The page works like a Google search, but instead of a keyword, it requires an input face. There are three options for an input face, either via upload, via picture from webcam or via a URL of an existing image. In order to make sure that the image is not already posted anywhere online, I decide to shoot a new one from my webcam:
The image is distilled to a small version of the face, and this face is then checked against any images that are posted online with a similar facial structure. In my case, the first three entries are actually different pictures of myself online:
The algorithm seems to go from the best match to successively worse matches, so after the real matches are over, it moves to people who look a bit like me, but are not actually me.
In trying out the service I found that adding more pictures of the same face (from slightly different angles) leads to much improved outcomes, and up to 4 images can be used in one search.
I have been consistently impressed by the accuracy of the matches. If there was a “true match”, it was usually listed in the top three entries, and it allows for major changes in the face (teenager vs adult, making funny faces, beard / no beard etc).
On the negative side, the data sources seem to be limited. It does take into account many public websites, forums and some networking sites such as Xing. However, the biggest sources of images, LinkedIn and Facebook, are not indexed. This means that for many searches, it will not find a match (because the images are not indexed), even though a Google search might find it.
On another — and much more scary — note, what it does find are pictures that are not connected to the name of a person. This could for example be group pictures containing the person in question, pictures in places where the name is obscured (e.g. modeling databases), or pictures that should not be trivially findable (e.g. erotic pictures).
The business model of PimEyes is a subscription model, where the images are shown, but the source of the images are hidden. Each day of openly displaying the source costs €11,39, but this price seems to fluctuate wildly.
A trivial way to avoid these fees is Google reverse image search. Taking a screenshot and pasting the image into a regular Google image search query shows where this exact image is being posted:
In my case, the image alone was enough to get to my name, and also Google managed to find all the other instances, where this image has been posted. The combination of PimEyes and Google reverse image search results in essentially free image searches.
2. Feature of Google and Facebook products
Both Google and Facebook have extremely powerful face search algorithms as well. Google has utilized this in the search functions for Google Photos (which is the free, cloud-based standard backup solution for android phones).
In the search bar, along with a couple of suggested search terms (usually location based with common image locations), it suggests some faces to search:
(I anonymized the search terms, along with all suggested faces apart from mine and my parents’ cat)
The results are impressive, all faces that Google suggests as my face, are actually pictures of me. For images where it is unsure (e.g. face is obstructed by sunglasses), it suggests confirming that this is really me:
Again, all of these suggestions were actually me, even though some of them were hard to recognize for anyone who did not know the context.
Facebook has been doing something similar for years, with the feature of suggested friends for tagging.
3. Future considerations
The technology seems to be very stable and extremely reliable. All components are freely available, both the pictures that are indexed by Google can be accessed and processed by anyone with enough storage space, and the facial matching algorithms are public domain. Even though Google and Facebook have so far not offered it as a public feature (likely due to possible legal conflicts), I expect that the technical feasibility of it and the functional attractiveness will lead some players to offer it and users to start using it. The most relevant aspect here is that the face search results yield additional information that could not be otherwise found with a Google search.
First surges in adoption seem to be already happening:
Market leader PimEyes seems to be gaining some popularity in the last few months, even though much of the traffic seems to come from Algeria, Tunisia and Egypt, for some reasons unclear to me.
As with many technologies, the earliest adopters are often the most questionable ones, but over time there will be a lot of use-cases.
Some of the use-cases I imagine in the future:
- HR department undertaking a background check on applicants (pictures on nazi demonstrations, wild party pictures etc)
- Single users finding out more information of a potential partner if they just have a picture (think Tinder, Bumble or Spotted)
- Fans trying to find any additional material from their favourite Youtube, TikTok or OnlyFans star
- People wanting to find out more information about friends, acquaintances or spouses (stalking)
- Attendance tracking for events
The advent of this technology is a major attack on privacy, because some of the information that was previously online, but not trivially usable, becomes usable and can be connected with existing information. Having one picture of one person linked to the name, links all online pictures of this person to it. This in turn means that protecting pictures of one’s face and making sure that only images that one is comfortable with are appearing anywhere online is paramount. In the event of the written name being used in bad situations online, in addition to having it deleted, one can change their name or use a nickname going forward. The face is almost unchangeable and will be linked to other pictures of this face forever.
Impact of deep fakes
In conjunction with facial recognition, deep fakes (“replacing a person in an existing image or video with someone else’s likeness”) have also made major strides. There are many tools and platforms out there that allow swapping faces from one person to another. I used a relatively simple, free online tool called Reflect.tech (https://reflect.tech/faceswap) to transpose my face onto the body of gang leader Tommy Shelby (from BBC’s glorious Peaky Blinders):
If the face becomes more relevant for searches, then deep fakes become increasingly dangerous, particularly in the short term. As of now, pictures of a person are still seen as reality. The possibility of “Photoshopping” an image has been around for years, but the sheer complexity of creating something halfway believable has meant that most pictures online are depicting the person originally photographed. Through the ease of use of Deep Fakes, this might change. People with ill intentions can create fake pictures which are then easily found online. In the long run, deep fakes can be a blessing in disguise for purposes of privacy. If anyone can fake a picture within seconds, and many pictures are already fake, then the relevance of pictures overall will decrease. Instead of taking an uncharacteristic picture of someone at face value and accepting it as reality, deep fakes could lead to a general skepticism towards any pictures and ultimately reign in the worrying aspects of face recognition.
Face recognition by governments and major internet companies
Most of this post was focused on face searches performed by private consumers. The examples from Google or Facebook show that this is a major topic for them, too, and the internal use cases can be manifold (to improve understanding of the user-base and display hyper-targeted ads). Likewise, governments are very advanced in face recognition. All of these have access to extremely rich databases — Google and Facebook through their digital offerings, governments through mandatory high-quality biometric images that are specifically created for face recognition purpose — making the results even more accurate. While it seems possible to reduce the footprint of one’s face in the public web, resisting these searches will be extremely difficult, particularly since the extent of the usage is not fully known.
I hope that an increasing awareness of the technology will also lead to stronger demands for auditing of these non-public uses of facial recognition technology.
