Maltego Part 5. Your face betrayed you.

Технології розпізнавання облич від Social Links

Social Links and Maltego are a powerful combination as many data sources (including OpenCorporates, DocumentCloud and Pipl) are integrated into one tool. Additional benefits include the ability to search social networks using face recognition and plotting results on a network graph. But before you try uploading your identities there to see how they work, consider the risks involved. Markup’s Blacklight tool, which identifies user tracking technologies on websites, discovered a “session recording” tool on Findclone, meaning they can track your mouse movements and keystrokes.

Let’s start

While testing the functionality of searching for information in various social networks using Maltego (see Part 4), I was surprised that Maltego immediately found my LinkedIn account with just my name and a photo from an Instagram account. This is where, I believe, the Face Recognition mechanism from Social Links worked. You can’t just take such a topic and pass! We need to make sure how accurate the functionality is and whether it can be fully applied in OSINT, or if my case is an isolated one, and the issue will be clogged with similar people.

We will test it on various media personalities. First, they definitely have social media accounts. There should also be fakes with their photos. Secondly, we will be able to visually determine whether there is an error in recognition. Anyone can already distinguish Donald Trump from any other person. If I’ve said anything, let it be Donald Trump. For the first time, we will consider the procedure in detail, according to the instructions from Social Links, for all other cases we will only analyze the results. The claimed accuracy percentage from SL on recognition is 75%, which is a lot.

Approx. Author: there is one not very pleasant parameter in Maltego’s working mechanisms. It is called “Timeout” and is equal to 2 minutes. It defines how long Maltego will wait for a response from a third-party API when performing Transforms. There is an unpleasant moment here. This option can be increased in the settings, but there is no mechanism for Transform developers to find out what it has installed. And this means that when developing third-party transforms, companies like Social Links are forced to rely on the default value. Why it is so important, you have either already guessed, or you will find out in the conclusions at the end of the article.

Donald Trump

Go to the Social Links account and create a person’s profile there. We indicate the first and last name, country and gender.

We add a photo. We will choose both ordinary and not quite

Everything is ready. You can start!

For search, we will use Entity – Search Profile by Face. We uploaded the photo to the SL account in order to get a convenient direct link to it. But if it is already in another place, then you can use it.

After starting the search on the first photo, it became clear how the system works as a whole. The neural network responsible for recognizing people in Social Links makes a request to the social network through the API to search for all accounts by the given first and last name. After that, based on the photos of these accounts, it makes a comparison through face recognition with the photo that we specified as a sample.

So, according to the front photo of Trump, we have:

• Facebook – 1 account (not enough)

• LinkedIn – 1 account (not his, but Trump is in the photo with the owner of the account)

• Instagram – 8 accounts

• VKontakte – 8 accounts

• Twitter – 7 accounts

• MySpace – 3 accounts

• Foursquare – 6 accounts

It is also worth noting that the system was able to identify Trump’s face even in photo collages and shared photos.

According to the results of working with the first photo, there were no misses on accounts with random photos. After starting the search for photos 2 and 3, several additional accounts appeared. There are no left pictures, as before. All the accounts listed on the graph contain images of Trump in one form or another. Even photoshop art did not slip by.

So, after running Transforms on all reference photos, we have:

• Facebook – 2 accounts (Still not enough)

• LinkedIn – 1 account

• Instagram – 10 accounts

• VKontakte – 8 accounts

• Twitter – 10 accounts

• MySpace – 7 accounts

• Foursquare – 7 accounts

Only the LinkedIN account should be considered a mistake in this case, because it belongs to Blaine Kelly (Blaine Kelly), but in the photo he is depicted together with Trump, so, in fact, it is 30 for 70. Trump is still there. The summary graph is on the screen below.

Zemfira Ramazanova

Now let’s try to recognize a woman’s face. Everything is honest, no bias.

Zemfira’s fakes, of course, are fewer than Donald Trump’s fakes, accordingly, the information distribution is cleaner.

Together we managed to find:

• Facebook – 5 accounts

• Instagram – 3 accounts

• VKontakte – 2 accounts

Random photos of Zemfira-like people have not been noticed. I believe that this is a consequence of the fact that the primary sample goes to the full name. And if there is an account with the name of Zemfir and an image of a left-handed person or a picture, it will already be screened out by the face recognition system.

Jackie Chan

Let’s complicate the task. Let’s check how the system will work on an actor with a pronounced Asian appearance. I specially chose for the test both pictures of the artist in his old age and his photo in the prime of his career. Photo for the test:

Together we managed to find:

• Facebook – 9 accounts

• Instagram – 8 accounts

• VKontakte – 6 accounts

• Twitter – 4 accounts

There are no random pictures of old Asians. Information delivery is quite accurate. If you don’t count the piles of fakes with Jackie Chan’s name and photo. As you can see, accounts with Jackie’s photo at any age do equally well.

Thandie Newton

And finally, let’s check how the system will work when searching for photos of black people. Here, for the search, I chose the actress Thandie Newton, known to all of us for her roles in such films as “The Chronicles of Riddick”, “Rock and Roll”, “Han Solo”. Star Wars: The Stories and, of course, Wild West.

Together we managed to find:

• Facebook – 2 accounts

• Instagram – 3 accounts

• Twitter – 6 accounts

• MySpace – 5 accounts

There are no mistakes in recognition.

Summary and conclusions

Let’s summarize. Based on the results of my tests and reading the documentation on the work of Transforms with the Face Recognition mechanism from Social Links, I can safely say that they work perfectly. While writing this article, I even managed to close one OSINT project using them.

However, there are a few BUTs to keep in mind when working with them:

1. Due to the large number of photos on social networks and the presence of a “Timeout” option in the Maltego settings, we cannot simply take and scan all the photos in the selected site for a match. First, it would take too much time, and second, we are limited to a 2 minute response window from the SL servers, otherwise Maltego will drop the connection and terminate the Transforms.

All of the above leads to the fact that before the phase of direct recognition of the person of the search object, we have to do a preparatory stage. It appears in the pre-selection of social network accounts by some parameter. In our case, it was automated in Transform – Search Profile by Face and Name in the form of a sample of a person’s first and last name.

This stage can be performed in different ways, and as a sampling parameter can be taken, for example, geolocation (specific place, hometown, work address, etc.)

2. In case of any obvious discrepancy of the face in the photo, the account is cut off from the sample. This can be caused not only by the fact that the person in the photo is not the one in the sample. Recognition is seriously hindered by the presence of glasses and hats.

3. Another factor that interferes with the correct operation of the face recognition mechanism is the degree of “shaking” of the photo. Both the original sample and the photo on the account from the sample. So if you have a low-resolution source photo or such a photo on your account, then don’t expect good results. The resolution of the sample should be the higher the better, the desired file format is *.jpg, but if it is PNG, then the neural network will not be offended.

Approx. Author: By the way, in Moscow, a system of recognizing people through surveillance cameras is already being introduced in full swing in the city. Rumor has it that they will even launch drones for patrolling. It is good or bad – everyone decides for himself. Don’t miss new parts. In the next article, we’ll talk about how you can apply location information to OSINT using Maltego.