Article Archive Details
Hendon Publishing

Facial Recognition – Art or Science?

Facial recognition technology is a critical tool used at our country’s borders every day. Federal agents compare passport photos with the scanned image in their passport photo database to ensure the document has not been falsified. Many times these agents perform a second level or tier verification to verify what was already put on the screen.


How Facial Recognition Works

As with most facial recognition implementations, our Homeland Security system requires a combination of technology and human interaction. Facial recognition technology is often misunderstood both by the public and law enforcement.

Hollywood would have you believe facial recognition technology allows law enforcement to identify an individual from an image while simultaneously accessing all of their personal information. While the Big Brother narrative may make for good television, it is a vast misrepresentation of how facial recognition is actually used by law enforcement agencies.

Think of facial recognition as the 21st century evolution of the sketch artist. It holds the promise to be much more accurate, but in the end, it is still just one lead that needs to be verified and followed up by intelligent police work.

The four key components to the facial recognition (FR) system are: 1) camera captures an image, 2) an algorithm creates a Face Print or Facial Template, 3) this template is stored in a database of many stored face prints or facial templates, and 4) probe to template matching, i.e., facial recognition compares the captured image against the database.


Image Analysis

Once an image is submitted for facial recognition analysis, the analyst or investigator’s initial responsibility is to evaluate or triage the probe image. They must answer two questions. First, does the probe image meet the criteria for facial recognition searching? Second, does the probe image need pre-processing for image enhancements?

If the answer to the first question is Yes, then the image is classified as a controlled or high-quality image, and proceeds to facial recognition searching. If the answer is No, then the image is most likely uncontrolled, and of lower quality, meaning that one or more factors exist that make the probe image ineffective for facial recognition searching.

In the past, an uncontrolled classification meant the investigator could not conduct a facial recognition search. Today, new approaches in image pre-processing, and easy-to-use enhancement tools, make it possible to enhance lower quality images for facial recognition searching. These enhancements are changing the paradigm in the facial recognition process by expanding its effectiveness as a lead generator in the space of public safety.

Once a probe image has met the criteria for facial recognition, either initially or through image enhancements, it is enrolled by the investigator into the facial recognition application for searching against any available galleries from the agency and/or commercial sources.


Search-Image Quality

For an analyst or investigator enrolling a probe, it is important to align matching accuracy expectations with the initial image quality of the probe. When enrolling a high-resolution probe, the candidate will often return higher in the candidate return list. When enrolling a low- to medium-quality probe, the return will most certainly reside deeper in the candidate list.

Why? Images with lower resolution have data loss. When data is missing, it significantly impacts the ranking. Simply put, more data equals a higher ranking; less data equals a lower ranking. Because of this, an analyst should never expect a Top 10 ranking on lower quality probes.

When this occurs, an analyst should do a number of things. Start by expanding the gallery and standardizing the return list between 200–500 candidates as a candidate may reside deeper in the return list. Then, use filters that are built into the GUI of most systems. Filters read the metadata behind each gallery image. They also act as a process of elimination for larger databases. When looking for a known gender, race, or location, applying filters to a search only helps to narrow a list of candidates, and substantially drives the results to smaller groups of specificity.

Properly using filters for lower resolution probes only increases the likelihood of obtaining a possible match residing in your list of candidates because you are leveraging the metadata to work in tandem with facial recognition algorithms. When the probe is of lower quality, the analyst should be aware of a need to become more actively involved in the facial identification process.


The Human Aspect

When particular faces become of interest to the analyst or investigator, the analysis process becomes more detailed and methodical. Performing a visual scan of faces automatically eliminates candidates quickly, allowing the analyst or investigator to navigate through larger candidate lists rather easily. Several strategies and areas of focus are recommended to more efficiently make use of time during the identification process.

Analyze the ears and hairline on all the returned candidates. Ears are unique. Lobe shapes on frontal images are easily identifiable and lobe patterns on profile images are as distinctive as the fingerprints on a hand. Even receding hairlines maintain levels of consistency with patterns, and fuller sets of hair have unique parts, widow’s peaks, or may display a particular ethnic hair type. 

Divide a face into four quadrants. Top left, top right, bottom left, and bottom right. When conducting a visual comparison of probe to candidate, the analyst or investigator must carefully review both, looking for similarities and differences in each.

Look for ‘locks’ or certainties that may exist between probe and candidate. These validate the physical characteristics between both images, and assist in the identification process during peer review. These locks may be found in disfigurements, scars, moles, piercings, hairlines, tattoos, etc.

Once satisfied with a particular candidate, an immediate background investigation is advised. After the candidate’s physical characteristics have been satisfied in the identification process, the validation process begins. Check for incarceration status, criminal background checks, residences in relation to the crime location, and modus operandi. Careful review and analysis of these factors strengthens the investigation and solidifies or discredits your potential possible match candidate.

Facial Recognition is not an absolute science. It is not quantifiable like DNA, so any and all intelligence information gathered on your candidate will greatly contribute to the greater good of the investigation by making your single choice, a strong investigative lead.


Art Meets Science

When it comes to facial recognition, the greatest challenge to law enforcement is the fact that most probe images obtained by law enforcement are uncontrolled in nature. They often originate from off-axis CCTV camera feeds, low-quality ATM photos, social media images, and other sources where the image is less than ideal for facial recognition.

Prevalent to most present-day facial recognition systems is an inability to read medium- to low-quality probes. In order to overcome these limitations, Vigilant Solutions has created a suite of specialized facial-recognition enhancement tools. These tools enable analysts and investigators to enhance select lower quality images that previously could not meet the criteria for facial recognition searching.

They can include certain images with poor lighting, poor subject poses (looking slightly down, slightly up, and certain profile image captures), some low-resolution images, heavily pixelated, overexposed, some fisheye camera captures, some distorted or skewed images, and occlusion (blocking any part of the face).

These preprocessing image enhancements are made following the probe image analysis discussed earlier. Once complete, and the probe image meets facial recognition searching criteria, it is enrolled into the application for searching against the gallery. Pre-processing enhancements simply raise lower probe image facial recognition quality to levels conducive for searching.


The Eyes Have It

Facial recognition systems rely heavily on pre-determined eye locations to properly orient the probe face before a search against the gallery. Eye positions and placements are the root of any facial recognition search. The matching process begins with the eyes, and the algorithm reads the face systematically thereafter. Due to this, eye capture and placement are critical. The eyes represent the cog on the facial recognition wheel, a centralized focal point, allowing the recognition process to work around it.

For most images that are controlled and higher in resolution, facial recognition applications tend to select the eye placements automatically. The same cannot be said for uncontrolled lower quality images. It is critical that an analyst be able to identify situations where manual eye placements are needed, especially on probes of lower quality.

There are instances where a person’s nostrils will be mistaken for eyes. This normally happens

when human heads are positioned looking slightly upward, and the nostrils are found to be more prevalent within the photo. If the analyst opts for a sole reliance on the software, his/her matching ability is severely compromised right at the start of the search.

When an analyst establishes a careful analysis of photos, and implements routine manual eye placements into the workflow, the guesswork is taken away from the application, and the integrity of the search is intact from start to finish.

Since facial recognition is not a science nor regulated, and there are no restrictions in place; it cannot be deemed as absolute, and all matches remain POSSIBLE. Even after due diligence in the investigative process is made. The end result of any facial recognition analysis is that it must provide the analyst with a very good investigative lead. Any enhancements made during image pre-processing are a good faith effort to triage lower quality photos in an attempt to leverage the technology to generate a potential lead.

This process can be comparable to a caller phoning into a particular law enforcement agency and stating he/she has just seen a suspect wanted for a particular crime on the news. The caller further states that he/she recognizes the perpetrator, and also provides a location where to find him/her. The caller has provided the agency with potential intelligence information. The responding agency official does not have a right to affect an immediate arrest.

Probable cause must be established from this potential lead before an arrest is made. The onus falls on the investigator to establish probable cause from the information provided to make the lead credible and eventually make an arrest.

The time is finally here when the promises of facial recognition become reality. Part science, part art, applications will enable law enforcement to secure high-quality investigative leads, protect personal privacy, and keep communities safe. Just set your expectations accordingly.


The Terms to Know                                           

Facial Recognition: An application that uses biometric algorithms to detect multiple landmarks and measurements of a face that may be compared to a gallery of known images to find potential matches.

Facial Identification: the manual process (the human aspect) of examining potential matches from facial recognition, looking for similarities or differences.

Algorithm: A process or set of rules to be followed in calculations, or other operations, which are set by a computer. In facial recognition the algorithms are rules on how to read a face.

Gallery: Any database of known images. Gallery images can come from a number of sources including mug shots, watch lists, or hotlists.

Probe Image: Any unknown image captured for facial recognition. Probe images can be taken by an officer in the field using a camera or mobile phone or from other sources such as security and CCTV cameras.

Gallery Image: An image from an existing facial recognition database. Once a probe image is run through the facial recognition system, it is manually compared to gallery images to identify potential match(es).

Controlled (Constrained) Images: Images with good lighting, frontal face positioning, high resolution, acceptable distance from the camera (Examples: Taken by a field officer, kiosk station, identification card photos). Controlled images are optimal for facial recognition matching.

Uncontrolled (Unconstrained) Images: Images with poor lighting, poor poses (looking down, up and certain profile captures), low resolution, heavily pixelated, overexposed, underexposed, subject is too far away, fish eye camera captures, distorted or skewed images, pictures or recordings of a screen, photocopies with excessive noise, occlusion: blocking any part of the face.


Roger Rodriguez is the Manager of Image Analytics, Vigilant Solutions. He joined Vigilant Solutions after serving over 20 years with the NYPD where he spearheaded the department’s first dedicated facial recognition unit. The world-renowned unit has conducted more than 8,500 facial recognition investigations, with over 3,000 possible matches, and approximately 2,000 arrests.

Published in Law and Order, Sep 2016

Rating : Not Yet Rated

Related Products



No Comments

Related Companies

Article Images

Click to enlarge images.

Close ...