Fixing the FM industry’s broken clock-in/out process using facial recognition

Since the disruption caused by the COVID pandemic, Facilities Management (FM) businesses and contract owners have been desperate to find new ways to track employee hours, reduce blowouts and calculate payroll with assurance. With the high uptake of workers’ flexible shifts during COVID and lack of dedicated human contact and supervision, a next-gen time and attendance system must operate seamlessly across mixed working patterns, flexitime/shift-work, multiple sites, and payment centres.

Anyone running a large-scale FM control centre will appreciate that managing hundreds of daily shifts is error-prone and challenging. Especially the challenge becomes even more complicated with multiple subcontracting layers involved in providing the workforce to deliver contracts to demanding customers at varied schedules.

Source DeepFace paper, EarnFlex App (on the left)

EarnFlex technology is designed to simplify and scale the FM industry’s broken clock-in/out process by utilizing AI and facial recognition technologies along with location built into its smart free-to-download and use App. Facial recognition technology opens new digital opportunities and changes the dynamics of the how FM industry oversees the workforce.

A medium-large scale FM company faces three main challenges when ensuring that an operative delivers the job on customers’ site.

1- The worker does not show up and does not inform anyone of his unplanned (planned) absence — The blowout.
2- The worker does not adhere to the contracted hours and either comes in late or leaves early.
3- The worker sends someone else, a friend, to cover his shift. Often the person replacing the contracted worker does not have the necessary licenses and permits, jeopardizing the business as the insurance will be invalid and it’s a breach of contract.

Managing an FM control room is one of the most stressful jobs — getty images

EarnFlex worker recognition and shift to location mapping system is based on Facebook’s face recognition algorithms in the DeepFace library. It uses reference images, usually from an identity document like a passport or license and a live BookOn/Off photo taken by the operative through our App. The system is designed to recognize the person and match his identity with the submitted documents. The primary facial recognition algorithms used by EarnFlex technology are based on state-of-the-art Convolutional Neural Networks (CNN) and provide best-in-class results.

Facial recognition is an exciting area of science which is getting mature enough to improve the day-to-day use-cases. However, the system is underpinned by the consistent human face structure across all the images. Slight differences are essential, especially when taking a picture from different mobile camera types in various lighting conditions with varying face angles. Facebook DeepFace Libray directly address this issue and is the critical reason EarnFlex face recognition is so effective.

EarnFlex Shift Assurance Reporting System — copyright EarnFlex Ltd.

Humans know a lot about human faces, and it seems only reasonable that a model could benefit from some human support. DeepFace, Facebook’s facial recognition model, used this human knowledge about the dataset and the task to develop a multi-step pipeline.

1. Detect the face in the image/face cropping. This step removes the arbitrary background from the image, such that the face is the subject of recognition.
2. Identify facial landmarks. This is the visual equivalent of putting a “mesh” on the face; several points corresponding to various facial landmarks like the centre of the forehead or the nose are identified.
3. Frontalization. Using these data points, the face is “warped”, so the person faces forward. Two people might look different if they’re looking in different directions, so this correction helps highlight potential differences in identity.
4. Pass preprocessed face into the neural network. Once the person’s face has been aligned and adjusted, it is passed into the neural network to classify. [source: DeepFace paper]

Especially in this current time, it may be surprising how extensive this pipeline is. The first three steps of facial detection, landmark identification, and frontalization are not trivial. Furthermore, many human assumptions are being made, such as:

• Facial detection algorithms do not obscure vital features, like the bottom of the jaw or the tip of the forehead.
• Frontalization is helpful to the model.
• Frontalization does not obscure or distort essential elements.

Nevertheless, DeepFace was conceived early in deep learning’s meteoric rise. Previously, more traditional machine learning algorithms like Linear Discriminant Analysis and Support Vector Machines were employed. Instead, DeepFace approached human performance on facial recognition and made impressive improvements in computer accuracy.

There is less reliance on preprocessing and postprocessing and instead greater trust that the model, given the correct framing of the task, can go farther than any human can. Human domain knowledge continues to cease providing help, and AI models will continue to increase in generality across various contexts. Each passing Book-On/Off image that the EarnFlex App successfully recognizes helps improve the face recognition model. It increases the accuracy and trust of businesses and workers in the effective and practical use of this technology.

In short, it’s a “winner takes all” in the AI tech. We believe EarnFlex is the first to the market with AI-led Book On/Off detection system for the FM industry. We know that our model will continue to improve with each book-on, making it more challenging for an incumbent company to compete with our technology on this crucial feature.