1 Intro

The article discusses the application of modern video analytics in the retail, services and restaurant industry. We will cover customer traffic and flow, loyalty, churn, demographics, gender, age and emotions, and touch upon workplace supervision. 

Facial recognition is a socially sensitive topic and requires a judicious and cautious approach to data processing. On one hand, it is critically important to not get enmeshed in processing privileged data; on the other hand, when faces are used, even collecting benign demographic data about customers which is necessary to build an aggregate profile of the customer base is bound to raise privacy concerns. 

To alleviate these concerns and prevent data mishandling, we consider a person in a public place as an anonymous entity described by a set of observable parameters: gender, age, emotions. We do not use additional identifying data (government ID numbers, names, birthdates) and do not cross-reference collected data with third-party sources. Thus, the data collected does not exceed the theoretical limit of a real living human watching a video legally recorded in a public place (a shop, restaurant, beauty salon, or other venue). 

Our solutions discussed here are already operational. We particularly emphasize affordability and a low cost of entry. Being a small business, we aim to make our products cost-effective for, primarily, small businesses. 

As new technologies are emerging, what was previously only available to multinational corporations and government agencies is entering the consumer market. As the price decreases, the return on investment becomes considerable and may provide a decisive advantage over competitors. The important thing is to extract meaningful insights from the raw data so it transforms into an invaluable strategic resource, not a data silo of dubious worth.   

2 Tools and methods for customer analytics 

Which methods are practiced today to assist businesses in collecting and analyzing data on clients, customers and visitors to brick-and-mortar venues? 

Visitor counters are considered practically mandatory in rented spaces in shopping malls, but are also increasingly commonly encountered in standalone stores, beauty salons, etc.  These include infrared counters which sum up the number of times an invisible beam was interrupted, video-based 2d counters, 3d counters, and Wi-Fi scanners. Price and quality varies depending on type and use case.  

Manual counting is still practiced. The two variants are conducting a live in-person count in the store, and using CCTV footage; the former is used selectively, the second can be integrated as part of the daily workflow. The drawbacks of manual counting are obvious: labor intensity, the human factor, and a wide and unbounded margin of error. 

Why count people? 

Among the reasons are to know the real visitor traffic in your clothing boutique, beauty salon, or café, see visits over time by the hour, day, or weekday, predict and make use of peak hours, assess sales conversion (how store visits translate to actual purchases) and the effect from advertising campaigns; finally, to prevent employee misconduct: for instance, a hairdresser can report five clients but actually serve eight. 

Reward cards. These are even more widespread people counters and are meant to encourage a customer to return, ideally become a regular. Reward programs are designed to retain one-off customers and maintain brand awareness.  The technology behind the cards is simple; the real challenge is designing an effective incentives program. 

Polling. Reward card accounts need to be augmented with a customer’s demographic data: gender, age, interests, hobbies – any information that can help tailor communication and personalize offers to the client more effectively. 

Another common tool in the service industry is instant customer feedback. After the service is provided, the client can opt to press one of two buttons, “Satisfied” and “Unsatisfied”. 

Finally, there’s the cash register data, potentially matched against the data from all the above sources. 

The chief purpose of counting visitors of a venue, be it a store, a sports center, a coffee shop or other locale, is continuously finding leverage to increase and sustain customer traffic. Business logistics, workflow, and spatial geometry inform the selection and types of the methods to use. 

Large businesses are free to try, mix and match various tools and strategies. Small businesses are expense-conscious. Every additional potential expense is scrutinized, a dedicated marketing budget may be nonexistent, and, absent a salaried marketing specialist, the owner might have to wear many hats and play it by ear.  Regardless, time constraints necessitate delegating data collection to either humans or machines. 

Thus, the main sticking point of monitoring and supervision of operating activities is data fidelity. This applies to both machine-collected and human-collected data. If the data is trustworthy enough to comprise the foundation of a business strategy, the implementation cost of a technology often becomes a secondary concern. 

3 Computer vision: the modern tool for customer traffic analytics 

 CVizi‘s Track Expert solution uses two different methods of collecting and processing people data. The first one is the headcount – literally, counting people by the tops of their heads. The second the innovative one, is counting faces, which adds a breadth of extra options. 

3.1 2D and 3D Visitor Counters 

2D counters use two coordinates to track entrance and exit events. 

The camera is installed directly above the entrance area and counts all passers-by in both directions. Track Expert uses a proprietary 2D counter algorithm which is straightforward and reliable. We guarantee a minimum precision of 85%, while the average precision in practice exceeds 90%.

As indicated above, these kinds of counters (2D and 3D) are widely used, but the quality of data can be questionable. We provide a photo confirmation utility, with which a human user can verify the accuracy of the data.

We’ve extracted actionable info from the headcount data. Now if we’re interested in distinguishing customers from staff, identifying repeat visits, segmenting traffic by gender and age, this placement of the camera – above the entrance, looking directly down and counting tops of heads – would not be suitable. This is when Track Expert‘s second data-gathering method – facial recognition – comes into play.

The human face is a unique identifier which allows to detect and verify a person, their gender and age, find out the day and time when the person was last seen by the camera, classify the person as an employee or a customer, and distinguish emotions.

3.2 Fully automated face recognition: reality and perspectives

Facial recognition is far from a bleeding-edge technology. The first thing that comes to mind is banking and everything related to law enforcement and industrial safety: mass transit, airports, defense contractors, etc. However, the in-house solutions used in these fields are out of reach for a typical business owner.

A number of vendors are offering turnkey face recognition solutions of varying nominal quality. The real issue here is integration. Once we’ve got the faces, now what?  Is your marketing department prepared to handle this new data array? What about the CRM, can it process and analyze this data? A business needs a solution ready to provide actionable insights, not just any proof-of-concept implementation

4 Face-based statistical data for marketing tactics and strategy

What follows is an overview of reports to estimate the span and volume of information gained from a single installed camera with Track Expert.

The data has to be trustworthy. The first question you should ask is to demand proof the data is representative of the reality, not randomly generated.

This figure shows a portion of the daily customer profile gallery:

Now for the aggregate statistics and analytics. We’ll be using a span of one month, and one venue (a café) as a running example.

Customer Traffic and Repeat Visits

Just this single view allows to visualize the demographic slice which comprises the customer base of the venue. It’s definitely more popular with the women. Women aged 18 to 25 and men aged 35 to 50 are prominent on the combined gender and age bar chart.

The two possible strategies are as follows:

– promote the venue among women aged 18 to 25 and men aged 35 to 50, as these two groups constitute the apparent target audience of the café as it is now: there’s evidence the venue is of particular interest to these groups.

– focus marketing efforts on the underrepresented gender within age groups and thus balance out the genders.

Repeat visits constitute about 70% of the total count:

Checkout time

Checkout time has major impact on customer satisfaction and is one of the key metrics of store efficiency. For this use case, the camera is placed at checkout to allow Track Expert to collect data. The figure below shows a distribution of checkout time depending on time of day. The height of the bars represents customer traffic, and checkout time is color-coded: green for fast, red for slow.

This particular checkout terminal looks fine. There’s some noticeable slowdown in the morning, but whether the cause is the cashier or the customers themselves needs further investigation. The data can be used to make comparisons across different registers, cashiers, and venues, as well as a basis for automated staff KPI calculation.

Another useful metric is what we call “checkout approaches”. The number of extra times a customer approaches the checkout area indirectly shows if the produce in the shelves is laid out in a prominent and accessible way. Are these instances sporadic or regular? The metric helps to zoom in on these irregular scenarios and investigate the reasons.

Visitor emotions

Customer emotions are ephemeral, but, handled correctly, assessing the emotional background over the length of customer-employee interaction can add up to valuable, representative aggregate data and become another fully automated quality of service metric.

Each unique person provides an extensive dataset which contributes to the aggregate. The chart below shows normalized customer emotions over a month.

No noticeable outliers. Let us zoom in on a random day and see if we can notice irregularities on a smaller timescale:

We can also look for outliers on a per-person emotion chart:

Here the visitors are sorted by decreasing positive emotions (top) and decreasing negative emotions (bottom). The 5-digit numbers are anonymized person IDs.

This empowers you to do individual case studies. Drill down into the data and see what disappointed or excited them on any particular day.

Employees and diligence metrics

Separating customer data from employee data is a known problem in offline customer analytics. Not every venue has a separate entrance for employees. Employees can bias the data walking through the active zone of the counter, sometimes intentionally. Finally, business owners and supervisors will want to make use of the data to assess employee diligence in an automated fashion, without having to watch (or outsource the watching of) hours and hours of CCTV footage. TrackExpert  data can be presented as a clear and concise daily report that makes it easy to see who goes where and when. This is access monitoring with photographic evidence. Of course, a camera cannot fully replace physical security, only complement it.

Here’s the data, visualized:

5 Conclusion

The solutions demonstrated in the article are not just proof of concept; they have been put into production, implemented, and have been working in real commercial environments. Extracting useful data from camera footage in a fully automated and scalable fashion is not as trivial as it may seem. Standalone facial recognition services abound, but turnkey solutions providing actionable business insights are few and far between. Now the CVizi team has made a step toward realizing the ambitions of business owners and making modern technology affordable and accessible.