What is rPPG (Remote Photoplethysmography)

What is RPPG?

Tools for automated analysis of facial expressions provide an objective assessment of a person’s emotional expressions in consumer behavior and sensory research. Wellness Based AI Application also offers other video-based functionalities that are useful for sensory and consumer research such as automated registration of the frequency and variability of heart rate.

Remote photoplethysmography (rPPG) development uses computer vision technology which is used to extract information about changes in light absorption on an individual’s facial skin that is reflective of their physiological condition. Such changes are captured within the RGB pixel values of camera frames and converted into various physiological vitals, using remote ppg signals delivered as a remote health monitor application Read More

How rPPG works?

Whenever the heart beats, the skin color changes because of the extra blood running through the vessels. These tiny skin color changes are undetectable by the human eye, but the Ima Appweb rPPG algorithms amplify these color changes to calculate an accurate pulse rate signal by analyzing the frequency of the color changes. For respiration the camera focuses on the rise and fall of the chest and abdomen. Again, by amplify these motions via the unique Ima Appweb algorithms, an accurate breathing rate is calculated. The measurements are completely unobtrusive and no uncomfortable pads or electrodes need to be worn.

The technology works with any skin type. What’s more, it has been proven in real-world tests to be the most accurate contactless technique for measuring vital signs. It can be integrated with low-cost cameras, and doesn’t require any specialist knowledge to use – making it ideal for a wide range of applications.

Ima Appweb rPPG is motion robust. It uses facial tracking to get an accurate reading during motion, for example when the user is moving in front of the camera or the whole environment is moving. This ensures always a pulse and breathing rate can be calculated. What is rPPG

Robust Heart rate estimation from facial videos

The Application monitors real time cardiac activities of a person through remote photoplethysmography(rPPG) without any physical contact with sensor, by detecting blood volume pulse induced subtle color changes from video stream through webcam sensor or a video file.

Pre Processing

Skin pixels play significant role in extraction of rPPG signal therefore, we trained first ever deep learning model for semantic segmentation of skin and non skin pixels. This is novel technique for region of interest (ROI) selection and tracking. The model is robust to motion, multiple postures and segments skin pixels from non skin very accurately. rPPG signal exhibit different waveform when sampled from different regions of skin, therefore, to consistently sample ROI from same part of skin we detect face as prerequisite step to semantic segmentation.

rPPG Signal Extraction

After detection and tracking ROI for signal extraction we compute the spatial red, green and blue channel mean of skin segmented pixels to minimize camera quantization error. Averaged values of RGB channel are temporally normalized and projected to plane orthogonal to skin-tone. The projected signal is alpha tuned to extract signal.

Post processing

We apply signal processing techniques, moving average filter of order 6 to remove outliers from signal. To estimate heart rate we compute power spectral density PSD applying fast Fourier transformation (FFT) on rPPG signal. It is then band pass filtered to analyze only frequencies of interest. The maximum power spectrum represents the frequency of instant heart rate.

Uses remote PPG (PhotoPlethysmoGraphy)

Camera tracks rise/fall of chest and abdomen to determine respiration rate
Tiny changes in skin color caused by HR are detected (invisible micro-blushes)
Ima Appweb solved many challenges associated with this technique (motion, light, etc.)

Heart rate and heart rate variability

Heart rate (HR), the measure of how often the heart beats per minute, is influenced by the autonomic nervous system (ANS) which consists of two primary branches, the sympathetic and the parasympathetic nervous system.

Heart rate is active when we are in an exciting situation (threat, fear, stress, and exercise) and increases heart rate. heart rate variability is active when we are in a relaxed situation (feelings of love, compassion, and calm) and decreases heart rate. As you can see, the responses are influenced by mental and physical conditions.

Heart rate variability (HRV)

Heart rate variability (HRV) is the indicator of the balance of those conditions, being a marker of health and stress. When the sympathetic nervous system is predominant like when people feel stressed, HRV decreases. When the parasympathetic nervous system is predominant like when people are relaxed, HRV increases.

There has been growing attention for HRV as an indicator of cardiovascular disease, and internal emotions such as depression and work stress, etc.

Measuring HR and HRV remotely

Wellness Based AI Application can measure heart rate and heart rate variability remotely using remote photoplethysmography (RPPG), a tool to measure heart rate remotely without attaching sensors. It only requires video recording with a high-resolution camera. It can be beneficial in various kinds of physical, health and emotional monitoring, such as monitoring of drivers, consumers, the elderly, and infants [2].

Those kinds of monitoring are now available with less discomfort, using remote PPG. As part of my research, I looked at HR and HRV while participants consumed different kinds of foods, because I wanted to observe the physiological mechanism changes during food consumption.

The principles of PPG

The principles of PPG are the following: Using the example of a wearable, such as a heartrate-sensing watch, light emits to the skin from one sensor and a second sensor detects how much light is returned to the device. This forms the contrast between emitted light and reflected light. The amount of reflected light changes according to the blood volume, which is caused by capillary dilation and constriction, hence it can estimate the heart rate.

Recommended conditions of rPPG

As a result of my research, the following conditions are recommended for RPPG:

1. Use of high resolution and uncompressed cameras, since both lead to a more accurate heart rate detection.
2. Use of good light conditions, solely from a front lamp. However, avoid frontal lighting that is too strong, to prevent overexposure. A ceiling lamp causes overexposure, which interrupts the skin color detection.
3. Avoid video compression. Video compression decreases the resolution and video quality.

Moreover, RPPG measurement is sensitive to motion of the head. It is recommended to use a setting where participants are sitting and not moving too much. A high correlation between the gold standard and RPPG in these situations has been observed in many studies.

Because of the increasing attention to remote PPG, these drawbacks are expected to be solved, at least partly, with the improvement of algorithms in the near future. What is rPPG

Applications of rPPG

1. Wellness products

Manufacturers can combine rPPG with a camera, in a phone, tablet, toys or any electronic wellness product, and you have an ideal solution for “quantified self” applications. By providing accurate, unobtrusive pulse and breathing rate measurements anywhere, they enable ‘self-tracking’ of moods, sleep patterns, activity levels and other everyday activities.

2. In the car

High-end vehicles are starting to feature driver monitoring cameras to improve safety and well-being. ImaAppweb’s rPPG lets you take this trend further. Its accurate pulse and breathing rate data can help spot driver drowsiness, stress and attention levels, allowing the car to provide feedback or adjust environmental controls. The contactless technology doesn’t disturb the driver and works reliably irrespective of driver movement or posture – without having to be adjusted for different drivers.

Other potential applications of rPPG

1. Sport and fitness
2. Gaming
3. Broadcasting
4. (Outdoor) advertisement
5. Security

References

1. Thayer, J. F.; Yamamoto, S. S. & Brosschot, J. F. (2010). The relationship of autonomic imbalance, heart rate variability and cardiovascular disease risk factors. International journal of cardiology, 141 (2), 122-131.
2. Wang, W.; Den Brinker, A. C. & De Haan, G. (2018). Single Element Remote-PPG. IEEE Transactions on Biomedical Engineering.
3. NeuroData Lab. (2019, March). Every beat counts. Comparing Remote Webcam Heart Rate Detector to Wearables. https://medium.com/@neurodatalab/every-beat-counts-comparing-remote-heart-rate-webcam-detector-to-wearables-d8d59aab863c
4. Wang, W.; den Brinker, A. C.; Stuijk, S. & de Haan, G. (2016). Algorithmic principles of remote PPG. IEEE Transactions on Biomedical Engineering, 64 (7), 1479-1491.
5. Gudi, A.; Bittner, M.; Lochmans, R. & Van Gemert, J. (2019) Efficient Real-Time Camera Based Estimation of Heart Rate and Its Variability. International Conference on Computer Vision (ICCV) Workshop.