Facial micro-expression spotting and recognition using deep learning / Liong Gen Bing

Liong , Gen Bing (2024) Facial micro-expression spotting and recognition using deep learning / Liong Gen Bing. PhD thesis, Universiti Malaya.

	PDF (The Candidate's Agreement) Restricted to Repository staff only Download (218Kb)
	PDF (Thesis PhD) Restricted to Repository staff only until 31 December 2025. Download (2043Kb)

Abstract

Facial Micro-Expressions (MEs) are subtle involuntary actions that reveal a person’s hidden emotions in high-stakes situations within a fraction of a second. The diverse range of practical applications has garnered significant interest among researchers. However, limited research exists on ME analysis in untrimmed video datasets, where the challenge lies in seamlessly spotting the MEs and recognizing the associated emotions within videos containing unrelated facial movements. Furthermore, despite the recent emergence of multimodal ME datasets with an additional depth dimension, the exploration of ME analysis in line with 3D human visual perception remains subdued. Additionally, the scarcity of labeled ME samples presents a challenge to developing robust networks. These limitations are addressed explicitly in this thesis. This thesis proposes a network architecture named the Micro-Expression Analysis Network (MEAN), characterized by its shallow, multi-stream, and multi-output design tailored for the task of ME analysis. The MEAN architecture consists of three modules: a shared module for extracting lower-level features, a spotting module for identifying the ME intervals, and a recognition module for predicting the emotion classes. To preserve learned knowledge, an Inductive Transfer Learning (ITL) is adopted in the two-step network learning process. Besides, a fairer metric is promoted to measure the efficacy of a complete ME analysis system. The effectiveness is evaluated through qualitative and quantitative assessments on both trimmed and untrimmed video datasets. As one of the pioneering works on multimodal ME datasets, the additional depth information is incorporated to enhance the representations of MEs. Particularly, the scene flow is computed as a feature descriptor to estimate 3D motion changes on the face by leveraging both color and depth modalities. In line with this, the Scene Flow Attention-based Network (SFAMNet) is presented in this thesis. SFAMNet takes the scene flow as input for performing ME spotting, ME recognition, and ME analysis tasks. To facilitate the learning process, data augmentation and end-to-end network optimization are employed. Experimental studies have been conducted to highlight the importance of depth dimension in capturing the subtle temporal changes associated with MEs. Given the limited availability of labeled ME samples, it is hypothesized that Self- Supervised Learning (SSL) can suggest valuable visual cues to the networks by utilizing the self-supervisory signals generated from unlabeled data. Drawing inspiration from the distinct micro-movements phases, a novel SSL pretext task is introduced, allowing the network to learn meaningful spatiotemporal features from unlabeled video sequences. The learned knowledge from this pretext task is then transferred and fine-tuned on several downstream tasks, including ME recognition, ME spotting, as well as Micro-Gesture (MG) recognition. As SSL is still relatively new in the ME domain, some popular pretext tasks are implemented as the baselines for comparison. As a whole, promising results are obtained compared to the existing traditional, SSL, and supervised learning methods, thus demonstrating the effectiveness of the proposed SSL approach.

Actions (For repository staff only : Login required)