A lightweight Gaussian-based framework for real-time facial expression enhancement in 3D avatar systems

Facial expression modeling is a basic part of the contemporary 3D avatar systems as it plays an important role in emotional communication and engagement with the user and realism in the virtual environment. While there have been remarkable progress in neural rendering, Neural Radiance Field (NeRF) and Gaussian-based avatar representations have brought up impressive visual fidelity, the majority of the methods that have been developed are computationally demanding, involve large-scale training procedures, and require dedicated GPU hardware. This limitation makes their application difficult in the real world and resource-limited scenarios such as mobile devices, edge computing and CPU-based computing. To tackle these problems, this paper presents a novel method, named Expression-Focused Gaussian Enhancement (EFGE), which is lightweight and very efficient to enhance face expression without performing full 3D face reconstruction. The proposed approach is based on facial landmark tracking to localize expression critical regions like eyes, eyebrows, mouth and cheeks. A landmark displacement analysis is used to create dynamically adaptive Gaussians which are used to selectively enhance facial movements that are relevant to the viewer. The size and opaqueness of Gaussian components are controlled depending on the expression intensity so as to enhance the expressions perceptually meaningfully while keeping the computational complexity low. This framework is realized with MediaPipe face mesh and optimized to be executed real-time on CPU-based hardware. The experimental assessment shows the effectiveness of EFGE in enhancing the visibility of facial expression, emotional clarity, user perception, and with efficient computational performance. Comparative analysis shows that the proposed method provides a good balance between expressiveness and efficiency in comparison with the traditional landmark-based approach and the computationally expensive Gaussian avatar approach. The results indicate that EFGE can be a feasible and scalable approach to real-time expressive avatar rendering, especially in mobile, VR, AR and low-resource applications.