The authors would like to thank all editors and the anonymous reviewers for their valuable suggestions. This paper has utilized BHU facial expression database and MMI facial expression database. The authors would like to thank the providers of these databases. This work was supported in part by the National Natural Science Foundation of China (Grant no. 61602528 and Grant no. 61772561), the Hunan Provincial Natural Science Foundation of China (Grant no. 2017JJ3527), the Scientific Research Fund of Hunan Provincial Education Department, China (Grant no. 16B275), the Grant of China Scholarship Council (CSC no. 201808430002), and the Research Foundation for Advanced Talents of Central South University of Forestry and Technology, China (Grant no. 2015YJ013).