We consider the fully automated recognition of actions in uncontrolled environment Most existing work relies on domain knowledge to construct complex handcrafted features from inputs In addition the environments are usually assumed to be controlled Convolu- tional neural networks (CNNs) are a type of deep models that can act directly on the raw inputs thus automating the process of fea- ture construction However such models are currently limited to handle 2D inputs In this paper we develop a novel 3D CNN model for action recognition This model extracts fea- tures from both spatial and temporal dimen- sions by performing 3D convolutions thereby capturing the motion information encoded in multiple adjacent frames The developed model generates multiple channels of infor- mation from the input frames and the final feature representation is obtained by com- bining information from all channels We apply the developed model to recognize hu- man actions in real-world environment and it achieves superior performance without re- lying on handcrafted features 1 Introduction Recognizing human actions in real-world environment finds applications in a variety of domains including in- telligent video surveillance customer attributes and shopping behavior analysis However accurate recog- nition of actions is a highly challenging task due to Appearing in Proceedings of the 27 th International Confer- ence on Machine Learning Haifa Israel 2010 Copyright 2010 by the author(s)/owner(s) 95014 USA cluttered backgrounds occlusions and viewpoint vari- ations etc Therefore most of the existing approaches (Efros et al 2003 Schu ̈ldt et al 2004 Dolla ́r et al 2005 Laptev & P ́erez 2007 Jhuang et al 2007) make certain assumptions (e g small scale and view- point changes) about the circumstances under which the video was taken However such assumptions sel- dom hold in real-world environment In addition most of these approaches follow the conventional paradigm of pattern recognition which consists of two steps in which the first step computes complex handcrafted fea- tures from raw video frames and the second step learns classifiers based on the obtained features In real-world scenarios it is rarely known which features are impor- tant for the task at hand since the choice of feature is highly problem-dependent Especially for human ac- tion recognition different action classes may appear dramatically different in terms of their appearances and motion patterns Deep learning models (Fukushima 1980 LeCun et al 1998 Hinton & Salakhutdinov 2006 Hinton et al 2006 Bengio 2009) are a class of machines that can learn a hierarchy of features by building high-level features from low-level ones thereby automating the process of feature construction Such learning ma- chines can be trained using either supervised or un- supervised approaches and the resulting systems have been shown to yield competitive performance in visual object recognition (LeCun et al 1998 Hinton et al 2006 Ranzato et al 2007 Lee et al 2009a) natu- ral language processing (Collobert & Weston 2008) and audio classification (Lee et al 2009b) tasks The convolutional neural networks (CNNs) (LeCun et al 1998) are a type of deep models in which trainable filters and local neighborhood pooling operations are applied alternatingly on the raw input images result- ing in a hierarchy of increasingly complex features It has been shown that when trained with appropri- 3D Convolutional Neural Networks for Human Action Recognition ate regularization (Ahmed et al 2008 Yu et al 2008 Mobahi et al 2009) CNNs can achieve superior per- formance on visual object recognition tasks without relying on handcrafted features In addition CNNs have been shown to be relatively insensitive to certain variations on the inputs (LeCun et al 2004) As a class of attractive deep models for automated fea- ture construction CNNs have been primarily applied on 2D images In this paper we consider the use of CNNs for human action recognition in videos A sim- ple approach in this direction is to treat video frames as still images and apply CNNs to recognize actions at the individual frame level Indeed this approach has been used to analyze the videos of developing embryos (Ning et al 2005) However such approach does not consider the motion information encoded in multiple contiguous frames To effectively incorporate the motion information in video analysis we propose to perform 3D convolution in the convolutional layers of CNNs so that discriminative features along both spatial and temporal dimensions are captured We show that by applying multiple distinct convolutional operations at the same location on the input multi- ple types of features can be extracted Based on the proposed 3D convolution a variety of 3D CNN archi- tectures can be devised to analyze video data We develop a 3D CNN architecture that generates multi- ple channels of information from adjacent video frames and performs convolution and subsampling separately in each channel The final feature representation is obtained by combining information from all channels An additional advantage of the CNN-based models is that the recognition phase is very efficient due to their feed-forward nature We evaluated the developed 3D CNN model on the TREC Video Retrieval Evaluation (TRECVID) data1 which consist of surveillance video data recorded in London Gatwick Airport We constructed a multi- module event detection system which includes 3D CNN as a module and participated in three tasks of the TRECVID 2009 Evaluation for Surveillance Event Detection Our system achieved the best performance on all three participated tasks To provide indepen- dent evaluation of the 3D CNN model we report its performance on the TRECVID 2008 development set in this paper We also present results on the KTH data as published performance for this data is avail- able Our experiments show that the developed 3D CNN model outperforms other baseline methods on the TRECVID data and it achieves competitive per- formance on the KTH data without depending on against-all linear SVM is learned for each action class Specifically we extract dense SIFT descriptors (Lowe 2004) from raw gray images or motion edge history images (MEHI) (Yang et al 2009) Local features on raw gray images preserve the appearance information while MEHI concerns with the shape and motion pat- terns These SIFT descriptors are calculated every 6 pixels from 7 × 7 and 16 × 16 local image patches in the same cubes as in the 3D CNN model Then they are softly quantized using a 512-word codebook to build the BoW features To exploit the spatial layout in- formation we employ similar approach as the spatial pyramid matching (SPM) (Lazebnik et al 2006) to partition the candidate region into 2 × 2 and 3 × 4 cells and concatenate their BoW features The dimension- ality of the entire feature vector is 512×(2×2+3×4) = 8192 We denote the method based on gray images as SPMcube and the one based on MEHI as SPMcube gray MEHI We report the 5-fold cross-validation results in which the data for a single day are used as a fold The per- formance measures we used are precision recall and area under the ROC curve (ACU) at multiple values of FALSE positive rates (FPR) The performance of the four methods is summarized in Table 2 We can observe from Table 2 that the 3D CNN model outperforms the frame-based 2D CNN model SPMcube and SPMcube gray MEHI significantly on the action classes CellToEar and Ob- jectPut in all cases For the action class Pointing 3D CNN model achieves slightly worse performance than the other three methods From Table 1 we can see that the number of positive samples in the Pointing class is significantly larger than those of the other two classes Hence we can conclude that the 3D CNN model is more effective when the number of positive samples is small Overall the 3D CNN model outperforms other three methods consistently as can be seen from the average performance in Table 2 4 2 Action Recognition on KTH Data We evaluate the 3D CNN model on the KTH data (Schu ̈ldt et al 2004) which consist of 6 action classes performed by 25 subjects To follow the setup in the HMAX model we use a 9-frame cube as input and ex- tract foreground as in (Jhuang et al 2007) To reduce the memory requirement the resolutions of the input frames are reduced to 80 × 60 in our experiments as compared to 160 × 120 used in (Jhuang et al 2007) We use a similar 3D CNN architecture as in Figure 3 with the sizes of kernels and the number of feature maps in each layer modified to consider the 80 × 60 × 9 inputs In particular the three convolutional layers use kernels of sizes 9×7 7×7 and 6×4 respec- tively and the two subsampling layers use kernels of size 3×3 By using this setting the 80×60×9 in- puts are converted into 128D feature vectors The final layer consists of 6 units corresponding to the 6 classes As in (Jhuang et al 2007) we use the data for 16 ran- domly selected subjects for training and the data for the other 9 subjects for testing The recognition per- formance averaged across 5 random trials is reported in Table 3 along with published results in the litera- ture The 3D CNN model achieves an overall accu- racy of 90 2% as compared with 91 7% achieved by the HMAX model Note that the HMAX model use handcrafted features computed from raw images with 4-fold higher resolution 5 Conclusions and Discussions We developed a 3D CNN model for action recognition in this paper This model construct features from both spatial and temporal dimensions by performing 3D convolutions The developed deep architecture gener- ates multiple channels of information from adjacent in- put frames and perform convolution and subsampling separately in each channel The final feature represen- tation is computed by combining information from all channels We evaluated the 3D CNN model using the TRECVID and the KTH data sets Results show that the 3D CNN model outperforms compared methods on the TRECVID data while it achieves competitive performance on the KTH data demonstrating its su- perior performance in real-world environments
We r consider the fully automated recognition of actions in uncontrolled environment Most existing work relies on domain knowledge to construct complex handcrafted features from inputs In addition the environments are usually assumed to be controlled Convolu- tional neural networks (CNNs) are a type of deep models that can act directly on the raw inputs thus automating the process of fea- ture construction However such models are currently limited to handle 2D inputs In this paper we develop a novel 3D CNN model for action recognition This model extracts fea- tures from both spatial and temporal dimen- sions by performing 3D convolutions thereby capturing the motion information encoded in multiple adjacent frames The developed model generates multiple channels of infor- mation from the input frames and the final feature representation is obtained by com- bining information from all channels We apply the developed model to recognize hu- man actions in real-world environment and it achieves superior performance without re- lying on handcrafted features 1 Introduction Recognizing human actions in real-world environment finds applications in a variety of domains including in- telligent video surveillance customer attributes and shopping behavior analysis However accurate recog- nition of actions is a highly challenging task due to Appearing in Proceedings of the 27 th International Confer- ence on Machine Learning Haifa Israel 2010 Copyright 2010 by the author(s)/owner(s) 95014 USA cluttered backgrounds occlusions and viewpoint vari- ations etc Therefore most of the existing approaches (Efros et al 2003 Schu ̈ldt et al 2004 Dolla ́r et al 2005 Laptev & P ́erez 2007 Jhuang et al 2007) make certain assumptions (e g small scale and view- point changes) about the circumstances under which the video was taken However such assumptions sel- dom hold in real-world environment In addition most of these approaches follow the conventional paradigm of pattern recognition which consists of two steps in which the first step computes complex handcrafted fea- tures from raw video frames and the second step learns classifiers based on the obtained features In real-world scenarios it is rarely known which features are impor- tant for the task at hand since the choice of feature is highly problem-dependent Especially for human ac- tion recognition different action classes may appear dramatically different in terms of their appearances and motion patterns Deep learning models (Fukushima 1980 LeCun et al 1998 Hinton & Salakhutdinov 2006 Hinton et al 2006 Bengio 2009) are a class of machines that can learn a hierarchy of features by building high-level features from low-level ones thereby automating the process of feature construction Such learning ma- chines can be trained using either supervised or un- supervised approaches and the resulting systems have been shown to yield competitive performance in visual object recognition (LeCun et al 1998 Hinton et al 2006 Ranzato et al 2007 Lee et al 2009a) natu- ral language processing (Collobert & Weston 2008) and audio classification (Lee et al 2009b) tasks The convolutional neural networks (CNNs) (LeCun et al 1998) are a type of deep models in which trainable filters and local neighborhood pooling operations are applied alternatingly on the raw input images result- ing in a hierarchy of increasingly complex features It has been shown that when trained with appropri- 3D Convolutional Neural Networks for Human Action Recognition ate regularization (Ahmed et al 2008 Yu et al 2008 Mobahi et al 2009) CNNs can achieve superior per- formance on visual object recognition tasks without relying on handcrafted features In addition CNNs have been shown to be relatively insensitive to certain variations on the inputs (LeCun et al 2004) As a class of attractive deep models for automated fea- ture construction CNNs have been primarily applied on 2D images In this paper we consider the use of CNNs for human action recognition in videos A sim- ple approach in this direction is to treat video frames as still images and apply CNNs to recognize actions at the individual frame level Indeed this approach has been used to analyze the videos of developing embryos (Ning et al 2005) However such approach does not consider the motion information encoded in multiple contiguous frames To effectively incorporate the motion information in video analysis we propose to perform 3D convolution in the convolutional layers of CNNs so that discriminative features along both spatial and temporal dimensions are captured We show that by applying multiple distinct convolutional operations at the same location on the input multi- ple types of features can be extracted Based on the proposed 3D convolution a variety of 3D CNN archi- tectures can be devised to analyze video data We develop a 3D CNN architecture that generates multi- ple channels of information from adjacent video frames and performs convolution and subsampling separately in each channel The final feature representation is obtained by combining information from all channels An additional advantage of the CNN-based models is that the recognition phase is very efficient due to their feed-forward nature We evaluated the developed 3D CNN model on the TREC Video Retrieval Evaluation (TRECVID) data1 which consist of surveillance video data recorded in London Gatwick Airport We constructed a multi- module event detection system which includes 3D CNN as a module and participated in three tasks of the TRECVID 2009 Evaluation for Surveillance Event Detection Our system achieved the best performance on all three participated tasks To provide indepen- dent evaluation of the 3D CNN model we report its performance on the TRECVID 2008 development set in this paper We also present results on the KTH data as published performance for this data is avail- able Our experiments show that the developed 3D CNN model outperforms other baseline methods on the TRECVID data and it achieves competitive per- formance on the KTH data without depending on against-all linear SVM is learned for each action class Specifically we extract dense SIFT descriptors (Lowe 2004) from raw gray images or motion edge history images (MEHI) (Yang et al 2009) Local features on raw gray images preserve the appearance information while MEHI concerns with the shape and motion pat- terns These SIFT descriptors are calculated every 6 pixels from 7 × 7 and 16 × 16 local image patches in the same cubes as in the 3D CNN model Then they are softly quantized using a 512-word codebook to build the BoW features To exploit the spatial layout in- formation we employ similar approach as the spatial pyramid matching (SPM) (Lazebnik et al 2006) to partition the candidate region into 2 × 2 and 3 × 4 cells and concatenate their BoW features The dimension- ality of the entire feature vector is 512×(2×2+3×4) = 8192 We denote the method based on gray images as SPMcube and the one based on MEHI as SPMcube gray MEHI We report the 5-fold cross-validation results in which the data for a single day are used as a fold The per- formance measures we used are precision recall and area under the ROC curve (ACU) at multiple values of FALSE positive rates (FPR) The performance of the four methods is summarized in Table 2 We can observe from Table 2 that the 3D CNN model outperforms the frame-based 2D CNN model SPMcube and SPMcube gray MEHI significantly on the action classes CellToEar and Ob- jectPut in all cases For the action class Pointing 3D CNN model achieves slightly worse performance than the other three methods From Table 1 we can see that the number of positive samples in the Pointing class is significantly larger than those of the other two classes Hence we can conclude that the 3D CNN model is more effective when the number of positive samples is small Overall the 3D CNN model outperforms other three methods consistently as can be seen from the average performance in Table 2 4 2 Action Recognition on KTH Data We evaluate the 3D CNN model on the KTH data (Schu ̈ldt et al 2004) which consist of 6 action classes performed by 25 subjects To follow the setup in the HMAX model we use a 9-frame cube as input and ex- tract foreground as in (Jhuang et al 2007) To reduce the memory requirement the resolutions of the input frames are reduced to 80 × 60 in our experiments as compared to 160 × 120 used in (Jhuang et al 2007) We use a similar 3D CNN architecture as in Figure 3 with the sizes of kernels and the number of feature maps in each layer modified to consider the 80 × 60 × 9 inputs In particular the three convolutional layers use kernels of sizes 9×7 7×7 and 6×4 respec- tively and the two subsampling layers use kernels of size 3×3 By using this setting the 80×60×9 in- puts are converted into 128D feature vectors The final layer consists of 6 units corresponding to the 6 classes As in (Jhuang et al 2007) we use the data for 16 ran- domly selected subjects for training and the data for the other 9 subjects for testing The recognition per- formance averaged across 5 random trials is reported in Table 3 along with published results in the litera- ture The 3D CNN model achieves an overall accu- racy of 90 2% as compared with 91 7% achieved by the HMAX model Note that the HMAX model use handcrafted features computed from raw images with 4-fold higher resolution 5 Conclusions and Discussions We developed a 3D CNN model for action recognition in this paper This model construct features from both spatial and temporal dimensions by performing 3D convolutions The developed deep architecture gener- ates multiple channels of information from adjacent in- put frames and perform convolution and subsampling separately in each channel The final feature represen- tation is computed by combining information from all channels We evaluated the 3D CNN model using the TRECVID and the KTH data sets Results show that the 3D CNN model outperforms compared methods on the TRECVID data while it achieves competitive performance on the KTH data demonstrating its su- perior performance in real-world environments
