We r	consider	the	fully	automated	recognition	of	actions	in	uncontrolled	environment	Most	existing	work	relies	on	domain	knowledge	to	construct	complex	handcrafted	features	from	inputs	In	addition	the	environments	are	usually	assumed	to	be	controlled	Convolu-	tional	neural	networks	(CNNs)	are	a	type	of	deep	models	that	can	act	directly	on	the	raw	inputs	thus	automating	the	process	of	fea-	ture	construction	However	such	models	are	currently	limited	to	handle	2D	inputs	In	this	paper	we	develop	a	novel	3D	CNN	model	for	action	recognition	This	model	extracts	fea-	tures	from	both	spatial	and	temporal	dimen-	sions	by	performing	3D	convolutions	thereby	capturing	the	motion	information	encoded	in	multiple	adjacent	frames	The	developed	model	generates	multiple	channels	of	infor-	mation	from	the	input	frames	and	the	final	feature	representation	is	obtained	by	com-	bining	information	from	all	channels	We	apply	the	developed	model	to	recognize	hu-	man	actions	in	real-world	environment	and	it	achieves	superior	performance	without	re-	lying	on	handcrafted	features	1	Introduction	Recognizing	human	actions	in	real-world	environment	finds	applications	in	a	variety	of	domains	including	in-	telligent	video	surveillance	customer	attributes	and	shopping	behavior	analysis	However	accurate	recog-	nition	of	actions	is	a	highly	challenging	task	due	to	Appearing	in	Proceedings	of	the	27	th	International	Confer-	ence	on	Machine	Learning	Haifa	Israel	2010	Copyright	2010	by	the	author(s)/owner(s)	95014	USA	cluttered	backgrounds	occlusions	and	viewpoint	vari-	ations	etc	Therefore	most	of	the	existing	approaches	(Efros	et	al	2003	Schu ̈ldt	et	al	2004	Dolla ́r	et	al	2005	Laptev	&	P ́erez	2007	Jhuang	et	al	2007)	make	certain	assumptions	(e	g	small	scale	and	view-	point	changes)	about	the	circumstances	under	which	the	video	was	taken	However	such	assumptions	sel-	dom	hold	in	real-world	environment	In	addition	most	of	these	approaches	follow	the	conventional	paradigm	of	pattern	recognition	which	consists	of	two	steps	in	which	the	first	step	computes	complex	handcrafted	fea-	tures	from	raw	video	frames	and	the	second	step	learns	classifiers	based	on	the	obtained	features	In	real-world	scenarios	it	is	rarely	known	which	features	are	impor-	tant	for	the	task	at	hand	since	the	choice	of	feature	is	highly	problem-dependent	Especially	for	human	ac-	tion	recognition	different	action	classes	may	appear	dramatically	different	in	terms	of	their	appearances	and	motion	patterns	Deep	learning	models	(Fukushima	1980	LeCun	et	al	1998	Hinton	&	Salakhutdinov	2006	Hinton	et	al	2006	Bengio	2009)	are	a	class	of	machines	that	can	learn	a	hierarchy	of	features	by	building	high-level	features	from	low-level	ones	thereby	automating	the	process	of	feature	construction	Such	learning	ma-	chines	can	be	trained	using	either	supervised	or	un-	supervised	approaches	and	the	resulting	systems	have	been	shown	to	yield	competitive	performance	in	visual	object	recognition	(LeCun	et	al	1998	Hinton	et	al	2006	Ranzato	et	al	2007	Lee	et	al	2009a)	natu-	ral	language	processing	(Collobert	&	Weston	2008)	and	audio	classification	(Lee	et	al	2009b)	tasks	The	convolutional	neural	networks	(CNNs)	(LeCun	et	al	1998)	are	a	type	of	deep	models	in	which	trainable	filters	and	local	neighborhood	pooling	operations	are	applied	alternatingly	on	the	raw	input	images	result-	ing	in	a	hierarchy	of	increasingly	complex	features	It	has	been	shown	that	when	trained	with	appropri-	3D	Convolutional	Neural	Networks	for	Human	Action	Recognition	ate	regularization	(Ahmed	et	al	2008	Yu	et	al	2008	Mobahi	et	al	2009)	CNNs	can	achieve	superior	per-	formance	on	visual	object	recognition	tasks	without	relying	on	handcrafted	features	In	addition	CNNs	have	been	shown	to	be	relatively	insensitive	to	certain	variations	on	the	inputs	(LeCun	et	al	2004)	As	a	class	of	attractive	deep	models	for	automated	fea-	ture	construction	CNNs	have	been	primarily	applied	on	2D	images	In	this	paper	we	consider	the	use	of	CNNs	for	human	action	recognition	in	videos	A	sim-	ple	approach	in	this	direction	is	to	treat	video	frames	as	still	images	and	apply	CNNs	to	recognize	actions	at	the	individual	frame	level	Indeed	this	approach	has	been	used	to	analyze	the	videos	of	developing	embryos	(Ning	et	al	2005)	However	such	approach	does	not	consider	the	motion	information	encoded	in	multiple	contiguous	frames	To	effectively	incorporate	the	motion	information	in	video	analysis	we	propose	to	perform	3D	convolution	in	the	convolutional	layers	of	CNNs	so	that	discriminative	features	along	both	spatial	and	temporal	dimensions	are	captured	We	show	that	by	applying	multiple	distinct	convolutional	operations	at	the	same	location	on	the	input	multi-	ple	types	of	features	can	be	extracted	Based	on	the	proposed	3D	convolution	a	variety	of	3D	CNN	archi-	tectures	can	be	devised	to	analyze	video	data	We	develop	a	3D	CNN	architecture	that	generates	multi-	ple	channels	of	information	from	adjacent	video	frames	and	performs	convolution	and	subsampling	separately	in	each	channel	The	final	feature	representation	is	obtained	by	combining	information	from	all	channels	An	additional	advantage	of	the	CNN-based	models	is	that	the	recognition	phase	is	very	efficient	due	to	their	feed-forward	nature	We	evaluated	the	developed	3D	CNN	model	on	the	TREC	Video	Retrieval	Evaluation	(TRECVID)	data1	which	consist	of	surveillance	video	data	recorded	in	London	Gatwick	Airport	We	constructed	a	multi-	module	event	detection	system	which	includes	3D	CNN	as	a	module	and	participated	in	three	tasks	of	the	TRECVID	2009	Evaluation	for	Surveillance	Event	Detection	Our	system	achieved	the	best	performance	on	all	three	participated	tasks	To	provide	indepen-	dent	evaluation	of	the	3D	CNN	model	we	report	its	performance	on	the	TRECVID	2008	development	set	in	this	paper	We	also	present	results	on	the	KTH	data	as	published	performance	for	this	data	is	avail-	able	Our	experiments	show	that	the	developed	3D	CNN	model	outperforms	other	baseline	methods	on	the	TRECVID	data	and	it	achieves	competitive	per-	formance	on	the	KTH	data	without	depending	on	against-all	linear	SVM	is	learned	for	each	action	class	Specifically	we	extract	dense	SIFT	descriptors	(Lowe	2004)	from	raw	gray	images	or	motion	edge	history	images	(MEHI)	(Yang	et	al	2009)	Local	features	on	raw	gray	images	preserve	the	appearance	information	while	MEHI	concerns	with	the	shape	and	motion	pat-	terns	These	SIFT	descriptors	are	calculated	every	6	pixels	from	7	×	7	and	16	×	16	local	image	patches	in	the	same	cubes	as	in	the	3D	CNN	model	Then	they	are	softly	quantized	using	a	512-word	codebook	to	build	the	BoW	features	To	exploit	the	spatial	layout	in-	formation	we	employ	similar	approach	as	the	spatial	pyramid	matching	(SPM)	(Lazebnik	et	al	2006)	to	partition	the	candidate	region	into	2	×	2	and	3	×	4	cells	and	concatenate	their	BoW	features	The	dimension-	ality	of	the	entire	feature	vector	is	512×(2×2+3×4)	=	8192	We	denote	the	method	based	on	gray	images	as	SPMcube	and	the	one	based	on	MEHI	as	SPMcube	gray	MEHI	We	report	the	5-fold	cross-validation	results	in	which	the	data	for	a	single	day	are	used	as	a	fold	The	per-	formance	measures	we	used	are	precision	recall	and	area	under	the	ROC	curve	(ACU)	at	multiple	values	of	FALSE	positive	rates	(FPR)	The	performance	of	the	four	methods	is	summarized	in	Table	2	We	can	observe	from	Table	2	that	the	3D	CNN	model	outperforms	the	frame-based	2D	CNN	model	SPMcube	and	SPMcube	gray	MEHI	significantly	on	the	action	classes	CellToEar	and	Ob-	jectPut	in	all	cases	For	the	action	class	Pointing	3D	CNN	model	achieves	slightly	worse	performance	than	the	other	three	methods	From	Table	1	we	can	see	that	the	number	of	positive	samples	in	the	Pointing	class	is	significantly	larger	than	those	of	the	other	two	classes	Hence	we	can	conclude	that	the	3D	CNN	model	is	more	effective	when	the	number	of	positive	samples	is	small	Overall	the	3D	CNN	model	outperforms	other	three	methods	consistently	as	can	be	seen	from	the	average	performance	in	Table	2	4	2	Action	Recognition	on	KTH	Data	We	evaluate	the	3D	CNN	model	on	the	KTH	data	(Schu ̈ldt	et	al	2004)	which	consist	of	6	action	classes	performed	by	25	subjects	To	follow	the	setup	in	the	HMAX	model	we	use	a	9-frame	cube	as	input	and	ex-	tract	foreground	as	in	(Jhuang	et	al	2007)	To	reduce	the	memory	requirement	the	resolutions	of	the	input	frames	are	reduced	to	80	×	60	in	our	experiments	as	compared	to	160	×	120	used	in	(Jhuang	et	al	2007)	We	use	a	similar	3D	CNN	architecture	as	in	Figure	3	with	the	sizes	of	kernels	and	the	number	of	feature	maps	in	each	layer	modified	to	consider	the	80	×	60	×	9	inputs	In	particular	the	three	convolutional	layers	use	kernels	of	sizes	9×7	7×7	and	6×4	respec-	tively	and	the	two	subsampling	layers	use	kernels	of	size	3×3	By	using	this	setting	the	80×60×9	in-	puts	are	converted	into	128D	feature	vectors	The	final	layer	consists	of	6	units	corresponding	to	the	6	classes	As	in	(Jhuang	et	al	2007)	we	use	the	data	for	16	ran-	domly	selected	subjects	for	training	and	the	data	for	the	other	9	subjects	for	testing	The	recognition	per-	formance	averaged	across	5	random	trials	is	reported	in	Table	3	along	with	published	results	in	the	litera-	ture	The	3D	CNN	model	achieves	an	overall	accu-	racy	of	90	2%	as	compared	with	91	7%	achieved	by	the	HMAX	model	Note	that	the	HMAX	model	use	handcrafted	features	computed	from	raw	images	with	4-fold	higher	resolution	5	Conclusions	and	Discussions	We	developed	a	3D	CNN	model	for	action	recognition	in	this	paper	This	model	construct	features	from	both	spatial	and	temporal	dimensions	by	performing	3D	convolutions	The	developed	deep	architecture	gener-	ates	multiple	channels	of	information	from	adjacent	in-	put	frames	and	perform	convolution	and	subsampling	separately	in	each	channel	The	final	feature	represen-	tation	is	computed	by	combining	information	from	all	channels	We	evaluated	the	3D	CNN	model	using	the	TRECVID	and	the	KTH	data	sets	Results	show	that	the	3D	CNN	model	outperforms	compared	methods	on	the	TRECVID	data	while	it	achieves	competitive	performance	on	the	KTH	data	demonstrating	its	su-	perior	performance	in	real-world	environments
