Created by: typhoonzero

I've been writing implementations about manipulating SelectedRows and Tensor calculations. Found several places that might be able to improve:

1. Operators' "Kernels" are in fact the same as functors or can be assembled by several operators or functors. For example, merge SelectedRows duplicated rows is a functor used both in adam_op and adagrad_op. And, many operators can also be reused in other operators which are now difficult to do.
2. Operators and Functors are not “lazy evaluated”. I’ve written some code using Expression templates at here indicating how to do lazy evaluation. If we have Operators/Functors as “Expression templates”, the calculation is done at the last stage and probably using only one for-loop.

How to implement:

1. For eigen Operators/Functors, we can return the non evaluated tensor Operations in Compute call of current Kernel implement. And then evaluate the Operations only when necessary. Operators need to hold the expression object containing either Functor expression or Eigen Operations inside it.
2. For Functors that need standalone CUDA kernels, we can use “Expression templates” to implement functor expression.
3. A Block of operators will then be expanded as a large “Functor” containing both Eigen Operations and Functors, it’s type may look like: ….Add<Square<SparseAddTo<SparseMul<SelectedRows, SelectedRows>, Tensor>>, Scalar>….