Created by: reyoung

Dim<N> and DDim have two advantages:

1. Checking Rank during compile time.
2. The memory of dims are static allocation, not dynamic allocation as std::vector.

Checking Rank

The tensor should be like this

template <size_t Rank>
class Tensor {
 public:
  Dim<Rank> dims_;
};

So, the different rank of a tensor is different type. Then we use Variable::Get<Tensor<2>>() to get a matrix. If the vairable is not typed Tensor<2>, then an error occurs.

But we do not want to have template argument now. Our tensor is like

class Tensor {
 public:
  DDim dims_;
};

then we cannot use type to check Tensor's rank, i.e., we cannot use the major advantage of Dim<N>.

Static Allocation

Dim<N> and DDim are allocated in once, not like std::vector has a growth memory. std::vector is very slow when memory needs realloc, but DDim and Dim<N> do have this issue.

However, DDim is a boost::variant. sizeof(DDim) is as large as the maximum rank that Paddle supports. The sizeof(DDim) is 48 now, can hold 12 integers. In the neural network, most of tensors' rank is less than 5.

Simpler solution

Here, I propose two solutions, which could simplify Dim<N> and DDim design.

1. Just use a constexpr size_t MAX_RANK=12; and each tensor holds a variable size_t dims_[MAX_RANK];. The advantages of this design and DDim are same, static allocation. The disadvantage of this design is we limit our tensor's rank less than 13.

2. Use the boost::small_vector<size_t, 5> dims_ to represent dims. boost::small_vector is a vector-like container optimized for the case when it contains few elements. If rank is less than 5, dims_ is static allocation at once. Otherwise, it just likes a plain std::vector.

Maybe we should change our dim design?