- Removed all main_program and startup_program in the demo.
- Using paddle.default_main_program() hides the implementation detail (e.g., using g_main_program) from the user, we can change the implementation in the future much easier.

* init commit

* change some dir name

* Conditional Block Forward

* Assign Operator.

Out=X, when type in [LoDTensor/SelectedRows/LoDTensorArray]

* Stash

* Add Scope::Rename

it is useful in gradient phase of an operator with block

* ConditionalBlock Grad Done

* Add comments

* yapf format code

* first commit

* Python API for while op

* Python Unittest for simple while_op forward

* fix out to be list

* Fix UT

* VarType

* Fix several bugs

* Fix bug

* Fix bug

* Fix Bug

* Fix bug

* Fix unittest

* Remove debug log

* Add comments

* add PADDLE_ENFORCE

* while_grad_op first commit

* Add `BlockDescBind::FindRecursiveOrCreateVar()` and fix bugs

* refine code

* fix unittest bug

* Use stable_sort in lod_rank_table

It is easy to debug and test when use `stable_sort`and the time
complexity is not changed.

* Add LoDTensorArray

* Stash

* Better debug message for IsInitialized

* Stash

* Better debug message for IsInitialized

* Complete array read/write op unittests

* Add LoDRankTable

LoD Rank Table stores the `level` of `lod` which is ordered by sequence
length in descending order. It is useful when implement dynamic RNN and
is shared by dynamic RNN memory, dynamic RNN slice input and dynamic
RNN slice output operators.

* Add InferVarType

* Init commit

* Make executor use ProgramDescBind

* Change Attribute from BlockDesc to BlockDescBind

* Since we will get the program desc in RNN, just BlockDesc is not
  enough.

* Add DeviceContext to Executor API

* Rewrite RNN

* Pass Python

* AddBiasOp does not care num_flatten_dims

* Stash

* Fix MacOS Compile

* Pass RNN forward

* add python test

* refactor test

* Make compile pass

* add gradopmaker

* First draft done

* Polish code

* add grad op maker and grad infershape

* Polish code

* Fix backward.cc bug

* Fix infershape

* Rename function

* add backward test

* simplify recurrent test

* Update

* Pass unittest

* Add comments & refine test

* Add comments

* refactor test

* Complete Unittest

* fix StepScopes enforce

* Remove unused unittest

* no type error

* Update

* Make RNN Pass unittest

* optimizer use init_program

* create persistable variable

* add create_persistable_var to block

* optimizer use create_persistable_var

* fix prefix

* move create_global_persistable_var from Block to LayerHelper

* Polish Optimizer initialization code.

* Using the LayerHelper to create initialize operator and variables

* add_accumulator should use an independent data type

* default use param data type for accumulator

* improve unique_name, uniq id is related to prefix

* fix join

* add batch_norm_layer

* add img_conv_group layer and test

* add check to Tensor.type()

* forward can run

* with backward

* change label data time from int32 to int64

* refine code

* follow comment

* Add `dump_to_file()` for ProgrameDescBind in pybind

* Update

* Add utility.py

* typo

* Fix bugs

* Move add_feed/fetch_components to untility.py

* Compelete dump

* Follow comments

* Change output of Prune() from inference to pointer

* Expose Prune() to Python

* Compelete save/load API of inference model

* Fix errors

* Debuging

* Compelete unit tests

* follow comments

* Python API for save/load variables

* Polish names

* Add regularizer code
* Fix code

* compelete restoring program_bind from program_desc

* Fix bugs

* fix compile errors

* fix errors and add unit tests

* rename some vars

* Follow comments

* Init

* Stash

* Polish SaveLoadOp

* Fix CI

* Polish code

* Save GPU Tensor

* Stash

* Fix CI

* "add model format design doc"

* "add restore function"

* "add parse protobuf"

* "move necessary information to saver.proto"

* "format code"

* "add gpu option"

* "add lod info"

* "add saveop python test wrapper"

* "checkpoint reuse save operator"

* "rewrite model format design doc"

* "async support needed"

* "fix run once"

* "fix doc based on comments"

* "refine based on comments"

* "fix based comments"

* "remove persistable flag from framework.proto"

* "add IndicateDataType to restore op"

* "add save test"

* "modify save restore code"

* "modified the restore logic"

* rm checkpoint_op.cc

* rm test_checkpoint_op.py

* "get inputs outputs name from execution context"

* Saving each variable to a independent file

* Fix bugs

* Rewrite save_restore_op_test with new Python framework

* Move `SaveOp` and `RestoreOp` from OpWithKernel to OpBase

* Refine unit test of SaveOp and RestoreOp

* fix compile errorwq

Pass all forward op test

* init

* ensure ids in lookup table op must be a column vector

* add book4 configuration in test_layers

* debug test_book4

* add test_word2vec

* follow comments

* follow comments

* add test_fit_a_line

* Update

* fix persistable bug

* fix elementwise add bug

* set correct attr for bias op in fc layer

* set correct attr for bias op in fc layer

* Update

1. Add init_program to hold initializers
2. bug fix

* add test_fit_a_line

* fix persistable bug

* fix elementwise add bug

* fix type

* add gitignore

* Complete fit_a_line test

* revert code

* Clean up

* Revert "revert code"

This reverts commit eb1aa015.

* Refine

* Fix unit test

* Init

* unify layer names

* Update

* Add pool2d layer

* Test recognize_digits_conv

* Clean up

* Implement FC layer with helper

* Update LayerHelper

* Add debug string for Python ProtoBuf

and Rename `Sync` to `Flush`

* Add check of ProtoBuf initialization

* Layer wrapper for FC

* Fix unittest

* Fix CI

* Add code generator

* AttributeChecker Better error log and speicalize bool

Since lots of types can be cast to bool

* Complete mlp, fit_a_line

* Expose get global scope

* Make global scope not thread-safe

1. It is no need to make global scope thread-safe, since it will be
invoked in Python main thread.
2. Do not free the global scope when C++ exit. Let the OS free memories,
otherwise, we need to handle the destroy dependencies.

See
https://google.github.io/styleguide/cppguide.html#Static_and_Global_Variables

* Fix

* Implementation of simple conv_2d layer

* Stash

* Remove private data members in OpRegister

* Fix bugs

* Stash

* Expose FeedFetchList as VarType

* Change ProgramDesc not a global variable

* Polish code style

* Stash

* Correct implement BlockDesc destructor

* Correct implement BlockDesc destructor

* Unify program as parameter name

* Fix bugs

* Add unittest

* Fix unit test error

* Remove unused functions

* Add clone for Python Program

* Working on executor

* Stash

* Add glog as dependencies of ops

* Use VLOG to logging some information is helpful when we debug Paddle

* Expose VarDesc::persistable to Python

* Test executor

* Complete unittest

* Polish code

* Fix merge error

* Follow comment

* Polish Python Code

* remove Program.instance

* fix test_program.py

* Implement FC layer with helper

* Update LayerHelper

* Add debug string for Python ProtoBuf

and Rename `Sync` to `Flush`

* Add check of ProtoBuf initialization

* Layer wrapper for FC

* Fix unittest

* Fix CI

* Add code generator

* AttributeChecker Better error log and speicalize bool

Since lots of types can be cast to bool

* Complete mlp, fit_a_line

* Implementation of simple conv_2d layer

* Fix bugs

* Change ProgramDesc not a global variable

* Polish code style

* Stash

* Correct implement BlockDesc destructor

* Correct implement BlockDesc destructor

* Unify program as parameter name

* Fix bugs

* Add unittest

* Fix unit test error

* Remove unused functions

* Add clone for Python Program

* Compare OpDescBind directly

* Implement FC layer with helper

* Update LayerHelper

* Add debug string for Python ProtoBuf

and Rename `Sync` to `Flush`

* Add check of ProtoBuf initialization

* Layer wrapper for FC

* Fix unittest

* Fix CI

* Add code generator

* AttributeChecker Better error log and speicalize bool

Since lots of types can be cast to bool

* Complete mlp, fit_a_line

* Implementation of simple conv_2d layer

* Fix bugs

* Correct implement BlockDesc destructor

* Fix bugs

* Fix unit test error

* Follow comments

* Change ProgramDesc not a global variable

* Polish code style

* Correct implement BlockDesc destructor

* Unify program as parameter name

* Implement FC layer with helper

* Update LayerHelper

* Add debug string for Python ProtoBuf

and Rename `Sync` to `Flush`

* Add check of ProtoBuf initialization

* Layer wrapper for FC

* Fix unittest

* Fix CI

* Add code generator

* AttributeChecker Better error log and speicalize bool

Since lots of types can be cast to bool

* Complete mlp, fit_a_line

* Implementation of simple conv_2d layer

* Fix bugs

* Remove debug code

* init parameter base class

* optimize the Comments of optimizer

* basic implimentation of optimizer

* add test_optimizer

* add no_grad_set to interface

* update optimizer.py

* python code can run

* fix some problem

* add sync_with_cpp to Python Program and Block

* sync vars and ops in block from cpp

* optimize code and add some comment

* add more check for sync

* update optimizer with return value of Backward

* rm unused code

* infer shape when create gradient vairiable

* update test_optimizer

* update test_program.py

* update backward test

* follow comment

* Implement FC layer with helper

* Update LayerHelper

* Add debug string for Python ProtoBuf

and Rename `Sync` to `Flush`

* Add check of ProtoBuf initialization

* Layer wrapper for FC

* Fix unittest

* Fix CI

* Add code generator

* AttributeChecker Better error log and speicalize bool

Since lots of types can be cast to bool

* Complete mlp, fit_a_line

* add sync_with_cpp to Python Program and Block

* sync vars and ops in block from cpp

* optimize code and add some comment

* add more check for sync