scope.md 6.2 KB
Newer Older
Y
Yu Yang 已提交
1
# What is a scope.
Q
qiaolongfei 已提交
2

Y
Yu Yang 已提交
3
## Overview
Q
qiaolongfei 已提交
4

Y
Yibing Liu 已提交
5
Scope is an important concept in programming languages, which defines a program region that a set of bindings between names and entities applies. In a specific scope, a valid name is uniquely associated with an entity, such as a variable. And in another scope, this name may refer to other entity or nothing at all. It clearly restricts the visibility and validity of names in a program. Hence **Scope** is introduced to PaddlePaddle to manage variables in context. But different from the original abstract concept, Scope now becomes an object with two important attributes:
Q
qiaolongfei 已提交
6

Y
Yu Yang 已提交
7 8
- Scope is an association of a name to variable.
- Variables in a parent scope can be retrieved from local scope.
Y
Yibing Liu 已提交
9 10

A detailed explanation of these two attributes goes as following.
Q
qiaolongfei 已提交
11

Y
Yu Yang 已提交
12

Q
qiaolongfei 已提交
13
## Scope is a Container of Variables.
Y
Yu Yang 已提交
14

15 16 17
Scope is used to provide a running environment for Net.

1. Scope mainly has Variables as it's data member.
Q
qiaolongfei 已提交
18

19 20 21 22 23
    Scope is a running environment for Net. Net should get all it need to do computation from a scope, such as data buffer, state(momentum) etc.
    All these data/state can be abstracted and create as variable in Paddle, so the only thing Scope need to care about is Variable.
1. Variable can only be created by Scope.
1. Variable can only be got from Scope.
1. Scope contains methods that are used to manage Variables, such as Create/Get/Delete.
Q
qiaolongfei 已提交
24

25 26 27 28 29 30
    Because we only need to care about Variable, we only need method to manage the lifecycle of Variable.
    - `Create` is used to create a Variable by its name and add the mapping relation.
    - `Get` is used to find a Variable by name.
    - `Delete` is used to remove a Variable because sometimes we want to release memory or other resources.

1. Every variable only belongs to one certain Scope.
Q
qiaolongfei 已提交
31

32
    Variable can not be shared between scopes, if we want to use variables from different scope we can use `Parent scope`.
33 34

1. Scope should destruct all Variables within it when itself is destructed.
Q
qiaolongfei 已提交
35

36
    Because Variable can only be got from Scope, when destroying Scope, we also need to destroy all the Vars in it.
Q
qiaolongfei 已提交
37

38
1. Scope do not contain Operators and have no information to run them.
Q
qiaolongfei 已提交
39

40
    Net is designed to drive the computation, Scope is only used to provide a running environment.
Q
qiaolongfei 已提交
41 42 43 44 45 46 47 48 49 50 51 52

```cpp
class Scope {
 public:
  Variable* CreateVariable(const std::string& name);
  const Variable* GetVariable(const std::string& name) const;
  bool DeleteVariable(const std::string& name);

 private:
    std::unordered_map<std::string, std::shared_ptr<Vairable>> variable_map_;
};
```
Y
Yu Yang 已提交
53

Q
qiaolongfei 已提交
54

Y
Yu Yang 已提交
55 56
## Parent scope and local scope

Y
Yu Yang 已提交
57
Just like [scope](https://en.wikipedia.org/wiki/Scope_(computer_science)) in programming languages, `Scope` in the neural network can also be a local scope. There are two attributes about local scope.
Y
Yu Yang 已提交
58

Y
Yu Yang 已提交
59
1.  We can create local variables in a local scope. When that local scope are destroyed, all local variables should also be destroyed.
Y
Yu Yang 已提交
60
2.  Variables in a parent scope can be retrieved from local scopes of that parent scope, i.e., when user get a variable from a scope, it will try to search this variable in current scope. If there is no such variable in the local scope, `scope` will keep searching from its parent, until the variable is found or there is no parent.
Y
Yu Yang 已提交
61 62 63

```cpp
class Scope {
Y
Yu Yang 已提交
64
 public:
Y
Yu Yang 已提交
65 66
  Scope(const std::shared_ptr<Scope>& scope): parent_(scope) {}

Y
Yu Yang 已提交
67
  Variable* GetVar(const std::string& name) const {
Y
Yu Yang 已提交
68 69 70 71 72 73 74 75 76 77
    Variable* var = GetVarLocally(name);
    if (var != nullptr) {
      return var;
    } else if (parent_ != nullptr) {
      return parent_->Get(name);
    } else {
      return nullptr;
    }
  }

Y
Yu Yang 已提交
78
 private:
Y
Yu Yang 已提交
79 80 81
  std::shared_ptr<Scope> parent_ {nullptr};
};
```
Y
Yu Yang 已提交
82

Y
Yu Yang 已提交
83
In `Scope` class, there is a private data member called `parent_`. `parent_` is a smart pointer to its parent scope. When user `Get` a variable by its `name`, the `name` will be searched inside the current scope. If the variable cannot be found locally and parent scope is not a `nullptr`, the variable will be searched inside that parent scope. `parent_` pointer's default value is `nullptr`. It means that the scope is a global scope when `parent_` is nullptr.
Y
Yu Yang 已提交
84

Y
Yu Yang 已提交
85
A local scope is very useful when we implement Recurrent Neural Network. Each timestep of an RNN should be a `Net`. Each `Net` of timestep (`StepNet` for short) should use an independent local scope. Just like variables in a while loop is inside a local scope in programming languages. By using a single `StepNet` and changing local scope, we can implement an RNN easily.
Y
Yu Yang 已提交
86

Y
Yu Yang 已提交
87
# Interface Design
Y
Yu Yang 已提交
88

Y
Yu Yang 已提交
89 90
```cpp
class Variable {
Y
Yu Yang 已提交
91
 private:
Y
Yu Yang 已提交
92 93 94 95 96
  Variable() = default;
  friend class Scope;
};

class Scope {
Y
Yu Yang 已提交
97
 private:
Y
Yu Yang 已提交
98 99
  Scope(const std::shared_ptr<Scope>& parent = nullptr);

Y
Yu Yang 已提交
100
 public:
Y
Yu Yang 已提交
101 102
  static std::shared_ptr<Scope> Create(const std::shared_ptr<Scope>& parent = nullptr);

Y
Yu Yang 已提交
103
  // return nullptr if not found.
104
  Variable* GetVariable(const std::string& name) const;
Y
Yu Yang 已提交
105 106 107 108

  // return Error if already contains same name variable.
  Error CreateVariable(const std::string& name);

Y
Yu Yang 已提交
109
 private:
Y
Yu Yang 已提交
110
  std::shared_ptr<Scope> parent_;
111
  std::unordered_map<std::string, std::unique_ptr<Scope>> attrs_;
Y
Yu Yang 已提交
112 113 114 115 116 117 118 119
};
```
## Only scope can create a variable

To ensure `only scope can create a variable`, we should mark `Variable`'s constructor as a private member function, and Scope is a friend class of Variable. And then only `CreateVariable` can construct `Variable`.

## When scope destroyed, all variables inside this scope should be destroyed together

120
The scope hold unique pointers for all variables. User can `GetVariable` from scope, but he should not hold this pointer as a member variable. Because when scope is destroyed, all variables inside this scope will be destroyed together.
Y
Yu Yang 已提交
121 122 123 124 125

## Sharing a parent scope

Local scope contains a `parent_` pointer. It is a linked-list for scopes. Using a `shared_ptr` because when a local scope is using, its parents cannot be destroyed.

Y
Yu Yang 已提交
126 127
Also, as the parent scope is a `shared_ptr`, we can only `Create()` a scope shared pointer. We cannot construct a scope variable, because it cannot be passed to other scope as `parent` pointer.

Y
Yu Yang 已提交
128 129 130
## Orthogonal interface

`GetVariable` will return `nullptr` when `name` is not found. It can be used as `Contains` method. `CreateVariable` will return a `Error` when there is a name conflict locally. Combine `GetVariable` and `CreateVariable`, we can implement `CreateOrGetVariable` easily.