未验证 提交 c5f32132 编写于 作者: D daminglu 提交者: GitHub

Translate VisualDL front-end & back-end info (#421)

上级 c5b5dc08
# 后台指南
VisualDL has three components.
1. The Python/C++ SDK that logs the data during training.
1. The single page client side app that visualizes training data.
1. The server powered by Flask, reads the log data and delivers it to the client side app for displaying graphs (scalars/histograms) and embeddings.
VisualDL有三个功能模块.
1. 在训练过程中用来记录日志数据的 Python/C++ SDK。
1. 用来可视化训练数据的客户端单页面应用。
1. 后端 Flask 服务器,用来读取日志数据并提供给前端应用,以便前端可以展示各类图标,如柱状图,嵌入图等等.
This doc will go over Backend / SDK structure with development guide.
这篇文档用来介绍 后端/SDK 的架构,以及指导用户自行开发。
For front end architecture and development guide, looks [here](https://github.com/PaddlePaddle/VisualDL/blob/develop/docs/how_to_dev_frontend_en.md)
关于前端的架构和开发指南,请看 [这里](https://github.com/PaddlePaddle/VisualDL/blob/develop/docs/how_to_dev_frontend_cn.md)
## Code structure
## 代码结构
All backend and sdk logic is under visualdl sub directory
所有后端和 SDK 的逻辑都在 visualdl 子文件夹里面
```
├── build
......@@ -27,20 +27,21 @@ All backend and sdk logic is under visualdl sub directory
└── utils - Helper function and classes
```
## Development
## 开发
Any code changes in ```server``` folder, simply run
任何在 ```server``` 文件夹里代码的改动,都可以通过运行以下命令
```
python visualdl/server/visualDL --logdir={LOG_DIR} --port=8080
```
to restart flask server
来重启 Flask 服务器
Notice if you install visualDL from pip install wheel, make sure resetting PYTHONPATH to get the command work.
需要注意的是如果你通过 `pip install wheel` 来安装 visualDL, 请确认重置 PYTHONPATH,以便是的上述命令可以运行。
Any code changes in ```logic```, ```python```, ```storage```, ```utils``` is modified, need to rebuild core.so or sdk to get effect.
任何在 ```logic```, ```python```, ```storage```, ```utils``` 中的代码修改,都需要重新 build core.so 或 SDK 来确保生效。
要么运行 VisualDL/build.sh 脚本来重新安装所有组件,要么安装特定的目标组件如下:
Either run VisualDL/build.sh that builds everything or a target like this
```
mkdir {VisualDL_ROOT}/build
cd build
......@@ -49,12 +50,12 @@ make {TARGET MODULE}
```
## Tests
## 测试
Modification in SDK API or any major function should be adding test cases.
任何关于 SDK API 或其他核心组件的修改都必须提供相应的测试案例。
In {VisualDL_ROOT}/CMakeList.txt, we have a list of test files as target
在 {VisualDL_ROOT}/CMakeList.txt 中,我们有一个测试文件清单
```
add_executable(vl_test
......@@ -70,7 +71,7 @@ add_executable(vl_test
)
```
Add new test cases in corresponding files where code is modified or create new test files. You can make vl_test target and just run vl_test executable for all test cases.
在相应被修改的代码文件处添加测试用例或者新的测试文件。你可以制作 vl_test 目标或者运行 vl_test 可执行文件来运行所有测试案例。
```
mkdir {VisualDL_ROOT}/build
......@@ -80,45 +81,45 @@ make vl_test
./vl_test
```
## Backend / SDK architecture
## 后端 / SDK 架构
### Server
### 服务器
- Implement a server using a lightweight framework Flask, provides two services:
- Host a main application with support of a front end web app
- Provide a series of HTTP end points, using JSON for front end data communication
- ```visualDL``` : main app entry point
- defines server arguments and config
- provides API and router for front end
- ```lib.py``` : delegate wrapper for function calls
- read and cache log data
- convert data to JSON
- retry mechanism
- ```graph.py``` and ```onnx```
- graph.py defines the logic to read and parse onnx model file
- create edges and nodes as JSON format for front end to read
- onnx folder contains onnx protobuf interface that ports from onnx repo
- 采用轻量级 Flask 框架来搭建服务器并提供以下两种服务
- 搭建主程序来支持前端应用
- 提供一系列 HTTP 访问点,通过返回的 JSON 来和前端沟通
- ```visualDL``` : 主程序入口
- 定义了服务器参数
- 给前端提供 API 和 路由匹配
- ```lib.py``` : 调用库函数的入口
- 读取并缓存日志数据
- 把数据变成 JSON
- 重试机制
- ```graph.py``` ```onnx```
- graph.py 定义了读取和解析 ONNX 模型文件的逻辑
- 给前端提供具有结点和边信息的图模型JSON
- ONNX 文件夹包含从 ONNX 代码库移植来的 ONNX Protobuf 接口
### Logic & SDK
### 逻辑和 SDK
#### Python SDK
- ```pybind.cc``` : create python interface from C++ layer
- ```storage.py``` : ```LogReader``` and ```LogWriter``` APIs of Python SDK that maps to pybind
- ```pybind.cc``` : 为 C++ 创建 Python 接口
- ```storage.py``` : Python SDK 的 ```LogReader``````LogWriter``` API,对应于 pybind
#### C++ SDK
- ```sdk.h```
- defines ```LogReader``` and ```LogWriter``` APIs of C++ SDK
- defines feature components such as ```Scalar```, ```Histogram```, ```Image``` and their corresponding component```Reader```
- ```sdk.cc``` : implements ```Reader``` to use ```Storage``` layer for reading and writing
#### Information Container
- Currently it just handles logic of sync cycle (when to write)
#### Storage
- Defines log data format as following:
- ```Storage``` : main set of data with different modes such as "train" or "test", contain multiple ```Tablet```
- ```Tablet``` : group a single type of data with ```tag```. One tablet will only represent one data type such as scalar, histogram, image and contains multiple ```Record```. For example one tablet represents a line in scalar chart.
- ```Record``` : represent any data structure for any component
- Handles the actual logic to serialize to disk and deserialize from disk
- ```storage.proto```: protobuf file that defines the exact interface to be read/write
- 定义了 ```LogReader``````LogWriter``` 的 C++ SDK 的API
- 定义了诸如```Scalar```, ```Histogram```, ```Image``` 的功能模块以及它们相应的 ```Reader```
- ```sdk.cc``` : 实现了 ```Reader``` 以便使用 ```Storage``` 层来完成读和写
#### 信息容器
- 目前我们仅仅处理写操作时的同步逻辑
#### 存储
- 定义日志数据的格式如下:
- ```Storage``` : 主数据存储地, 包含不同模式,比如 "train" 或 "test", 包含多个 ```Tablet```
- ```Tablet``` : 把同一个数据类型用 ```tag``` 打包. 一个面板只代表一个数据类型,比如 ‘数值’, ‘柱状图’, ‘图像’。 可以包含多个 ```Record```。 比如,一个面板可以表达在 ‘数值’ 图像中的一条线。
- ```Record``` : 最小单位,代表任何功能模块中的任何数据结构。
- 处理和磁盘之间的 序列化/去序列化 传输
- ```storage.proto```: 用来定义确切 读/写 接口的 protobuf 文件。
# 前端指南
VisualDL has three components.
1. The Python/C++ SDK that logs the data during training.
1. The single page client side app that visualized training data.
1. The server (powered on Flask) that reads the logs data and delivers it to the client side app for displaying graphs (scalars/histograms) and embeddings.
VisualDL有三个功能模块.
1. 在训练过程中用来记录日志数据的 Python/C++ SDK。
1. 用来可视化训练数据的客户端单页面应用。
1. 后端 Flask 服务器,用来读取日志数据并提供给前端应用,以便前端可以展示各类图标,如柱状图,嵌入图等等.
This article will go over the basic web-app architecture and development guide.
这篇文档用来介绍前端的架构,以及指导用户自行开发。
## TL;DR
If you are not interested in the details of mumbo jumbo and want to start development as soon as possible, please run the following commands.
## 长话短说
如果您不关心细节,而是想马上开始开发,请运行以下命令。
To clone the repo and to prepare the environment
克隆代码库以及准备开发环境
```bash
git clone git@github.com:PaddlePaddle/VisualDL.git
cd VisualDL
......@@ -19,60 +19,62 @@ pip install -r requirements.txt
# Set up the node dependencies and the VisualDL SDK
./scripts/setup_dev_env.sh
```
To run the dev server and start coding
运行开发服务器,开始码代码喽~
```bash
./scripts/start_dev_server.sh --logdir=<LOG_DIR>
```
The script will run a dev server at `http://0.0.0.0:8040`. Open a browser and navigate to `http://0.0.0.0:8040/` to see it
这个脚本会在 `http://0.0.0.0:8040` 启动一个开发服务器。打开浏览器,来到 `http://0.0.0.0:8040/`,您就可以发现开发版的 VisualDL 啦~
## Web App Architecture
The VisualDL Web app uses multiple frameworks to help manage the project. They are
## 前端网络应用架构
1. webpack: To manage all assets
1. npm: To manage dependencies
1. Vue: Javascript Component framework
1. ECharts: To pilot charts
VisualDL 前端网络应用使用多个框架来管理项目。这些框架包括:
1. webpack: 管理所有组件
1. npm: 管理依赖性
1. Vue: 流行的Javascript组件框架
1. ECharts: 专业的数据可视化库
## Webpack
webpack is a module bundler. Its main purpose is to bundle JavaScript files for usage in a browser.
webpack 是一个模块打包框架. 它主要的工作是把前端JavaScript文件带包在一起,提供给浏览器。
For a long time, web developers need to worry about backward compatibility or how to support different versions of browsers.
webpack can help to transpile the Javascript files to the version where a browser can take. The developers don't need to implement version specific code anymore.
It can also help to minify and uglify Javascript files. In a nutshell, webpack helps to manage all of your assets (.js, .css, .gif, .sass, etc) so you don't have to.
很长一段时间,网络开发者不得不关心如何向后兼容以及如何支持不同的浏览器。Webpack可以把JavaScript代码同步编译到不同的版本,以便适应不同的浏览器。
网络开发者不必再担心为了特定的浏览器而订制特定的代码版本。
它也可以 '最小化' 和 '丑化' JavaScript文件。一句话,webpack可以帮助管理你的所有资料(.js, .css, .gif, .sass, etc).
To learn more about [webpack](https://webpack.js.org/).
更多Webpack内容请参考[这里](https://webpack.js.org/).
## npm package
npm is the package manager for JavaScript and is a great tool to manage multiple dependencies. To confirm that you have npm installed. Please run this command
npm 是一个JavaScript的代码包管理器。而且是一个很优秀的管理多种依赖性的工具。请运行以下命令来确保您已经安装了 npm
``` bash
npm -v
```
If your environment does not have npm already, please [download npm](https://www.npmjs.com/get-npm).
如果您的环境没有npm,请在 [这里](https://www.npmjs.com/get-npm) 下载.
The basic packaging information is stored in `package.json`. It specifies build scripts and dependencies for each environment.
npm reads `package.json` to build different package according to the packaging script.
If you are not familiar with npm, please refer to the [npm documentation](https://docs.npmjs.com/) for more detail.
基本的包裹信息在 `package.json`中。它用来定义构造脚本的依赖关系。
npm 读取 `package.json` 来根据打包脚本构造不同的包裹。
如果您对 npm 还不熟,请看 [这里](https://docs.npmjs.com/)
Install the necessary dependencies by running
运行如下命令来安装所需的依赖库
```bash
# Navigate to the web-app folder
cd VisualDL/frontend
npm install
```
This command will go through `package.json` and install the dependencies in the local node_modules folder.
这个命令会逐行检查`package.json`并且安装相应的依赖库到本地的 `node_modules` 文件夹里
## Vue
Vue is a JavaScript component framework that helps the developer to implement web component in MVVM architecture pattern.
Vue 是一个很潮的 JavaScript 框架。可以帮助开发者用 MVM 架构模式 开发网络组件。
Vue allows you to define a self-contained view model in a .vue file and attach view model objects to DOM objects.
Vue 允许开发者自己定义视图模型并保存在 .vue 文件里。然后把视图模型绑定到 DOM 对象上。
To learn more about [Vue](https://vuejs.org/)
更多内容参见 [这里](https://vuejs.org/)
## ECharts
We use ECharts javascript library to render our charts and graphs. ECharts is a leading open source charting library that supports numerous data visualization.
我们使用一个国内流行的 ECharts 来显示图表和图像。 ECharts 是领先的开源前端数据可视化库。它支持多种多样的数据可视化需求。
To learn more about ECharts framework, please visit [ECharts](https://ecomfe.github.io/echarts-doc/public/en/index.html).
更多关于 [ECharts](https://ecomfe.github.io/echarts-doc/public/en/index.html)的信息。
Markdown is supported
0% .
You are about to add 0 people to the discussion. Proceed with caution.
先完成此消息的编辑!
想要评论请 注册