未验证 提交 c5f32132 编写于 作者: D daminglu 提交者: GitHub

Translate VisualDL front-end & back-end info (#421)

上级 c5b5dc08
# 后台指南 # 后台指南
VisualDL has three components. VisualDL有三个功能模块.
1. The Python/C++ SDK that logs the data during training. 1. 在训练过程中用来记录日志数据的 Python/C++ SDK。
1. The single page client side app that visualizes training data. 1. 用来可视化训练数据的客户端单页面应用。
1. The server powered by Flask, reads the log data and delivers it to the client side app for displaying graphs (scalars/histograms) and embeddings. 1. 后端 Flask 服务器,用来读取日志数据并提供给前端应用,以便前端可以展示各类图标,如柱状图,嵌入图等等.
This doc will go over Backend / SDK structure with development guide. 这篇文档用来介绍 后端/SDK 的架构,以及指导用户自行开发。
For front end architecture and development guide, looks [here](https://github.com/PaddlePaddle/VisualDL/blob/develop/docs/how_to_dev_frontend_en.md) 关于前端的架构和开发指南,请看 [这里](https://github.com/PaddlePaddle/VisualDL/blob/develop/docs/how_to_dev_frontend_cn.md)
## Code structure ## 代码结构
All backend and sdk logic is under visualdl sub directory 所有后端和 SDK 的逻辑都在 visualdl 子文件夹里面
``` ```
├── build ├── build
...@@ -27,20 +27,21 @@ All backend and sdk logic is under visualdl sub directory ...@@ -27,20 +27,21 @@ All backend and sdk logic is under visualdl sub directory
└── utils - Helper function and classes └── utils - Helper function and classes
``` ```
## Development ## 开发
Any code changes in ```server``` folder, simply run 任何在 ```server``` 文件夹里代码的改动,都可以通过运行以下命令
``` ```
python visualdl/server/visualDL --logdir={LOG_DIR} --port=8080 python visualdl/server/visualDL --logdir={LOG_DIR} --port=8080
``` ```
to restart flask server 来重启 Flask 服务器
Notice if you install visualDL from pip install wheel, make sure resetting PYTHONPATH to get the command work. 需要注意的是如果你通过 `pip install wheel` 来安装 visualDL, 请确认重置 PYTHONPATH,以便是的上述命令可以运行。
Any code changes in ```logic```, ```python```, ```storage```, ```utils``` is modified, need to rebuild core.so or sdk to get effect. 任何在 ```logic```, ```python```, ```storage```, ```utils``` 中的代码修改,都需要重新 build core.so 或 SDK 来确保生效。
要么运行 VisualDL/build.sh 脚本来重新安装所有组件,要么安装特定的目标组件如下:
Either run VisualDL/build.sh that builds everything or a target like this
``` ```
mkdir {VisualDL_ROOT}/build mkdir {VisualDL_ROOT}/build
cd build cd build
...@@ -49,12 +50,12 @@ make {TARGET MODULE} ...@@ -49,12 +50,12 @@ make {TARGET MODULE}
``` ```
## Tests ## 测试
Modification in SDK API or any major function should be adding test cases. 任何关于 SDK API 或其他核心组件的修改都必须提供相应的测试案例。
In {VisualDL_ROOT}/CMakeList.txt, we have a list of test files as target 在 {VisualDL_ROOT}/CMakeList.txt 中,我们有一个测试文件清单
``` ```
add_executable(vl_test add_executable(vl_test
...@@ -70,7 +71,7 @@ add_executable(vl_test ...@@ -70,7 +71,7 @@ add_executable(vl_test
) )
``` ```
Add new test cases in corresponding files where code is modified or create new test files. You can make vl_test target and just run vl_test executable for all test cases. 在相应被修改的代码文件处添加测试用例或者新的测试文件。你可以制作 vl_test 目标或者运行 vl_test 可执行文件来运行所有测试案例。
``` ```
mkdir {VisualDL_ROOT}/build mkdir {VisualDL_ROOT}/build
...@@ -80,45 +81,45 @@ make vl_test ...@@ -80,45 +81,45 @@ make vl_test
./vl_test ./vl_test
``` ```
## Backend / SDK architecture ## 后端 / SDK 架构
### Server ### 服务器
- Implement a server using a lightweight framework Flask, provides two services: - 采用轻量级 Flask 框架来搭建服务器并提供以下两种服务
- Host a main application with support of a front end web app - 搭建主程序来支持前端应用
- Provide a series of HTTP end points, using JSON for front end data communication - 提供一系列 HTTP 访问点,通过返回的 JSON 来和前端沟通
- ```visualDL``` : main app entry point - ```visualDL``` : 主程序入口
- defines server arguments and config - 定义了服务器参数
- provides API and router for front end - 给前端提供 API 和 路由匹配
- ```lib.py``` : delegate wrapper for function calls - ```lib.py``` : 调用库函数的入口
- read and cache log data - 读取并缓存日志数据
- convert data to JSON - 把数据变成 JSON
- retry mechanism - 重试机制
- ```graph.py``` and ```onnx``` - ```graph.py``` ```onnx```
- graph.py defines the logic to read and parse onnx model file - graph.py 定义了读取和解析 ONNX 模型文件的逻辑
- create edges and nodes as JSON format for front end to read - 给前端提供具有结点和边信息的图模型JSON
- onnx folder contains onnx protobuf interface that ports from onnx repo - ONNX 文件夹包含从 ONNX 代码库移植来的 ONNX Protobuf 接口
### Logic & SDK ### 逻辑和 SDK
#### Python SDK #### Python SDK
- ```pybind.cc``` : create python interface from C++ layer - ```pybind.cc``` : 为 C++ 创建 Python 接口
- ```storage.py``` : ```LogReader``` and ```LogWriter``` APIs of Python SDK that maps to pybind - ```storage.py``` : Python SDK 的 ```LogReader``````LogWriter``` API,对应于 pybind
#### C++ SDK #### C++ SDK
- ```sdk.h``` - ```sdk.h```
- defines ```LogReader``` and ```LogWriter``` APIs of C++ SDK - 定义了 ```LogReader``````LogWriter``` 的 C++ SDK 的API
- defines feature components such as ```Scalar```, ```Histogram```, ```Image``` and their corresponding component```Reader``` - 定义了诸如```Scalar```, ```Histogram```, ```Image``` 的功能模块以及它们相应的 ```Reader```
- ```sdk.cc``` : implements ```Reader``` to use ```Storage``` layer for reading and writing - ```sdk.cc``` : 实现了 ```Reader``` 以便使用 ```Storage``` 层来完成读和写
#### Information Container #### 信息容器
- Currently it just handles logic of sync cycle (when to write) - 目前我们仅仅处理写操作时的同步逻辑
#### Storage #### 存储
- Defines log data format as following: - 定义日志数据的格式如下:
- ```Storage``` : main set of data with different modes such as "train" or "test", contain multiple ```Tablet``` - ```Storage``` : 主数据存储地, 包含不同模式,比如 "train" 或 "test", 包含多个 ```Tablet```
- ```Tablet``` : group a single type of data with ```tag```. One tablet will only represent one data type such as scalar, histogram, image and contains multiple ```Record```. For example one tablet represents a line in scalar chart. - ```Tablet``` : 把同一个数据类型用 ```tag``` 打包. 一个面板只代表一个数据类型,比如 ‘数值’, ‘柱状图’, ‘图像’。 可以包含多个 ```Record```。 比如,一个面板可以表达在 ‘数值’ 图像中的一条线。
- ```Record``` : represent any data structure for any component - ```Record``` : 最小单位,代表任何功能模块中的任何数据结构。
- Handles the actual logic to serialize to disk and deserialize from disk - 处理和磁盘之间的 序列化/去序列化 传输
- ```storage.proto```: protobuf file that defines the exact interface to be read/write - ```storage.proto```: 用来定义确切 读/写 接口的 protobuf 文件。
# 前端指南 # 前端指南
VisualDL has three components. VisualDL有三个功能模块.
1. The Python/C++ SDK that logs the data during training. 1. 在训练过程中用来记录日志数据的 Python/C++ SDK。
1. The single page client side app that visualized training data. 1. 用来可视化训练数据的客户端单页面应用。
1. The server (powered on Flask) that reads the logs data and delivers it to the client side app for displaying graphs (scalars/histograms) and embeddings. 1. 后端 Flask 服务器,用来读取日志数据并提供给前端应用,以便前端可以展示各类图标,如柱状图,嵌入图等等.
This article will go over the basic web-app architecture and development guide. 这篇文档用来介绍前端的架构,以及指导用户自行开发。
## TL;DR ## 长话短说
If you are not interested in the details of mumbo jumbo and want to start development as soon as possible, please run the following commands. 如果您不关心细节,而是想马上开始开发,请运行以下命令。
To clone the repo and to prepare the environment 克隆代码库以及准备开发环境
```bash ```bash
git clone git@github.com:PaddlePaddle/VisualDL.git git clone git@github.com:PaddlePaddle/VisualDL.git
cd VisualDL cd VisualDL
...@@ -19,60 +19,62 @@ pip install -r requirements.txt ...@@ -19,60 +19,62 @@ pip install -r requirements.txt
# Set up the node dependencies and the VisualDL SDK # Set up the node dependencies and the VisualDL SDK
./scripts/setup_dev_env.sh ./scripts/setup_dev_env.sh
``` ```
To run the dev server and start coding 运行开发服务器,开始码代码喽~
```bash ```bash
./scripts/start_dev_server.sh --logdir=<LOG_DIR> ./scripts/start_dev_server.sh --logdir=<LOG_DIR>
``` ```
The script will run a dev server at `http://0.0.0.0:8040`. Open a browser and navigate to `http://0.0.0.0:8040/` to see it 这个脚本会在 `http://0.0.0.0:8040` 启动一个开发服务器。打开浏览器,来到 `http://0.0.0.0:8040/`,您就可以发现开发版的 VisualDL 啦~
## Web App Architecture
The VisualDL Web app uses multiple frameworks to help manage the project. They are ## 前端网络应用架构
1. webpack: To manage all assets VisualDL 前端网络应用使用多个框架来管理项目。这些框架包括:
1. npm: To manage dependencies
1. Vue: Javascript Component framework 1. webpack: 管理所有组件
1. ECharts: To pilot charts 1. npm: 管理依赖性
1. Vue: 流行的Javascript组件框架
1. ECharts: 专业的数据可视化库
## Webpack ## Webpack
webpack is a module bundler. Its main purpose is to bundle JavaScript files for usage in a browser. webpack 是一个模块打包框架. 它主要的工作是把前端JavaScript文件带包在一起,提供给浏览器。
For a long time, web developers need to worry about backward compatibility or how to support different versions of browsers. 很长一段时间,网络开发者不得不关心如何向后兼容以及如何支持不同的浏览器。Webpack可以把JavaScript代码同步编译到不同的版本,以便适应不同的浏览器。
webpack can help to transpile the Javascript files to the version where a browser can take. The developers don't need to implement version specific code anymore. 网络开发者不必再担心为了特定的浏览器而订制特定的代码版本。
It can also help to minify and uglify Javascript files. In a nutshell, webpack helps to manage all of your assets (.js, .css, .gif, .sass, etc) so you don't have to. 它也可以 '最小化' 和 '丑化' JavaScript文件。一句话,webpack可以帮助管理你的所有资料(.js, .css, .gif, .sass, etc).
To learn more about [webpack](https://webpack.js.org/). 更多Webpack内容请参考[这里](https://webpack.js.org/).
## npm package ## npm package
npm is the package manager for JavaScript and is a great tool to manage multiple dependencies. To confirm that you have npm installed. Please run this command npm 是一个JavaScript的代码包管理器。而且是一个很优秀的管理多种依赖性的工具。请运行以下命令来确保您已经安装了 npm
``` bash ``` bash
npm -v npm -v
``` ```
If your environment does not have npm already, please [download npm](https://www.npmjs.com/get-npm). 如果您的环境没有npm,请在 [这里](https://www.npmjs.com/get-npm) 下载.
The basic packaging information is stored in `package.json`. It specifies build scripts and dependencies for each environment. 基本的包裹信息在 `package.json`中。它用来定义构造脚本的依赖关系。
npm reads `package.json` to build different package according to the packaging script. npm 读取 `package.json` 来根据打包脚本构造不同的包裹。
If you are not familiar with npm, please refer to the [npm documentation](https://docs.npmjs.com/) for more detail. 如果您对 npm 还不熟,请看 [这里](https://docs.npmjs.com/)
Install the necessary dependencies by running 运行如下命令来安装所需的依赖库
```bash ```bash
# Navigate to the web-app folder # Navigate to the web-app folder
cd VisualDL/frontend cd VisualDL/frontend
npm install npm install
``` ```
This command will go through `package.json` and install the dependencies in the local node_modules folder. 这个命令会逐行检查`package.json`并且安装相应的依赖库到本地的 `node_modules` 文件夹里
## Vue ## Vue
Vue is a JavaScript component framework that helps the developer to implement web component in MVVM architecture pattern. Vue 是一个很潮的 JavaScript 框架。可以帮助开发者用 MVM 架构模式 开发网络组件。
Vue allows you to define a self-contained view model in a .vue file and attach view model objects to DOM objects. Vue 允许开发者自己定义视图模型并保存在 .vue 文件里。然后把视图模型绑定到 DOM 对象上。
To learn more about [Vue](https://vuejs.org/) 更多内容参见 [这里](https://vuejs.org/)
## ECharts ## ECharts
We use ECharts javascript library to render our charts and graphs. ECharts is a leading open source charting library that supports numerous data visualization. 我们使用一个国内流行的 ECharts 来显示图表和图像。 ECharts 是领先的开源前端数据可视化库。它支持多种多样的数据可视化需求。
To learn more about ECharts framework, please visit [ECharts](https://ecomfe.github.io/echarts-doc/public/en/index.html). 更多关于 [ECharts](https://ecomfe.github.io/echarts-doc/public/en/index.html)的信息。
Markdown is supported
0% .
You are about to add 0 people to the discussion. Proceed with caution.
先完成此消息的编辑!
想要评论请 注册