diff --git a/doc/fluid/user_guides/howto/evaluation_and_debugging/debug/visualdl.md b/doc/fluid/user_guides/howto/evaluation_and_debugging/debug/visualdl.md old mode 100644 new mode 100755 index 747f32c31dffc873fa5ed0459046f93cd86834c0..e301d0a0611fec79310464cdcfd9839876a26ea5 --- a/doc/fluid/user_guides/howto/evaluation_and_debugging/debug/visualdl.md +++ b/doc/fluid/user_guides/howto/evaluation_and_debugging/debug/visualdl.md @@ -13,25 +13,28 @@ VisualDL是一个面向深度学习任务设计的可视化工具,包含了sca 实现原生的性能和定制效果。 ## 组件 -VisualDL 目前支持4种组件: +VisualDL 目前支持以下组件: -- graph - scalar -- image - histogram +- image +- audio +- graph +- high dimensional -### Graph -兼容 ONNX(Open Neural Network Exchange)[https://github.com/onnx/onnx], 通过与 python SDK的结合,VisualDL可以兼容包括 PaddlePaddle, pytorch, mxnet在内的大部分主流DNN平台。 +### Scalar +可以用于展示训练测试的误差趋势

- +

-### Scalar -可以用于展示训练测试的误差趋势 +### Histogram + +用于可视化任何tensor中元素分布的变化趋势

- +

### Image @@ -41,12 +44,21 @@ VisualDL 目前支持4种组件:

-### Histogram +### Audio +可用于播放输入或生成的音频样本 -用于可视化任何tensor中元素分布的变化趋势 +### Graph +兼容 ONNX(Open Neural Network Exchange)[https://github.com/onnx/onnx], 通过与 python SDK的结合,VisualDL可以兼容包括 PaddlePaddle, pytorch, mxnet在内的大部分主流DNN平台。

- + +

+ +### High Dimensional +用高维度数据映射在2D/3D来可视化嵌入 + +

+

## 快速尝试 @@ -58,12 +70,14 @@ pip install --upgrade visualdl # 运行一个例子,vdl_create_scratch_log 将创建测试日志 vdl_create_scratch_log -visualDL --logdir=scratch_log --port=8080 +visualdl --logdir=scratch_log --port=8080 # 访问 http://127.0.0.1:8080 ``` -如果以上步骤出现问题,很可能是因为python或pip不同版本或不同位置所致,以下安装方法能解决。 +如果出现`TypeError: __init__() got an unexpected keyword argument 'file'`, 是因为protobuf不是3.5以上,运行`pip install --upgrade protobuf`就能解决。 + +如果以上步骤还有出现其他问题,很可能是因为python或pip不同版本或不同位置所致,以下安装方法能解决。 ## 使用 virtualenv 安装 @@ -100,13 +114,11 @@ pip install --upgrade visualdl # 运行一个例子,vdl_create_scratch_log 将创建测试日志 vdl_create_scratch_log -visualDL --logdir=scratch_log --port=8080 +visualdl --logdir=scratch_log --port=8080 # 访问 http://127.0.0.1:8080 ``` -如果出现`TypeError: __init__() got an unexpected keyword argument 'file'`, 是因为protobuf不是3.5以上,运行`pip install --upgrade protobuf`就能解决。 - 如果在虚拟环境下仍然遇到安装问题,请尝试以下方法。 @@ -134,7 +146,7 @@ pip install --upgrade visualdl # 运行一个例子,vdl_create_scratch_log 将创建测试日志 vdl_create_scratch_log -visualDL --logdir=scratch_log --port=8080 +visualdl --logdir=scratch_log --port=8080 # 访问 http://127.0.0.1:8080 ``` @@ -151,7 +163,7 @@ python setup.py bdist_wheel pip install --upgrade dist/visualdl-*.whl ``` -如果打包和安装遇到其他问题,不安装只想运行Visual DL可以看[这里](https://github.com/PaddlePaddle/VisualDL/blob/develop/docs/develop/how_to_dev_frontend_cn.md) +如果打包和安装遇到其他问题,不安装只想运行Visual DL可以看[这里](https://github.com/PaddlePaddle/VisualDL/blob/develop/docs/how_to_dev_frontend_en.md) ## SDK @@ -210,11 +222,16 @@ int main() { 当训练过程中已经产生了日志数据,就可以启动board进行实时预览可视化信息 ``` -visualDL --logdir +visualdl --logdir ``` board 还支持一下参数来实现远程的访问: - `--host` 设定IP - `--port` 设定端口 -- `--model_pb` 指定 ONNX 格式的模型文件 +- `-m / --model_pb` 指定 ONNX 格式的模型文件 + +### 贡献 + +VisualDL 是由 [PaddlePaddle](http://www.paddlepaddle.org/) 和 +[ECharts](http://echarts.baidu.com/) 合作推出的开源项目。我们欢迎所有人使用,提意见以及贡献代码。 diff --git a/doc/fluid/user_guides/howto/evaluation_and_debugging/debug/visualdl_en.md b/doc/fluid/user_guides/howto/evaluation_and_debugging/debug/visualdl_en.md new file mode 100755 index 0000000000000000000000000000000000000000..41f7991e29b846d030569ebf713c988ba9c3ce77 --- /dev/null +++ b/doc/fluid/user_guides/howto/evaluation_and_debugging/debug/visualdl_en.md @@ -0,0 +1,253 @@ +# Visual DL Toolset + +

+ +

+ +## Introduction +VisualDL is a deep learning visualization tool that can help design deep learning jobs. +It includes features such as scalar, parameter distribution, model structure and image visualization. +Currently it is being developed at a high pace. +New features will be continuously added. + +At present, most DNN frameworks use Python as their primary language. VisualDL supports Python by nature. +Users can get plentiful visualization results by simply add a few lines of Python code into their model before training. + +Besides Python SDK, VisualDL was writen in C++ on the low level. It also provides C++ SDK that +can be integrated into other platforms. + + +## Component +VisualDL provides following components: + +- scalar +- histogram +- image +- audio +- graph +- high dimensional + +### Scalar +Scalar can be used to show the trends of error during training. + +

+ +

+ +### Histogram +Histogram can be used to visualize parameter distribution and trends for any tensor. + +

+ +

+ +### Image +Image can be used to visualize any tensor or intermediate generated image. + +

+ +

+ +### Audio +Audio can be used to play input audio samples or generated audio samples. + +### Graph +Graph is compatible with ONNX ([Open Neural Network Exchange](https://github.com/onnx/onnx)), +Cooperated with Python SDK, VisualDL can be compatible with most major DNN frameworks, including +PaddlePaddle, PyTorch and MXNet. + +

+ +

+ +### High Dimensional +High Dimensional can be used to visualize data embeddings by projecting high-dimensional data into 2D / 3D. + +

+ +

+ +## Quick Start +To give the VisualDL a quick test, please use the following commands. + +``` +# Install the VisualDL. Preferably under a virtual environment or anaconda. +pip install --upgrade visualdl + +# run a demo, vdl_create_scratch_log will create logs for testing. +vdl_create_scratch_log +visualdl --logdir=scratch_log --port=8080 + +# visit http://127.0.0.1:8080 +``` + +If you encounter the error `TypeError: __init__() got an unexpected keyword argument 'file'`, that is due to protobuf version is not 3.5+,simply run `pip install --upgrade protobuf` will fix the issue. + +If you run into any other issues in above steps, it could be error caused by environmental issues by different python or pip versions. +Following installation methods might fix the issues. + +## Install with Virtualenv + +[Virtualenv](https://virtualenv.pypa.io/en/stable/) creates isolated Python environment that prevents interfering +by other Python programs on the same machine and make sure Python and pip are located properly. + +On macOS, install pip and virtualenv by: +``` +sudo easy_install pip +pip install --upgrade virtualenv +``` + +On Linux, install pip and virtualenv by: +``` +sudo apt-get install python3-pip python3-dev python-virtualenv +``` + +Then create a Virtualenv environment by one of following command: +``` +virtualenv ~/vdl # for Python2.7 +virtualenv -p python3 ~/vdl for Python 3.x +``` + +```~/vdl``` will be your Virtualenv directory, you may choose to install anywhere. + +Activate your Virtualenv environment by: +``` +source ~/vdl/bin/activate +``` + +Now you should be able to install VisualDL and run our demo: + +``` +pip install --upgrade visualdl + +# run a demo, vdl_create_scratch_log will create logs for testing. +vdl_create_scratch_log +visualdl --logdir=scratch_log --port=8080 + +# visit http://127.0.0.1:8080 +``` + +If you still have issues installing VisualDL from Virtualenv, try following installation method. + + +## Install with Anaconda + +Anaconda is a python distribution, with installation and package management tools. Also it is an environment manager, +which provides the facility to create different python environments, each with their own settings. + +Follow the instructions on the [Anaconda download site](https://www.anaconda.com/download) to download and install Anaconda. +Download Python 3.6 version command-Line installer. + +Create a conda environment named ```vdl``` or anything you want by: +``` +conda create -n vdl pip python=2.7 # or python=3.3, etc. +``` + +Activate the conda environment by: +``` +source activate vdl +``` + +Now you should be able to install VisualDL and run our demo: + +``` +pip install --upgrade visualdl + +# run a demo, vdl_create_scratch_log will create logs for testing. +vdl_create_scratch_log +visualdl --logdir=scratch_log --port=8080 + +# visit http://127.0.0.1:8080 +``` + +If you still have issues installing VisualDL, try installing from sources as in following section. + + +### Install from source +``` +#Preferably under a virtualenv or anaconda. +git clone https://github.com/PaddlePaddle/VisualDL.git +cd VisualDL + +python setup.py bdist_wheel +pip install --upgrade dist/visualdl-*.whl +``` + +If there are still issues regarding the ```pip install```, you can still start Visual DL by starting the dev server +[here](https://github.com/PaddlePaddle/VisualDL/blob/develop/docs/how_to_dev_frontend_en.md) + + +## SDK +VisualDL provides both Python SDK and C++ SDK in order to fit more use cases. + + +### Python SDK +VisualDL now supports both Python 2 and Python 3. +Below is an example of creating a simple Scalar component and inserting data from different timestamps: + +```python +import random +from visualdl import LogWriter + +logdir = "./tmp" +logger = LogWriter(logdir, sync_cycle=10000) + +# mark the components with 'train' label. +with logger.mode("train"): + # create a scalar component called 'scalars/scalar0' + scalar0 = logger.scalar("scalars/scalar0") + +# add some records during DL model running. +for step in range(100): + scalar0.add_record(step, random.random()) +``` + +### C++ SDK +Here is the C++ SDK identical to the Python SDK example above: + +```c++ +#include +#include +#include "visualdl/logic/sdk.h" + +namespace vs = visualdl; +namespace cp = visualdl::components; + +int main() { + const std::string dir = "./tmp"; + vs::LogWriter logger(dir, 10000); + + logger.SetMode("train"); + auto tablet = logger.AddTablet("scalars/scalar0"); + + cp::Scalar scalar0(tablet); + + for (int step = 0; step < 1000; step++) { + float v = (float)std::rand() / RAND_MAX; + scalar0.AddRecord(step, v); + } + + return 0; +} +``` + +## Launch Visual DL +After some logs have been generated during training, users can launch Visual DL application to see real-time data visualization by: + + +``` +visualdl --logdir +``` + +visualDL also supports following optional parameters: + +- `--host` set IP +- `--port` set port +- `-m / --model_pb` specify ONNX format for model file to view graph + + +### Contribute + +VisualDL is initially created by [PaddlePaddle](http://www.paddlepaddle.org/) and +[ECharts](http://echarts.baidu.com/). +We welcome everyone to use, comment and contribute to Visual DL :)