Skip to content

V
VisualDL

PaddlePaddle / VisualDL
大约 1 年 前同步成功

通知 88
Star 4655
Fork 642
V
VisualDL
未验证 提交 917c7204 编写于 作者: 走神的阿圆's avatar 走神的阿圆 提交者： GitHub
浏览文件
操作

Add Chinese docs for visualdl 2.0 (#619)

上级 e4a5f266
显示空白变更内容
内联 并排
Showing with 468 addition and 209 deletion
README.md
浏览文件 @ 917c7204
......@@ -4,177 +4,79 @@
[![License](https://img.shields.io/badge/license-Apache%202-blue.svg)](LICENSE)
<p align="center">
<img src="https://raw.githubusercontent.com/PaddlePaddle/VisualDL/develop/docs/images/vs-logo.png" width="60%" />
<img src="/frontend/packages/core/public/images/logo-visualdl.svg" width="60%"/>
</p>
## Introduction
VisualDL is a deep learning visualization tool that can help design deep learning jobs.
It includes features such as scalar, parameter distribution, model structure and image visualization.
Currently it is being developed at a high pace.
New features will be continuously added.
## 介绍
VisualDL是一个面向深度学习任务设计的可视化工具，利用丰富的图表展示数据，用户可以更直观、清晰地查看数据的特征与变化趋势，有助于分析数据、及时发现错误，进而改进神经网络模型的设计。
At present, most DNN frameworks use Python as their primary language. VisualDL supports Python by nature.
Users can get plentiful visualization results by simply add a few lines of Python code into their model before training.
目前，VisualDL支持Scalar, Image, High Dimensional, Graph 这四个组件，项目正处于高速迭代中，新的组件会不断加入。
Besides Python SDK, VisualDL was writen in C++ on the low level. It also provides C++ SDK that
can be integrated into other platforms.
由于大多数DNN平台均使用Python作为配置语言，VisualDL原生支持python的使用，
通过在模型的Python配置中添加几行，便可以为训练过程提供丰富的可视化支持。
## Component
VisualDL provides following components:
## 组件
VisualDL 目前支持以下组件：
- scalar
- histogram
- image
- audio
- graph
- high dimensional
- graph
### Scalar
Scalar can be used to show the trends of error during training.
可以用于展示训练测试的指标变化趋势
<p align="center">
<img src="https://raw.githubusercontent.com/daming-lu/large_files/master/loss_scalar.gif" width="60%"/>
<img src="/docs/images/scalar_test.png" width="100%"/>
</p>
### Histogram
Histogram can be used to visualize parameter distribution and trends for any tensor.
<p align="center">
<img src="https://raw.githubusercontent.com/daming-lu/large_files/master/histogram.gif" width="60%"/>
</p>
### Image
Image can be used to visualize any tensor or intermediate generated image.
可以用于可视化任何图片，包括模型训练和预测得到的结果
<p align="center">
<img src="https://raw.githubusercontent.com/daming-lu/large_files/master/loss_image.gif" width="60%"/>
<img src="/docs/images/image_test.png" width="100%"/>
</p>
### Audio
Audio can be used to play input audio samples or generated audio samples.
### Graph
VisualDL graph supports displaying paddle model, furthermore is compatible with ONNX ([Open Neural Network Exchange](https://github.com/onnx/onnx)),
Cooperated with Python SDK, VisualDL can be compatible with most major DNN frameworks, including
PaddlePaddle, PyTorch and MXNet.
<p align="center">
<img src="https://raw.githubusercontent.com/PaddlePaddle/VisualDL/develop/docs/images/graph_demo.gif" width="60%" />
</p>
To display the paddle model, all you have to do is:
1. call the `fluid.io.save_inference_model()`interface to save paddle model
2. use `visualdl --model_pb [paddle_model_dir]` to load paddle model in command line
### High Dimensional
High Dimensional can be used to visualize data embeddings by projecting high-dimensional data into 2D / 3D.
可以用于将高维度数据映射到2D/3D可实现可视化
<p align="center">
<img src="https://raw.githubusercontent.com/PaddlePaddle/VisualDL/develop/docs/getting_started/high_dimensional_3d.png" width="60%"/>
<img src="/docs/images/high_dimensional_test.png" width="100%"/>
</p>
## Quick Start
To give the VisualDL a quick test, please use the following commands.
## 安装
### 使用 Anaconda 安装
```
# Install the VisualDL. Preferably under a virtual environment or anaconda.
pip install --upgrade visualdl
Anaconda是一个用于科学计算的Python发行版，提供了包管理与环境管理的功能，可以很方便地解决多版本python并存、切换以及各种第三方包安装问题。
# run a demo, vdl_create_scratch_log will create logs for testing.
vdl_create_scratch_log
visualdl --logdir=scratch_log --port=8080
请根据[Anaconda下载网站](https://www.anaconda.com/download) 的指示去下载和安装Anaconda.
下载Python 3.6版本的command-Line installer.
# visit http://127.0.0.1:8080
创建conda环境名字为```vdl```或任何名字:
```
If you encounter the error `TypeError: __init__() got an unexpected keyword argument 'file'`, that is due to protobuf version is not 3.5+，simply run `pip install --upgrade protobuf` will fix the issue.
If you run into any other issues in above steps, it could be error caused by environmental issues by different python or pip versions.
Following installation methods might fix the issues.
## Install with Virtualenv
[Virtualenv](https://virtualenv.pypa.io/en/stable/) creates isolated Python environment that prevents interfering
by other Python programs on the same machine and make sure Python and pip are located properly.
On macOS, install pip and virtualenv by:
```
sudo easy_install pip
pip install --upgrade virtualenv
```
On Linux, install pip and virtualenv by:
```
sudo apt-get install python3-pip python3-dev python-virtualenv
```
Then create a Virtualenv environment by one of following command:
conda create -n vdl python=3.7
```
virtualenv ~/vdl # for Python2.7
virtualenv -p python3 ~/vdl for Python 3.x
```
```~/vdl``` will be your Virtualenv directory, you may choose to install anywhere.
Activate your Virtualenv environment by:
```
source ~/vdl/bin/activate
```
Now you should be able to install VisualDL and run our demo:
```
pip install --upgrade visualdl
# run a demo, vdl_create_scratch_log will create logs for testing.
vdl_create_scratch_log
visualdl --logdir=scratch_log --port=8080
# visit http://127.0.0.1:8080
```
If you still have issues installing VisualDL from Virtualenv, try following installation method.
## Install with Anaconda
Anaconda is a python distribution, with installation and package management tools. Also it is an environment manager,
which provides the facility to create different python environments, each with their own settings.
Follow the instructions on the [Anaconda download site](https://www.anaconda.com/download) to download and install Anaconda.
Download Python 3.6 version command-Line installer.
Create a conda environment named ```vdl``` or anything you want by:
```
conda create -n vdl pip python=2.7 # or python=3.3, etc.
```
Activate the conda environment by:
激活conda环境如下:
```
source activate vdl
```
Now you should be able to install VisualDL and run our demo:
现在再安装 VisualDL 和运行范例：
```
```shell
pip install --upgrade visualdl
# run a demo, vdl_create_scratch_log will create logs for testing.
vdl_create_scratch_log
visualdl --logdir=scratch_log --port=8080
# visit http://127.0.0.1:8080
```
If you still have issues installing VisualDL, try installing from sources as in following section.
也可以使用源代码安装方式，能够获取最新的VisualDL更新。
### 使用代码安装
### Install from source
```
#Preferably under a virtualenv or anaconda.
建议在anaconda环境下进行安装。
```shell
git clone https://github.com/PaddlePaddle/VisualDL.git
cd VisualDL
......@@ -182,94 +84,43 @@ python setup.py bdist_wheel
pip install --upgrade dist/visualdl-*.whl
```
If there are still issues regarding the ```pip install```, you can still start Visual DL by starting the dev server
[here](https://github.com/PaddlePaddle/VisualDL/blob/develop/docs/how_to_dev_frontend_en.md)
## SDK
VisualDL provides both Python SDK and C++ SDK in order to fit more use cases.
### Python SDK
VisualDL now supports both Python 2 and Python 3.
Below is an example of creating a simple Scalar component and inserting data from different timestamps:
以最简单的Scalar组件为例，尝试创建一个日志记录多个scalar的数据步骤如下：
```python
import random
from visualdl import LogWriter
logdir = "./tmp"
logger = LogWriter(logdir, sync_cycle=10000)
# mark the components with 'train' label.
with logger.mode("train"):
# create a scalar component called 'scalars/scalar0'
scalar0 = logger.scalar("scalars/scalar0")
# add some records during DL model running.
for step in range(100):
scalar0.add_record(step, random.random())
with LogWriter(logdir="./tmp") as writer:
for step in range(100):
writer.add_scalar(step=step, value=step*2)
```
### C++ SDK
Here is the C++ SDK identical to the Python SDK example above:
```c++
#include <cstdlib>
#include <string>
#include "visualdl/logic/sdk.h"
namespace vs = visualdl;
namespace cp = visualdl::components;
int main() {
const std::string dir = "./tmp";
vs::LogWriter logger(dir, 10000);
logger.SetMode("train");
auto tablet = logger.AddTablet("scalars/scalar0");
cp::Scalar<float> scalar0(tablet);
for (int step = 0; step < 1000; step++) {
float v = (float)std::rand() / RAND_MAX;
scalar0.AddRecord(step, v);
}
return 0;
}
```
## Launch Visual DL
After some logs have been generated during training, users can launch Visual DL application to see real-time data visualization by:
### Startup in command line
## 启动Board
当训练过程中已经产生了日志数据，就可以启动board进行实时预览可视化信息
### 在命令行中启动
```
visualdl --logdir <some log dir>
```
visualDL also supports following optional parameters:
board 还支持一些参数来实现远程的访问：
- `--host` set IP
- `--port` set port
- `-m / --model_pb` specify ONNX format for model file to view graph
### Startup in python script
- `--host` 设定IP
- `--port` 设定端口
### 在Python脚本中启动
```python
>>> from visualdl.server import app
>>> app.run(logdir="SOME_LOG_DIR")
```
`app.run()` support all parameters for command line startup, in addition, you can also specify `pen_browser=True` to open browser automatically。
### Contribute
`app.run()`支持命令行启动的所有参数，除此之外，还可以通过指定`open_browser=True`，自动打开浏览器。
## 贡献
VisualDL is initially created by [PaddlePaddle](http://www.paddlepaddle.org/) and
[ECharts](http://echarts.baidu.com/).
We welcome everyone to use, comment and contribute to Visual DL :)
VisualDL 是由 [PaddlePaddle](http://www.paddlepaddle.org/) 和 [ECharts](http://echarts.baidu.com/) 合作推出的开源项目。我们欢迎所有人使用，提意见以及贡献代码。
## More details
## 更多细节
For more details about how to use VisualDL, please take a look at [documents](https://github.com/PaddlePaddle/VisualDL/tree/develop/demo)
想了解更多关于VisualDL的使用介绍，请查看[文档](./docs/README.md)
VERSION_NUMBER
浏览文件 @ 917c7204
2.0.0-alpha.0
2.0.0-alpha.1
demo/components/high_dimensional_test.py 0 → 100644
浏览文件 @ 917c7204
# Copyright (c) 2020 VisualDL Authors. All Rights Reserve.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# =======================================================================
# coding=utf-8
from visualdl import LogWriter
if __name__ == '__main__':
hot_vectors = [
[1.3561076367500755, 1.3116267195134017, 1.6785401875616097],
[1.1039614644440658, 1.8891609992484688, 1.32030488587171],
[1.9924524852447711, 1.9358920727142739, 1.2124401279391606],
[1.4129542689796446, 1.7372166387197474, 1.7317806077076527],
[1.3913371800587777, 1.4684674577930312, 1.5214136352476377]]
labels = ["label_1", "label_2", "label_3", "label_4", "label_5"]
with LogWriter(logdir="./log/high_dimensional_test/train") as writer:
writer.add_embeddings(tag='default',
labels=labels,
hot_vectors=hot_vectors)
demo/components/image_test.py 0 → 100644
浏览文件 @ 917c7204
# Copyright (c) 2020 VisualDL Authors. All Rights Reserve.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# =======================================================================
# coding=utf-8
import numpy as np
from PIL import Image
from visualdl import LogWriter
def random_crop(img):
"""Get random block of img, which size is 100x100.
"""
img = Image.open(img)
w, h = img.size
random_w = np.random.randint(0, w - 100)
random_h = np.random.randint(0, h - 100)
r = img.crop((random_w, random_h, random_w + 100, random_h + 100))
return np.asarray(r)
if __name__ == '__main__':
with LogWriter(logdir="./log/image_test/train") as writer:
for step in range(6):
writer.add_image(tag="doge",
img=random_crop("../../docs/images/dog.jpg"),
step=step)
demo/components/scalar_test.py 0 → 100644
浏览文件 @ 917c7204
# Copyright (c) 2020 VisualDL Authors. All Rights Reserve.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# =======================================================================
from visualdl import LogWriter
if __name__ == '__main__':
value = [i/1000.0 for i in range(1000)]
with LogWriter(logdir="./log/scalar_test/train") as writer:
for step in range(1000):
writer.add_scalar(tag="acc", step=step, value=value[step])
writer.add_scalar(tag="loss", step=step, value=1/(value[step] + 1))
docs/README.md 0 → 100644
浏览文件 @ 917c7204
<p align="center">
<img src="/frontend/packages/core/public/images/logo-visualdl.svg" width="60%"/>
</p>
## 介绍
VisualDL是一个面向深度学习任务设计的可视化工具，利用丰富的图表展示数据，用户可以更直观、清晰地查看数据的特征与变化趋势，有助于分析数据、及时发现错误，进而改进神经网络模型的设计。
目前大多数DNN平台均使用Python作为配置语言，VisualDL原生支持python的使用，
通过在模型的Python配置中添加几行，便可以为训练过程提供丰富的可视化支持。
VisualDL拥有不同种类的组件，能够帮助开发者有效记录和监测训练中的各种指标和数据，为模型训练数据参考，及时发现错误，进而改进神经网络模型的设计，关于组件的具体信息可以参考[VisualDL组件](./components/README.md)。
## 特点
### 1 动态展示
在启动VisualDL Board后，LogReader将不断增量的读取日志中数据并供前端调用展示，因此能够在训练中同步观测指标变化，如下图：
<p align="center">
<img src="./images/dynamic_display.gif" width="60%"/>
</p>
### 2 多实验对比
VisualDL支持多实验对比，只需要在启动VisualDL Board的时候将每个实验日志所在路径同时传入即可，每个实验中相同tag的指标将绘制在一张图中同步呈现，如下图：
<p align="center">
<img src="./images/multi_experiments.gif" width="80%"/>
</p>
## 安装
建议在anaconda环境下进行安装，便于开发环境的隔离，pip安装命令为：
```shell
python -m pip install visualdl
```
需要注意，由于官方不再维护Python2，因此VisualDL在v2.0.0之后的版本仅支持Python3
## 使用
### 记录日志
VisualDL的后端提供了Python SDK，可通过LogWriter定制一个日志记录器，接口如下：
```python
class LogWriter(
logdir=None,
comment='',
max_queue=10,
flush_secs=120,
filename_suffix='',
write_to_disk=True,
**kwargs
)
```
接口参数说明如下：
|参数|格式|含义|
|-|-|-|
|logdir|string|日志文件所在的路径，VisualDL将在此路径下建立日志文件并进行记录，如果不填则默认为`runs/${CURRENT_TIME}`|
|comment|string|为日志文件夹名添加后缀，如果制定了logdir则此项无效|
|max_queue|int|日志记录消息队列的最大容量，达到此容量则立即写入到日志文件|
|flush_secs|int|日志记录消息队列的最大缓存时间，达到此时间则立即写入到日志文件|
|filename_suffix|string|为默认的日志文件名添加后缀|
|write_to_disk|boolean|是否写入到磁盘|
初始化并记录标量数据到日志文件的过程如下：
```python
from visualdl import LogWriter
# 在`./log/scalar_test/train`路径下建立日志文件
with LogWriter(logdir="./log/scalar_test/train") as writer:
# 使用scalar组件记录一个标量数据
writer.add_scalar(tag="acc", step=1, value=0.5678)
writer.add_scalar(tag="acc", step=2, value=0.6878)
writer.add_scalar(tag="acc", step=3, value=0.9878)
```
关于LogWriter组件的更多具体用法，请参考[VisualDL组件](./components/README.md)
### 启动面板
我们已经记录了三个标量数据到日志中，现在可以启动VisualDL面板查看日志的可视化结果，启动方式分为两种：
#### 在命令行启动
我们可以使用命令行启动VisualDL面板，命令格式为：
```shell
visualdl --logdir <dir_1, dir_2, ... , dir_n> --host <host> --port <port>
```
参数意义分别为：
- `--logdir` 设定日志所在目录，可以指定多个目录，VisualDL将遍历并且迭代寻找指定目录的子目录，将所有实验结果进行可视化
- `--host` 设定IP，默认为`127.0.0.1`
- `--port` 设定端口，默认为`8040`
针对上一步生成的日志，我们的启动命令为
```shell
visualdl --logdir ./log
```
#### 在Python脚本中启动
我们还可以在Python脚本中启动VisualDL面板，接口如下：
```python
visualdl.server.app.run(logdir,
host="127.0.0.1",
port=8080,
cache_timeout=20,
language=None,
open_browser=False)
```
接口参数说明如下：
|参数|格式|含义|
|-|-|-|
|logdir|string或list[string_1, string_2, ... , string_n]|日志文件所在的路径，VisualDL将在此路径下递归搜索日志文件并进行可视化，可指定单个或多个路径|
|host|string|指定启动服务的ip，默认为`127.0.0.1`|
|port|int|启动服务端口，默认为`8040`|
|cache_timeout|int|后端缓存时间，在缓存时间内前端多次请求同一url，返回的数据从缓存中获取，默认为20秒|
|language|string|VisualDL面板语言，可指定为'EN'或'CN'，默认自动匹配操作系统使用语言|
|open_browser|boolean|是否打开浏览器，设置为True则在启动后自动打开浏览器并访问VisualDL面板|
针对上一步生成的日志，我们的启动脚本为：
```python
from visualdl.server import app
app.run(logdir="./log")
```
在使用任意一种方式启动VisualDL面板后，我们可以打开浏览器访问VisualDL面板，在面板上就可以看到日志的可视化结果，如图：
<p align="center">
<img src="./images/3points_demo.png" width="60%"/>
</p>
docs/components/README.md 0 → 100644
浏览文件 @ 917c7204
## 介绍
VisualDL拥有不同种类的组件，能够帮助开发者有效记录和监测训练中的各种指标和数据，为模型训练数据参考，及时发现错误，进而改进神经网络模型的设计
使用组件之前必须先使用LogWriter创建一个记录器，如下：
```python
from visualdl import LogWriter
with LogWriter(logdir="./log") as writer:
```
关于记录器的更多参数配置，请参考[VisualDL说明文档](../README.md)
## Scalar -- 折线图组件
### 介绍
Scalar 组件的输入数据类型为标量，该组件的作用是画折线图。将损失函数值、准确率等标量数据作为参数传入 scalar 组件，即可画出折线图，便于观察变化趋势。
### 记录接口
Scalar 组件的记录接口如下：
```python
add_scalar(tag, value, step, walltime=None)
```
接口参数说明如下：
|参数|格式|含义|
|-|-|-|
|tag|string|记录指标的标志，如`train/loss`，不能含有`%`|
|value|float|要记录的数据值|
|step|int|记录的步数|
|walltime|int|记录数据的时间戳，默认为当前时间戳|
### demo
下面展示了使用 Scalar 组件记录数据的示例，代码见[Scalar组件](../../demo/components/scalar_test.py)
```python
from visualdl import LogWriter
if __name__ == '__main__':
value = [i/1000.0 for i in range(1000)]
# 初始化一个记录器
with LogWriter(logdir="./log/scalar_test/train") as writer:
for step in range(1000):
# 向记录器添加一个tag为`acc`的数据
writer.add_scalar(tag="acc", step=step, value=value[step])
# 向记录器添加一个tag为`loss`的数据
writer.add_scalar(tag="loss", step=step, value=1/(value[step] + 1))
```
运行上述程序后，在命令行执行
```shell
visualdl --logdir ./log --port 8080
```
接着在浏览器打开`http://127.0.0.1:8080`，即可查看以下折线图。
<p align="center">
<img src="../images/scalar_test.png" width="90%"/>
</p>
## Image -- 图片可视化组件
### 介绍
Image 组件用于显示图片。在程序运行过程中，将图片数据传入 Image 组件，就可在 VisualDL 的前端网页看到相应图片。
### 记录接口
Image 组件的记录接口如下：
```python
add_image(tag, img, step, walltime=None)
```
接口参数说明如下：
|参数|格式|含义|
|-|-|-|
|tag|string|记录指标的标志，如`train/loss`，不能含有`%`|
|img|numpy.ndarray|以ndarray格式表示的图片|
|step|int|记录的步数|
|walltime|int|记录数据的时间戳，默认为当前时间戳|
### demo
下面展示了使用 Image 组件记录数据的示例，代码见[Image组件](../../demo/components/image_test.py)
```python
import numpy as np
from PIL import Image
from visualdl import LogWriter
def random_crop(img):
"""获取图片的随机 100x100 分片
"""
img = Image.open(img)
w, h = img.size
random_w = np.random.randint(0, w - 100)
random_h = np.random.randint(0, h - 100)
r = img.crop((random_w, random_h, random_w + 100, random_h + 100))
return np.asarray(r)
if __name__ == '__main__':
# 初始化一个记录器
with LogWriter(logdir="./log/image_test/train") as writer:
for step in range(6):
# 添加一个图片数据
writer.add_image(tag="doge",
img=random_crop("../../docs/images/dog.jpg"),
step=step)
```
运行上述程序后，在命令行执行
```shell
visualdl --logdir ./log --port 8080
```
接着在浏览器打开`http://127.0.0.1:8080`，即可拖动查看记录的不同图片。
<p align="center">
<img src="../images/dynamic_image.gif" width="60%"/>
</p>
## High Dimensional -- 数据降维组件
### 介绍
High Dimensional 组件的作用就是将数据映射到 2D/3D 空间来做可视化嵌入，这有利于了解不同数据的相关性。High Dimensional 组件支持以下两种降维算法：
- PCA : Principle Component Analysis 主成分分析
- t-SNE : t-distributed stochastic neighbor embedding t-分布式随机领域嵌入
### 记录接口
High Dimensional 组件的记录接口如下：
```python
add_embeddings(tag, labels, hot_vectors, walltime=None)
```
接口参数说明如下：
|参数|格式|含义|
|-|-|-|
|tag|string|记录指标的标志，如`default`，不能含有`%`|
|labels|numpy.array 或 list|一维数组表示的标签，每个元素是一个string类型的字符串|
|hot_vectors|numpy.array or list|与labels一一对应，每个元素可以看作是某个标签的特征|
|walltime|int|记录数据的时间戳，默认为当前时间戳|
### demo
下面展示了使用 High Dimensional 组件记录数据的示例，代码见[High Dimensional组件](../../demo/components/high_dimensional_test.py)
```python
from visualdl import LogWriter
if __name__ == '__main__':
hot_vectors = [
[1.3561076367500755, 1.3116267195134017, 1.6785401875616097],
[1.1039614644440658, 1.8891609992484688, 1.32030488587171],
[1.9924524852447711, 1.9358920727142739, 1.2124401279391606],
[1.4129542689796446, 1.7372166387197474, 1.7317806077076527],
[1.3913371800587777, 1.4684674577930312, 1.5214136352476377]]
labels = ["label_1", "label_2", "label_3", "label_4", "label_5"]
# 初始化一个记录器
with LogWriter(logdir="./log/high_dimensional_test/train") as writer:
# 将一组labels和对应的hot_vectors传入记录器进行记录
writer.add_embeddings(tag='default',
labels=labels,
hot_vectors=hot_vectors)
```
运行上述程序后，在命令行执行
```shell
visualdl --logdir ./log --port 8080
```
接着在浏览器打开`http://127.0.0.1:8080`，即可查看降维后的可视化数据。
<p align="center">
<img src="../images/dynamic_high_dimensional.gif" width="80%"/>
</p>
setup.py
浏览文件 @ 917c7204
......@@ -101,6 +101,7 @@ setup(
license=LICENSE,
keywords="visualization deeplearning",
long_description=read('README.md'),
long_description_content_type='text/markdown',
install_requires=REQUIRED_PACKAGES,
package_data={
'visualdl.server': [('dist' + ('/*' * n)) for n in range(1, 20)],
......
visualdl/io/bfile.py
浏览文件 @ 917c7204
......@@ -112,7 +112,10 @@ class BFile(object):
return self
def __exit__(self, exc_type, exc_val, exc_tb):
pass
self.close()
self.buff = None
self.buff_offset = 0
self.continuation_token = None
def __iter__(self):
return self
......
visualdl/reader/reader.py
浏览文件 @ 917c7204
......@@ -45,7 +45,11 @@ class LogReader(object):
Args:
logdir: The dir include vdl log files, multiple subfolders allowed.
"""
if isinstance(logdir, str):
self.dir = [logdir]
else:
self.dir = logdir
self.reader = None
self.readers = {}
self.walks = None
......@@ -87,7 +91,8 @@ class LogReader(object):
def get_all_walk(self):
self.walks = {}
for root, dirs, files in bfile.walk(self.dir):
for dir in self.dir:
for root, dirs, files in bfile.walk(dir):
self.walks.update({root: files})
def components_listing(self):
......@@ -195,6 +200,8 @@ class LogReader(object):
return self._tags
def components(self):
"""Get components type used by vdl.
"""
return list(set(self._tags.values()))
def load_new_data(self, update=True):
......
visualdl/server/app.py
浏览文件 @ 917c7204
......@@ -105,6 +105,7 @@ def parse_args():
required=True,
action="store",
dest="logdir",
nargs="+",
help="log file directory")
parser.add_argument(
"--cache_timeout",
......@@ -148,11 +149,8 @@ def create_app(args):
app.config['BABEL_DEFAULT_LOCALE'] = default_language
babel = Babel(app)
log_reader = LogReader(args.logdir)
# mannully put graph's image on this path also works.
graph_image_path = os.path.join(args.logdir, 'graph.jpg')
# use a memory cache to reduce disk reading frequency.
CACHE = MemCache(timeout=args.cache_timeout)
cache_get = lib.cache_get(CACHE)
......@@ -237,6 +235,13 @@ def create_app(args):
result = gen_result(0, "", data)
return Response(json.dumps(result), mimetype='application/json')
@app.route("/api/embeddings/tags")
def embeddings_tags():
data = cache_get("/data/plugin/embeddings/tags", try_call,
lib.get_embeddings_tags, log_reader)
result = gen_result(0, "", data)
return Response(json.dumps(result), mimetype='application/json')
@app.route('/api/scalars/list')
def scalars():
run = request.args.get('run')
......@@ -273,12 +278,13 @@ def create_app(args):
@app.route('/api/embeddings/embedding')
def embeddings():
run = request.args.get('run')
tag = request.args.get('tag')
dimension = request.args.get('dimension')
reduction = request.args.get('reduction')
key = os.path.join('/data/plugin/embeddings/embeddings', run,
dimension, reduction)
data = cache_get(key, try_call, lib.get_embeddings, log_reader, run,
reduction, int(dimension))
tag, reduction, int(dimension))
result = gen_result(0, "", data)
return Response(json.dumps(result), mimetype='application/json')
......@@ -347,7 +353,7 @@ def _run(logdir,
def run(logdir,
host="127.0.0.1",
port=8080,
port=8040,
model_pb="",
cache_timeout=20,
language=None,
......@@ -371,7 +377,7 @@ def main():
args = parse_args()
logger.info(" port=" + str(args.port))
app = create_app(args=args)
app.run(debug=False, host=args.host, port=args.port, threaded=True)
app.run(debug=False, host=args.host, port=args.port, threaded=False)
if __name__ == "__main__":
......
visualdl/server/lib.py
浏览文件 @ 917c7204
......@@ -104,8 +104,11 @@ def get_individual_audio(log_reader, run, tag, step_index):
return result
def get_embeddings(log_reader, run, reduction, dimension=2):
tag = "default"
def get_embeddings_tags(log_reader):
return get_logs(log_reader, "embeddings")
def get_embeddings(log_reader, run, tag, reduction, dimension=2):
log_reader.load_new_data()
records = log_reader.data_manager.get_reservoir("embeddings").get_items(
run, decode_tag(tag))
......
visualdl/writer/writer.py
浏览文件 @ 917c7204
......@@ -157,10 +157,11 @@ class LogWriter(object):
self._get_file_writer().add_record(
image(tag=tag, image_array=img, step=step, walltime=walltime))
def add_embeddings(self, labels, hot_vectors, walltime=None):
def add_embeddings(self, tag, labels, hot_vectors, walltime=None):
"""Add embeddings to vdl record file.
Args:
tag (string): Data identifier
labels (numpy.array or list): A list of labels.
hot_vectors (numpy.array or list): A matrix which each row is
feature of labels.
......@@ -179,7 +180,6 @@ class LogWriter(object):
writer.add_embedding(labels=labels, vectors=hot_vectors,
walltime=round(time.time()))
"""
tag = "default"
if '%' in tag:
raise RuntimeError("% can't appear in tag!")
if isinstance(hot_vectors, np.ndarray):
......
Markdown is supported
0% 或 .
You are about to add 0 people to the discussion. Proceed with caution.
先完成此消息的编辑！
想要评论请 注册 或