diff --git a/README.md b/README.md index f7363a67258b106457c97dff22f539361f176789..de3fc4ebd24dc494898e6f423e1d33eb4c3c921b 100644 --- a/README.md +++ b/README.md @@ -4,177 +4,79 @@ [![License](https://img.shields.io/badge/license-Apache%202-blue.svg)](LICENSE)

- +

-## Introduction -VisualDL is a deep learning visualization tool that can help design deep learning jobs. -It includes features such as scalar, parameter distribution, model structure and image visualization. -Currently it is being developed at a high pace. -New features will be continuously added. +## 介绍 +VisualDL是一个面向深度学习任务设计的可视化工具，利用丰富的图表展示数据，用户可以更直观、清晰地查看数据的特征与变化趋势，有助于分析数据、及时发现错误，进而改进神经网络模型的设计。 -At present, most DNN frameworks use Python as their primary language. VisualDL supports Python by nature. -Users can get plentiful visualization results by simply add a few lines of Python code into their model before training. +目前，VisualDL支持Scalar, Image, High Dimensional, Graph 这四个组件，项目正处于高速迭代中，新的组件会不断加入。 -Besides Python SDK, VisualDL was writen in C++ on the low level. It also provides C++ SDK that -can be integrated into other platforms. +由于大多数DNN平台均使用Python作为配置语言，VisualDL原生支持python的使用， +通过在模型的Python配置中添加几行，便可以为训练过程提供丰富的可视化支持。 -## Component -VisualDL provides following components: +## 组件 +VisualDL 目前支持以下组件： - scalar -- histogram - image -- audio -- graph - high dimensional +- graph ### Scalar -Scalar can be used to show the trends of error during training. +可以用于展示训练测试的指标变化趋势

- +

-### Histogram -Histogram can be used to visualize parameter distribution and trends for any tensor. - -

- -

### Image -Image can be used to visualize any tensor or intermediate generated image. +可以用于可视化任何图片，包括模型训练和预测得到的结果

- +

-### Audio -Audio can be used to play input audio samples or generated audio samples. - -### Graph -VisualDL graph supports displaying paddle model, furthermore is compatible with ONNX ([Open Neural Network Exchange](https://github.com/onnx/onnx)), -Cooperated with Python SDK, VisualDL can be compatible with most major DNN frameworks, including -PaddlePaddle, PyTorch and MXNet. - -

- -

- -To display the paddle model, all you have to do is: - -1. call the `fluid.io.save_inference_model()`interface to save paddle model -2. use `visualdl --model_pb [paddle_model_dir]` to load paddle model in command line - ### High Dimensional -High Dimensional can be used to visualize data embeddings by projecting high-dimensional data into 2D / 3D. +可以用于将高维度数据映射到2D/3D可实现可视化

- +

-## Quick Start -To give the VisualDL a quick test, please use the following commands. +## 安装 +### 使用 Anaconda 安装 -``` -# Install the VisualDL. Preferably under a virtual environment or anaconda. -pip install --upgrade visualdl +Anaconda是一个用于科学计算的Python发行版，提供了包管理与环境管理的功能，可以很方便地解决多版本python并存、切换以及各种第三方包安装问题。 -# run a demo, vdl_create_scratch_log will create logs for testing. -vdl_create_scratch_log -visualdl --logdir=scratch_log --port=8080 +请根据[Anaconda下载网站](https://www.anaconda.com/download) 的指示去下载和安装Anaconda. +下载Python 3.6版本的command-Line installer. -# visit http://127.0.0.1:8080 +创建conda环境名字为```vdl```或任何名字: ``` - -If you encounter the error `TypeError: __init__() got an unexpected keyword argument 'file'`, that is due to protobuf version is not 3.5+，simply run `pip install --upgrade protobuf` will fix the issue. - -If you run into any other issues in above steps, it could be error caused by environmental issues by different python or pip versions. -Following installation methods might fix the issues. - -## Install with Virtualenv - -[Virtualenv](https://virtualenv.pypa.io/en/stable/) creates isolated Python environment that prevents interfering -by other Python programs on the same machine and make sure Python and pip are located properly. - -On macOS, install pip and virtualenv by: -``` -sudo easy_install pip -pip install --upgrade virtualenv -``` - -On Linux, install pip and virtualenv by: -``` -sudo apt-get install python3-pip python3-dev python-virtualenv -``` - -Then create a Virtualenv environment by one of following command: +conda create -n vdl python=3.7 ``` -virtualenv ~/vdl # for Python2.7 -virtualenv -p python3 ~/vdl for Python 3.x -``` - -```~/vdl``` will be your Virtualenv directory, you may choose to install anywhere. - -Activate your Virtualenv environment by: -``` -source ~/vdl/bin/activate -``` - -Now you should be able to install VisualDL and run our demo: -``` -pip install --upgrade visualdl - -# run a demo, vdl_create_scratch_log will create logs for testing. -vdl_create_scratch_log -visualdl --logdir=scratch_log --port=8080 - -# visit http://127.0.0.1:8080 -``` - -If you still have issues installing VisualDL from Virtualenv, try following installation method. - - -## Install with Anaconda - -Anaconda is a python distribution, with installation and package management tools. Also it is an environment manager, -which provides the facility to create different python environments, each with their own settings. - -Follow the instructions on the [Anaconda download site](https://www.anaconda.com/download) to download and install Anaconda. -Download Python 3.6 version command-Line installer. - -Create a conda environment named ```vdl``` or anything you want by: -``` -conda create -n vdl pip python=2.7 # or python=3.3, etc. -``` - -Activate the conda environment by: +激活conda环境如下: ``` source activate vdl ``` -Now you should be able to install VisualDL and run our demo: +现在再安装 VisualDL 和运行范例： -``` +```shell pip install --upgrade visualdl - -# run a demo, vdl_create_scratch_log will create logs for testing. -vdl_create_scratch_log -visualdl --logdir=scratch_log --port=8080 - -# visit http://127.0.0.1:8080 ``` -If you still have issues installing VisualDL, try installing from sources as in following section. +也可以使用源代码安装方式，能够获取最新的VisualDL更新。 +### 使用代码安装 -### Install from source -``` -#Preferably under a virtualenv or anaconda. +建议在anaconda环境下进行安装。 +```shell git clone https://github.com/PaddlePaddle/VisualDL.git cd VisualDL @@ -182,94 +84,43 @@ python setup.py bdist_wheel pip install --upgrade dist/visualdl-*.whl ``` -If there are still issues regarding the ```pip install```, you can still start Visual DL by starting the dev server -[here](https://github.com/PaddlePaddle/VisualDL/blob/develop/docs/how_to_dev_frontend_en.md) - - ## SDK -VisualDL provides both Python SDK and C++ SDK in order to fit more use cases. - - -### Python SDK -VisualDL now supports both Python 2 and Python 3. -Below is an example of creating a simple Scalar component and inserting data from different timestamps: +以最简单的Scalar组件为例，尝试创建一个日志记录多个scalar的数据步骤如下： ```python -import random from visualdl import LogWriter -logdir = "./tmp" -logger = LogWriter(logdir, sync_cycle=10000) - -# mark the components with 'train' label. -with logger.mode("train"): - # create a scalar component called 'scalars/scalar0' - scalar0 = logger.scalar("scalars/scalar0") - -# add some records during DL model running. -for step in range(100): - scalar0.add_record(step, random.random()) +with LogWriter(logdir="./tmp") as writer: + for step in range(100): + writer.add_scalar(step=step, value=step*2) ``` -### C++ SDK -Here is the C++ SDK identical to the Python SDK example above: - -```c++ -#include -#include -#include "visualdl/logic/sdk.h" - -namespace vs = visualdl; -namespace cp = visualdl::components; -int main() { - const std::string dir = "./tmp"; - vs::LogWriter logger(dir, 10000); - - logger.SetMode("train"); - auto tablet = logger.AddTablet("scalars/scalar0"); - - cp::Scalar scalar0(tablet); - - for (int step = 0; step < 1000; step++) { - float v = (float)std::rand() / RAND_MAX; - scalar0.AddRecord(step, v); - } - - return 0; -} -``` - -## Launch Visual DL -After some logs have been generated during training, users can launch Visual DL application to see real-time data visualization by: -### Startup in command line +## 启动Board +当训练过程中已经产生了日志数据，就可以启动board进行实时预览可视化信息 +### 在命令行中启动 ``` visualdl --logdir ``` -visualDL also supports following optional parameters: +board 还支持一些参数来实现远程的访问： -- `--host` set IP -- `--port` set port -- `-m / --model_pb` specify ONNX format for model file to view graph - -### Startup in python script +- `--host` 设定IP +- `--port` 设定端口 +### 在Python脚本中启动 ```python >>> from visualdl.server import app >>> app.run(logdir="SOME_LOG_DIR") ``` -`app.run()` support all parameters for command line startup, in addition, you can also specify `pen_browser=True` to open browser automatically。 - -### Contribute +`app.run()`支持命令行启动的所有参数，除此之外，还可以通过指定`open_browser=True`，自动打开浏览器。 +## 贡献 -VisualDL is initially created by [PaddlePaddle](http://www.paddlepaddle.org/) and -[ECharts](http://echarts.baidu.com/). -We welcome everyone to use, comment and contribute to Visual DL :) +VisualDL 是由 [PaddlePaddle](http://www.paddlepaddle.org/) 和 [ECharts](http://echarts.baidu.com/) 合作推出的开源项目。我们欢迎所有人使用，提意见以及贡献代码。 -## More details +## 更多细节 -For more details about how to use VisualDL, please take a look at [documents](https://github.com/PaddlePaddle/VisualDL/tree/develop/demo) +想了解更多关于VisualDL的使用介绍，请查看[文档](./docs/README.md) diff --git a/VERSION_NUMBER b/VERSION_NUMBER index 6cf46a6fda02d90e4796027f254ebde34fae6d9b..e05cd4698086b25ce1d3e554e4d53498e8282e19 100644 --- a/VERSION_NUMBER +++ b/VERSION_NUMBER @@ -1 +1 @@ -2.0.0-alpha.0 +2.0.0-alpha.1 diff --git a/demo/components/high_dimensional_test.py b/demo/components/high_dimensional_test.py new file mode 100644 index 0000000000000000000000000000000000000000..5e0b069777fa28e073e8d348fdca858da612d8b6 --- /dev/null +++ b/demo/components/high_dimensional_test.py @@ -0,0 +1,31 @@ +# Copyright (c) 2020 VisualDL Authors. All Rights Reserve. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# ======================================================================= +# coding=utf-8 +from visualdl import LogWriter + + +if __name__ == '__main__': + hot_vectors = [ + [1.3561076367500755, 1.3116267195134017, 1.6785401875616097], + [1.1039614644440658, 1.8891609992484688, 1.32030488587171], + [1.9924524852447711, 1.9358920727142739, 1.2124401279391606], + [1.4129542689796446, 1.7372166387197474, 1.7317806077076527], + [1.3913371800587777, 1.4684674577930312, 1.5214136352476377]] + + labels = ["label_1", "label_2", "label_3", "label_4", "label_5"] + with LogWriter(logdir="./log/high_dimensional_test/train") as writer: + writer.add_embeddings(tag='default', + labels=labels, + hot_vectors=hot_vectors) diff --git a/demo/components/image_test.py b/demo/components/image_test.py new file mode 100644 index 0000000000000000000000000000000000000000..80a16c9b6ce19e18a196b25d107db42fb6f9ca11 --- /dev/null +++ b/demo/components/image_test.py @@ -0,0 +1,37 @@ +# Copyright (c) 2020 VisualDL Authors. All Rights Reserve. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# ======================================================================= +# coding=utf-8 +import numpy as np +from PIL import Image +from visualdl import LogWriter + + +def random_crop(img): + """Get random block of img, which size is 100x100. + """ + img = Image.open(img) + w, h = img.size + random_w = np.random.randint(0, w - 100) + random_h = np.random.randint(0, h - 100) + r = img.crop((random_w, random_h, random_w + 100, random_h + 100)) + return np.asarray(r) + + +if __name__ == '__main__': + with LogWriter(logdir="./log/image_test/train") as writer: + for step in range(6): + writer.add_image(tag="doge", + img=random_crop("../../docs/images/dog.jpg"), + step=step) diff --git a/demo/components/scalar_test.py b/demo/components/scalar_test.py new file mode 100644 index 0000000000000000000000000000000000000000..e03a1dc68cbbaabca18999d4f1aa108923bb8af4 --- /dev/null +++ b/demo/components/scalar_test.py @@ -0,0 +1,23 @@ +# Copyright (c) 2020 VisualDL Authors. All Rights Reserve. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# ======================================================================= +from visualdl import LogWriter + +if __name__ == '__main__': + value = [i/1000.0 for i in range(1000)] + with LogWriter(logdir="./log/scalar_test/train") as writer: + for step in range(1000): + writer.add_scalar(tag="acc", step=step, value=value[step]) + writer.add_scalar(tag="loss", step=step, value=1/(value[step] + 1)) + diff --git a/docs/README.md b/docs/README.md new file mode 100644 index 0000000000000000000000000000000000000000..621e73f96f48ca87ece09f901534e91cacd39223 --- /dev/null +++ b/docs/README.md @@ -0,0 +1,121 @@ +

+ +

+ +## 介绍 +VisualDL是一个面向深度学习任务设计的可视化工具，利用丰富的图表展示数据，用户可以更直观、清晰地查看数据的特征与变化趋势，有助于分析数据、及时发现错误，进而改进神经网络模型的设计。 + +目前大多数DNN平台均使用Python作为配置语言，VisualDL原生支持python的使用， +通过在模型的Python配置中添加几行，便可以为训练过程提供丰富的可视化支持。 + +VisualDL拥有不同种类的组件，能够帮助开发者有效记录和监测训练中的各种指标和数据，为模型训练数据参考，及时发现错误，进而改进神经网络模型的设计，关于组件的具体信息可以参考[VisualDL组件](./components/README.md)。 + +## 特点 +### 1 动态展示 +在启动VisualDL Board后，LogReader将不断增量的读取日志中数据并供前端调用展示，因此能够在训练中同步观测指标变化，如下图： + +

+ +

+ +### 2 多实验对比 +VisualDL支持多实验对比，只需要在启动VisualDL Board的时候将每个实验日志所在路径同时传入即可，每个实验中相同tag的指标将绘制在一张图中同步呈现，如下图： + +

+ +

+ +## 安装 +建议在anaconda环境下进行安装，便于开发环境的隔离，pip安装命令为： +```shell +python -m pip install visualdl +``` +需要注意，由于官方不再维护Python2，因此VisualDL在v2.0.0之后的版本仅支持Python3 + +## 使用 +### 记录日志 +VisualDL的后端提供了Python SDK，可通过LogWriter定制一个日志记录器，接口如下： +```python +class LogWriter( + logdir=None, + comment='', + max_queue=10, + flush_secs=120, + filename_suffix='', + write_to_disk=True, + **kwargs + ) +``` +接口参数说明如下： +|参数|格式|含义| +|-|-|-| +|logdir|string|日志文件所在的路径，VisualDL将在此路径下建立日志文件并进行记录，如果不填则默认为`runs/${CURRENT_TIME}`| +|comment|string|为日志文件夹名添加后缀，如果制定了logdir则此项无效| +|max_queue|int|日志记录消息队列的最大容量，达到此容量则立即写入到日志文件| +|flush_secs|int|日志记录消息队列的最大缓存时间，达到此时间则立即写入到日志文件| +|filename_suffix|string|为默认的日志文件名添加后缀| +|write_to_disk|boolean|是否写入到磁盘| + +初始化并记录标量数据到日志文件的过程如下： +```python +from visualdl import LogWriter + +# 在`./log/scalar_test/train`路径下建立日志文件 +with LogWriter(logdir="./log/scalar_test/train") as writer: + # 使用scalar组件记录一个标量数据 + writer.add_scalar(tag="acc", step=1, value=0.5678) + writer.add_scalar(tag="acc", step=2, value=0.6878) + writer.add_scalar(tag="acc", step=3, value=0.9878) +``` +关于LogWriter组件的更多具体用法，请参考[VisualDL组件](./components/README.md) + + +### 启动面板 +我们已经记录了三个标量数据到日志中，现在可以启动VisualDL面板查看日志的可视化结果，启动方式分为两种： +#### 在命令行启动 +我们可以使用命令行启动VisualDL面板，命令格式为： +```shell +visualdl --logdir --host --port +``` +参数意义分别为： +- `--logdir` 设定日志所在目录，可以指定多个目录，VisualDL将遍历并且迭代寻找指定目录的子目录，将所有实验结果进行可视化 +- `--host` 设定IP，默认为`127.0.0.1` +- `--port` 设定端口，默认为`8040` + +针对上一步生成的日志，我们的启动命令为 +```shell +visualdl --logdir ./log +``` + +#### 在Python脚本中启动 +我们还可以在Python脚本中启动VisualDL面板，接口如下： + +```python +visualdl.server.app.run(logdir, + host="127.0.0.1", + port=8080, + cache_timeout=20, + language=None, + open_browser=False) +``` +接口参数说明如下： +|参数|格式|含义| +|-|-|-| +|logdir|string或list[string_1, string_2, ... , string_n]|日志文件所在的路径，VisualDL将在此路径下递归搜索日志文件并进行可视化，可指定单个或多个路径| +|host|string|指定启动服务的ip，默认为`127.0.0.1`| +|port|int|启动服务端口，默认为`8040`| +|cache_timeout|int|后端缓存时间，在缓存时间内前端多次请求同一url，返回的数据从缓存中获取，默认为20秒| +|language|string|VisualDL面板语言，可指定为'EN'或'CN'，默认自动匹配操作系统使用语言| +|open_browser|boolean|是否打开浏览器，设置为True则在启动后自动打开浏览器并访问VisualDL面板| + +针对上一步生成的日志，我们的启动脚本为： +```python +from visualdl.server import app + +app.run(logdir="./log") +``` + +在使用任意一种方式启动VisualDL面板后，我们可以打开浏览器访问VisualDL面板，在面板上就可以看到日志的可视化结果，如图： +

+ +

diff --git a/docs/components/README.md b/docs/components/README.md new file mode 100644 index 0000000000000000000000000000000000000000..07ba46a13927bc4dec47f355d5574e86d8e5d8a6 --- /dev/null +++ b/docs/components/README.md @@ -0,0 +1,176 @@ +## 介绍 +VisualDL拥有不同种类的组件，能够帮助开发者有效记录和监测训练中的各种指标和数据，为模型训练数据参考，及时发现错误，进而改进神经网络模型的设计 + +使用组件之前必须先使用LogWriter创建一个记录器，如下： +```python +from visualdl import LogWriter + +with LogWriter(logdir="./log") as writer: +``` +关于记录器的更多参数配置，请参考[VisualDL说明文档](../README.md) + +## Scalar -- 折线图组件 + +### 介绍 + +Scalar 组件的输入数据类型为标量，该组件的作用是画折线图。将损失函数值、准确率等标量数据作为参数传入 scalar 组件，即可画出折线图，便于观察变化趋势。 + +### 记录接口 + +Scalar 组件的记录接口如下： + +```python +add_scalar(tag, value, step, walltime=None) +``` +接口参数说明如下： +|参数|格式|含义| +|-|-|-| +|tag|string|记录指标的标志，如`train/loss`，不能含有`%`| +|value|float|要记录的数据值| +|step|int|记录的步数| +|walltime|int|记录数据的时间戳，默认为当前时间戳| + +### demo +下面展示了使用 Scalar 组件记录数据的示例，代码见[Scalar组件](../../demo/components/scalar_test.py) +```python +from visualdl import LogWriter + +if __name__ == '__main__': + value = [i/1000.0 for i in range(1000)] + # 初始化一个记录器 + with LogWriter(logdir="./log/scalar_test/train") as writer: + for step in range(1000): + # 向记录器添加一个tag为`acc`的数据 + writer.add_scalar(tag="acc", step=step, value=value[step]) + # 向记录器添加一个tag为`loss`的数据 + writer.add_scalar(tag="loss", step=step, value=1/(value[step] + 1)) +``` +运行上述程序后，在命令行执行 +```shell +visualdl --logdir ./log --port 8080 +``` + +接着在浏览器打开`http://127.0.0.1:8080`，即可查看以下折线图。 + +

+ +

+ + +## Image -- 图片可视化组件 + +### 介绍 + +Image 组件用于显示图片。在程序运行过程中，将图片数据传入 Image 组件，就可在 VisualDL 的前端网页看到相应图片。 + +### 记录接口 + +Image 组件的记录接口如下： + +```python +add_image(tag, img, step, walltime=None) +``` +接口参数说明如下： +|参数|格式|含义| +|-|-|-| +|tag|string|记录指标的标志，如`train/loss`，不能含有`%`| +|img|numpy.ndarray|以ndarray格式表示的图片| +|step|int|记录的步数| +|walltime|int|记录数据的时间戳，默认为当前时间戳| + +### demo +下面展示了使用 Image 组件记录数据的示例，代码见[Image组件](../../demo/components/image_test.py) +```python +import numpy as np +from PIL import Image +from visualdl import LogWriter + + +def random_crop(img): + """获取图片的随机 100x100 分片 + """ + img = Image.open(img) + w, h = img.size + random_w = np.random.randint(0, w - 100) + random_h = np.random.randint(0, h - 100) + r = img.crop((random_w, random_h, random_w + 100, random_h + 100)) + return np.asarray(r) + + +if __name__ == '__main__': + # 初始化一个记录器 + with LogWriter(logdir="./log/image_test/train") as writer: + for step in range(6): + # 添加一个图片数据 + writer.add_image(tag="doge", + img=random_crop("../../docs/images/dog.jpg"), + step=step) +``` +运行上述程序后，在命令行执行 +```shell +visualdl --logdir ./log --port 8080 +``` + +接着在浏览器打开`http://127.0.0.1:8080`，即可拖动查看记录的不同图片。 + +

+ +

+ + +## High Dimensional -- 数据降维组件 + +### 介绍 + +High Dimensional 组件的作用就是将数据映射到 2D/3D 空间来做可视化嵌入，这有利于了解不同数据的相关性。High Dimensional 组件支持以下两种降维算法： + + - PCA : Principle Component Analysis 主成分分析 + - t-SNE : t-distributed stochastic neighbor embedding t-分布式随机领域嵌入 + +### 记录接口 + +High Dimensional 组件的记录接口如下： + +```python +add_embeddings(tag, labels, hot_vectors, walltime=None) +``` +接口参数说明如下： +|参数|格式|含义| +|-|-|-| +|tag|string|记录指标的标志，如`default`，不能含有`%`| +|labels|numpy.array 或 list|一维数组表示的标签，每个元素是一个string类型的字符串| +|hot_vectors|numpy.array or list|与labels一一对应，每个元素可以看作是某个标签的特征| +|walltime|int|记录数据的时间戳，默认为当前时间戳| + +### demo +下面展示了使用 High Dimensional 组件记录数据的示例，代码见[High Dimensional组件](../../demo/components/high_dimensional_test.py) +```python +from visualdl import LogWriter + + +if __name__ == '__main__': + hot_vectors = [ + [1.3561076367500755, 1.3116267195134017, 1.6785401875616097], + [1.1039614644440658, 1.8891609992484688, 1.32030488587171], + [1.9924524852447711, 1.9358920727142739, 1.2124401279391606], + [1.4129542689796446, 1.7372166387197474, 1.7317806077076527], + [1.3913371800587777, 1.4684674577930312, 1.5214136352476377]] + + labels = ["label_1", "label_2", "label_3", "label_4", "label_5"] + # 初始化一个记录器 + with LogWriter(logdir="./log/high_dimensional_test/train") as writer: + # 将一组labels和对应的hot_vectors传入记录器进行记录 + writer.add_embeddings(tag='default', + labels=labels, + hot_vectors=hot_vectors) +``` +运行上述程序后，在命令行执行 +```shell +visualdl --logdir ./log --port 8080 +``` + +接着在浏览器打开`http://127.0.0.1:8080`，即可查看降维后的可视化数据。 + +

+ +

diff --git a/docs/images/3points_demo.png b/docs/images/3points_demo.png new file mode 100644 index 0000000000000000000000000000000000000000..1f787f56916df79bf8df7519c3659658c60ff307 Binary files /dev/null and b/docs/images/3points_demo.png differ diff --git a/visualdl/python/dog.jpg b/docs/images/dog.jpg similarity index 100% rename from visualdl/python/dog.jpg rename to docs/images/dog.jpg diff --git a/docs/images/dynamic_display.gif b/docs/images/dynamic_display.gif new file mode 100644 index 0000000000000000000000000000000000000000..84ed18c693d19004e09560fe79a171adc23029f5 Binary files /dev/null and b/docs/images/dynamic_display.gif differ diff --git a/docs/images/dynamic_high_dimensional.gif b/docs/images/dynamic_high_dimensional.gif new file mode 100644 index 0000000000000000000000000000000000000000..6bb8dc5e981bece4023a0247cae310be25b1932c Binary files /dev/null and b/docs/images/dynamic_high_dimensional.gif differ diff --git a/docs/images/dynamic_image.gif b/docs/images/dynamic_image.gif new file mode 100644 index 0000000000000000000000000000000000000000..554950f7d6c11484204f61e7c411ccf91231d00a Binary files /dev/null and b/docs/images/dynamic_image.gif differ diff --git a/docs/images/high_dimensional_test.png b/docs/images/high_dimensional_test.png new file mode 100644 index 0000000000000000000000000000000000000000..3aec3e2afad29dda7bfb319cc726da4c5431587e Binary files /dev/null and b/docs/images/high_dimensional_test.png differ diff --git a/docs/images/image_test.png b/docs/images/image_test.png new file mode 100644 index 0000000000000000000000000000000000000000..56968c8c62d4c67550ca86388f69de2870fcb976 Binary files /dev/null and b/docs/images/image_test.png differ diff --git a/docs/images/multi_experiments.gif b/docs/images/multi_experiments.gif new file mode 100644 index 0000000000000000000000000000000000000000..3047f0333b533222c5642fabe99ef9152d6a7b62 Binary files /dev/null and b/docs/images/multi_experiments.gif differ diff --git a/docs/images/scalar_test.png b/docs/images/scalar_test.png new file mode 100644 index 0000000000000000000000000000000000000000..42a8e5f3cf83b62a511ca334759520e2673b94e7 Binary files /dev/null and b/docs/images/scalar_test.png differ diff --git a/setup.py b/setup.py index fd5713ac340504f66db0563eac32cb2bab9ee592..eb72b6142b6b816b9bd1bdccb490947ac77ecd43 100644 --- a/setup.py +++ b/setup.py @@ -101,6 +101,7 @@ setup( license=LICENSE, keywords="visualization deeplearning", long_description=read('README.md'), + long_description_content_type='text/markdown', install_requires=REQUIRED_PACKAGES, package_data={ 'visualdl.server': [('dist' + ('/*' * n)) for n in range(1, 20)], diff --git a/visualdl/io/bfile.py b/visualdl/io/bfile.py index 3b76b6a5df3fdcd93176de860e7523c917960534..d074492cd0cb4a79cfeea3d9596c1ddc3608e42b 100644 --- a/visualdl/io/bfile.py +++ b/visualdl/io/bfile.py @@ -112,7 +112,10 @@ class BFile(object): return self def __exit__(self, exc_type, exc_val, exc_tb): - pass + self.close() + self.buff = None + self.buff_offset = 0 + self.continuation_token = None def __iter__(self): return self diff --git a/visualdl/reader/reader.py b/visualdl/reader/reader.py index 543df91630df503be751bdc0dfc8f77213700b9c..85d4f0d93d05f60560bf050e44ede703dbf367a6 100644 --- a/visualdl/reader/reader.py +++ b/visualdl/reader/reader.py @@ -45,7 +45,11 @@ class LogReader(object): Args: logdir: The dir include vdl log files, multiple subfolders allowed. """ - self.dir = logdir + if isinstance(logdir, str): + self.dir = [logdir] + else: + self.dir = logdir + self.reader = None self.readers = {} self.walks = None @@ -87,8 +91,9 @@ class LogReader(object): def get_all_walk(self): self.walks = {} - for root, dirs, files in bfile.walk(self.dir): - self.walks.update({root: files}) + for dir in self.dir: + for root, dirs, files in bfile.walk(dir): + self.walks.update({root: files}) def components_listing(self): """Get available component types. @@ -195,6 +200,8 @@ class LogReader(object): return self._tags def components(self): + """Get components type used by vdl. + """ return list(set(self._tags.values())) def load_new_data(self, update=True): diff --git a/visualdl/server/app.py b/visualdl/server/app.py index 5ba6b2e1e28a994d4029bc7614755aa4ba23ca07..e80075dc8cd40335ec2219fd494e0f6cc50ac8b5 100644 --- a/visualdl/server/app.py +++ b/visualdl/server/app.py @@ -105,6 +105,7 @@ def parse_args(): required=True, action="store", dest="logdir", + nargs="+", help="log file directory") parser.add_argument( "--cache_timeout", @@ -148,11 +149,8 @@ def create_app(args): app.config['BABEL_DEFAULT_LOCALE'] = default_language babel = Babel(app) - log_reader = LogReader(args.logdir) - # mannully put graph's image on this path also works. - graph_image_path = os.path.join(args.logdir, 'graph.jpg') # use a memory cache to reduce disk reading frequency. CACHE = MemCache(timeout=args.cache_timeout) cache_get = lib.cache_get(CACHE) @@ -237,6 +235,13 @@ def create_app(args): result = gen_result(0, "", data) return Response(json.dumps(result), mimetype='application/json') + @app.route("/api/embeddings/tags") + def embeddings_tags(): + data = cache_get("/data/plugin/embeddings/tags", try_call, + lib.get_embeddings_tags, log_reader) + result = gen_result(0, "", data) + return Response(json.dumps(result), mimetype='application/json') + @app.route('/api/scalars/list') def scalars(): run = request.args.get('run') @@ -273,12 +278,13 @@ def create_app(args): @app.route('/api/embeddings/embedding') def embeddings(): run = request.args.get('run') + tag = request.args.get('tag') dimension = request.args.get('dimension') reduction = request.args.get('reduction') key = os.path.join('/data/plugin/embeddings/embeddings', run, dimension, reduction) data = cache_get(key, try_call, lib.get_embeddings, log_reader, run, - reduction, int(dimension)) + tag, reduction, int(dimension)) result = gen_result(0, "", data) return Response(json.dumps(result), mimetype='application/json') @@ -347,7 +353,7 @@ def _run(logdir, def run(logdir, host="127.0.0.1", - port=8080, + port=8040, model_pb="", cache_timeout=20, language=None, @@ -371,7 +377,7 @@ def main(): args = parse_args() logger.info(" port=" + str(args.port)) app = create_app(args=args) - app.run(debug=False, host=args.host, port=args.port, threaded=True) + app.run(debug=False, host=args.host, port=args.port, threaded=False) if __name__ == "__main__": diff --git a/visualdl/server/lib.py b/visualdl/server/lib.py index 5a8ef05947ef746305fea2b1bf18a71801f039c8..ce1141b74b00c4dca41eb466ee2717e89b0105f4 100644 --- a/visualdl/server/lib.py +++ b/visualdl/server/lib.py @@ -104,8 +104,11 @@ def get_individual_audio(log_reader, run, tag, step_index): return result -def get_embeddings(log_reader, run, reduction, dimension=2): - tag = "default" +def get_embeddings_tags(log_reader): + return get_logs(log_reader, "embeddings") + + +def get_embeddings(log_reader, run, tag, reduction, dimension=2): log_reader.load_new_data() records = log_reader.data_manager.get_reservoir("embeddings").get_items( run, decode_tag(tag)) diff --git a/visualdl/writer/writer.py b/visualdl/writer/writer.py index 0c9c87c6eb42c3ee052add0f65396afb8d37f788..d05e3a6cd78a65341b2b7770b45b9f899163c92f 100644 --- a/visualdl/writer/writer.py +++ b/visualdl/writer/writer.py @@ -157,10 +157,11 @@ class LogWriter(object): self._get_file_writer().add_record( image(tag=tag, image_array=img, step=step, walltime=walltime)) - def add_embeddings(self, labels, hot_vectors, walltime=None): + def add_embeddings(self, tag, labels, hot_vectors, walltime=None): """Add embeddings to vdl record file. Args: + tag (string): Data identifier labels (numpy.array or list): A list of labels. hot_vectors (numpy.array or list): A matrix which each row is feature of labels. @@ -179,7 +180,6 @@ class LogWriter(object): writer.add_embedding(labels=labels, vectors=hot_vectors, walltime=round(time.time())) """ - tag = "default" if '%' in tag: raise RuntimeError("% can't appear in tag!") if isinstance(hot_vectors, np.ndarray):