未验证 提交 6fdfcdea 编写于 作者: C chenjian 提交者: GitHub

Upgrade version and docs

上级 15dbace6
...@@ -315,10 +315,18 @@ Developers can compare multiple experiments by specifying and uploading the path ...@@ -315,10 +315,18 @@ Developers can compare multiple experiments by specifying and uploading the path
### Graph ### Graph
**Graph** enables developers to visualize model structures by only one click. Moreover, **Graph** allows developers to explore model attributes, node information, node input and output. aiding them analyze model structures quickly and understand the direction of data flow easily. **Graph** enables developers to visualize model structures by only one click. Moreover, **Graph** allows developers to explore model attributes, node information, node input and output. aiding them analyze model structures quickly and understand the direction of data flow easily. Additionally, Graph supports the visualization of dynamic and static model graph respectively.
- dynamic graph
<p align="center">
<img src="https://user-images.githubusercontent.com/22424850/175811841-64b44d99-7d48-4fe9-a679-01156d15af74.gif" width="85%"/>
</p>
- static graph
<p align="center"> <p align="center">
<img src="https://user-images.githubusercontent.com/48054808/90869866-6aecee00-e3cb-11ea-8211-b8af070239e6.png" width="85%"/> <img src="https://user-images.githubusercontent.com/22424850/175811795-1fd21737-06f0-42fc-bea3-ef7a17216fc9.gif" width="85%"/>
</p> </p>
......
...@@ -337,12 +337,18 @@ value: 3.1297709941864014 ...@@ -337,12 +337,18 @@ value: 3.1297709941864014
### Graph ### Graph
一键可视化模型的网络结构。可查看模型属性、节点信息、节点输入输出等,并支持节点搜索,辅助用户快速分析模型结构与了解数据流向。 一键可视化模型的网络结构。可查看模型属性、节点信息、节点输入输出等,并支持节点搜索,辅助用户快速分析模型结构与了解数据流向,覆盖动态图与静态图两种格式。
- 动态图
<p align="center"> <p align="center">
<img src="https://user-images.githubusercontent.com/48054808/84483052-5acdd980-accb-11ea-8519-1608da7ee698.png" width="85%"/> <img src="https://user-images.githubusercontent.com/22424850/175770313-2509f7e9-041a-4654-9a0f-45f4bd76e1e8.gif" width="85%"/>
</p> </p>
- 静态图
<p align="center">
<img src="https://user-images.githubusercontent.com/22424850/175770315-11e2c3f5-141c-4f05-be86-0e1e2785e11f.gif" width="85%"/>
</p>
### Histogram ### Histogram
......
...@@ -239,12 +239,21 @@ Developers can compare with multiple experiments by specifying and uploading the ...@@ -239,12 +239,21 @@ Developers can compare with multiple experiments by specifying and uploading the
### Graph ### Graph
**Graph** enables developers to visualize model structures by only one click. Moreover, **Graph** allows Developers to explore model attributes, node information, node input and output. aiding them analyze model structure quickly and understand the direction of data flow easily. **Graph** enables developers to visualize model structures by only one click. Moreover, **Graph** allows developers to explore model attributes, node information, node input and output. aiding them analyze model structures quickly and understand the direction of data flow easily. Additionally, Graph supports the visualization of dynamic and static model graph respectively.
- dynamic graph
<p align="center">
<img src="https://user-images.githubusercontent.com/22424850/175811841-64b44d99-7d48-4fe9-a679-01156d15af74.gif" width="85%"/>
</p>
- static graph
<p align="center"> <p align="center">
<img src="https://user-images.githubusercontent.com/48054808/90869866-6aecee00-e3cb-11ea-8211-b8af070239e6.png" width="85%"/> <img src="https://user-images.githubusercontent.com/22424850/175811795-1fd21737-06f0-42fc-bea3-ef7a17216fc9.gif" width="85%"/>
</p> </p>
### Histogram ### Histogram
**Histogram** displays how the trend of tensors (weight, bias, gradient, etc.) changes during the training process in the form of histogram. Developers can adjust the model structures accurately by having an in-depth understanding of the effect of each layer. **Histogram** displays how the trend of tensors (weight, bias, gradient, etc.) changes during the training process in the form of histogram. Developers can adjust the model structures accurately by having an in-depth understanding of the effect of each layer.
......
...@@ -250,9 +250,17 @@ app.run(logdir="./log") ...@@ -250,9 +250,17 @@ app.run(logdir="./log")
### Graph ### Graph
一键可视化模型的网络结构。可查看模型属性、节点信息、节点输入输出等,并支持节点搜索,辅助用户快速分析模型结构与了解数据流向。 一键可视化模型的网络结构。可查看模型属性、节点信息、节点输入输出等,并支持节点搜索,辅助用户快速分析模型结构与了解数据流向,覆盖动态图与静态图两种格式。
- 动态图
<p align="center">
<img src="https://user-images.githubusercontent.com/22424850/175770313-2509f7e9-041a-4654-9a0f-45f4bd76e1e8.gif" width="85%"/>
</p>
- 静态图
<p align="center"> <p align="center">
<img src="https://user-images.githubusercontent.com/48054808/84483052-5acdd980-accb-11ea-8519-1608da7ee698.png" width="85%"/> <img src="https://user-images.githubusercontent.com/22424850/175770315-11e2c3f5-141c-4f05-be86-0e1e2785e11f.gif" width="85%"/>
</p> </p>
### Histogram ### Histogram
......
...@@ -454,44 +454,114 @@ Then, open the browser and enter the address`http://127.0.0.1:8080` to view: ...@@ -454,44 +454,114 @@ Then, open the browser and enter the address`http://127.0.0.1:8080` to view:
Graph can visualize the network structure of the model by one click. It enables developers to view the model attributes, node information, searching node and so on. These functions help developers analyze model structures and understand the directions of data flow quickly. Graph can visualize the network structure of the model by one click. It enables developers to view the model attributes, node information, searching node and so on. These functions help developers analyze model structures and understand the directions of data flow quickly.
### Record Interface
The interface of the Graph is shown as follows:
```python
add_graph(model, input_spec, verbose=False):
```
The interface parameters are described as follows:
| parameter | format | meaning |
| -------------- | --------------------- | ------------------------------------------- |
| model | paddle.nn.Layer | Dynamic model of paddle |
| input_spec | list\[paddle.static.InputSpec\|Tensor\] | Describes the input of the saved model's [forward arguments](https://www.paddlepaddle.org.cn/documentation/docs/zh/api/paddle/static/InputSpec_cn.html) |
| verbose | bool | Whether to print graph statistic information in console. |
**Note**
If you want to use add_graph interface, paddle package is required. Please refer to website of [PaddlePaddle](https://www.paddlepaddle.org.cn/install/quick?docurl=/documentation/docs/en/install/pip/linux-pip_en.html)
### Demo ### Demo
The following shows an example of how to use Graph component, and script can be found in [Graph Demo](https://github.com/PaddlePaddle/VisualDL/blob/develop/demo/components/graph_test.py)
There are two methods to launch this component: There are two methods to launch this component:
- By the front end: ```python
import paddle
import paddle.nn as nn
import paddle.nn.functional as F
- If developers only need to use Graph, developers can launch VisualDL (Graph) by executing `visualdl`on the command line. from visualdl import LogWriter
- If developers need to use Graph and other functions at the same time, they need to specify the log file path (using `./log` as an example):
```shell
visualdl --logdir ./log --port 8080
```
class MyNet(nn.Layer):
def __init__(self):
super(MyNet, self).__init__()
self.conv1 = nn.Conv2D(
in_channels=1, out_channels=20, kernel_size=5, stride=1, padding=2)
self.max_pool1 = nn.MaxPool2D(kernel_size=2, stride=2)
self.conv2 = nn.Conv2D(
in_channels=20,
out_channels=20,
kernel_size=5,
stride=1,
padding=2)
self.max_pool2 = nn.MaxPool2D(kernel_size=2, stride=2)
self.fc = nn.Linear(in_features=980, out_features=10)
def forward(self, inputs):
x = self.conv1(inputs)
x = F.relu(x)
x = self.max_pool1(x)
x = self.conv2(x)
x = F.relu(x)
x = self.max_pool2(x)
x = paddle.reshape(x, [x.shape[0], -1])
x = self.fc(x)
return x
net = MyNet()
with LogWriter(logdir="./log/graph_test/") as writer:
writer.add_graph(
model=net,
input_spec=[paddle.static.InputSpec([-1, 1, 28, 28], 'float32')],
verbose=True)
```
- By the backend:
- Add the parameter `--model` and specify the **model file** path (not the folder path) to launch the panel:
```shell After running the above program, developers can launch the panel by:
visualdl --model ./log/model --port 8080
```
```shell
visualdl --logdir ./log/graph_test/ --port 8080
```
After the launch, developers can view the network structure: Then, open the browser and enter the address`http://127.0.0.1:8080` to view:
<p align="center"> <p align="center">
<img src="https://user-images.githubusercontent.com/48054808/90877274-6548d580-e3d6-11ea-9804-74a1ead47b30.png" width="80%"/> <img src="https://user-images.githubusercontent.com/22424850/175811156-a80ca0c4-207d-44d7-bd5a-9701a7875722.gif" width="80%"/>
</p> </p>
**Note**
We provide option --model to specify model structure file in previous versions, and this option is still supported now. You can specify model exported by `add_graph` interface ("vdlgraph" contained in filename), which will be shown in dynamic graph page, and we use string "manual_input_model" in the page to denote the model you specify by this option. Other supported file formats are presented in static graph page.
For example
```shell
visualdl --model ./log/model.pdmodel --port 8080
```
which will be shown in static graph page. And
```shell
visualdl --model ./log/vdlgraph.1655783158.log --port 8080
```
shown in dynamic graph page.
### Functional Instructions ### Functional Instructions
- Upload the model file by one-click Graph page is divided into dynamic and static version currently. Dynamic version is used to visualize dynamic model of paddle, which is exported by add_graph interface.
- Supported model:PaddlePaddle、ONNX、Keras、Core ML、Caffe、Caffe2、Darknet、MXNet、ncnn、TensorFlow Lite The other is used to visualize static model of paddle, which is exported by [paddle.jit.save](https://www.paddlepaddle.org.cn/documentation/docs/en/api/paddle/jit/save_en.html) interface and other supported formats.
- Experimental supported model:TorchScript、PyTorch、Torch、 ArmNN、BigDL、Chainer、CNTK、Deeplearning4j、MediaPipe、ML.NET、MNN、OpenVINO、Scikit-learn、Tengine、TensorFlow.js、TensorFlow
<p align="center"> <p align="center">
<img src="https://user-images.githubusercontent.com/48054808/90877449-a80aad80-e3d6-11ea-8016-0a2f3afe6f5e.png" width="80%"/> <img src="https://user-images.githubusercontent.com/22424850/175810574-d3526ef5-859f-4ea9-b705-f55bfc8ed5af.png" width="80%"/>
</p> </p>
**Common functions**
- Developers are allowed to drag the model up and down,left and right,zoom in and zoom out. - Developers are allowed to drag the model up and down,left and right,zoom in and zoom out.
<p align="center"> <p align="center">
...@@ -534,6 +604,44 @@ After the launch, developers can view the network structure: ...@@ -534,6 +604,44 @@ After the launch, developers can view the network structure:
<img src="https://user-images.githubusercontent.com/48054808/90879247-34b66b00-e3d9-11ea-94ef-a26b1ba07dd0.png" width="25%"/> <img src="https://user-images.githubusercontent.com/48054808/90879247-34b66b00-e3d9-11ea-94ef-a26b1ba07dd0.png" width="25%"/>
</p> </p>
**Specific feature in dynamic version**
- Fold and unfold one node
<p align="center">
<img src="https://user-images.githubusercontent.com/22424850/175810800-4823b9f1-3d59-44e8-aaa5-a80577624452.png" width="80%"/>
</p>
<p align="center">
<img src="https://user-images.githubusercontent.com/22424850/175810790-a35f83bf-a23c-4a28-afb7-2e0cf7711b9c.png" width="80%"/>
</p>
- Fold and unfold all nodes
<p align="center">
<img src="https://user-images.githubusercontent.com/22424850/175810856-ff98a1ed-2a4f-4cc1-bc9b-3085857c0b81.png" width="80%"/>
</p>
<p align="center">
<img src="https://user-images.githubusercontent.com/22424850/175810837-a0953956-7320-4e78-9c52-72ad13962216.png" width="80%"/>
</p>
- Link api specification of paddle
If you use paddle.nn components to construct your network model, you can use alt+click mouse to direct to corresponding api specification.
<p align="center">
<img src="https://user-images.githubusercontent.com/22424850/175810992-b86e9aef-e700-4c2d-bcd0-21fc96fc2564.png" width="80%"/>
</p>
<p align="center">
<img src="https://user-images.githubusercontent.com/22424850/175810997-0672d836-4d7c-432d-b5de-187f97c421ae.png" width="80%"/>
</p>
**Specific feature in static version**
- Upload the model file by one-click
- Supported model:PaddlePaddle、ONNX、Keras、Core ML、Caffe、Caffe2、Darknet、MXNet、ncnn、TensorFlow Lite
- Experimental supported model:TorchScript、PyTorch、Torch、 ArmNN、BigDL、Chainer、CNTK、Deeplearning4j、MediaPipe、ML.NET、MNN、OpenVINO、Scikit-learn、Tengine、TensorFlow.js、TensorFlow
<p align="center">
<img src="https://user-images.githubusercontent.com/48054808/90877449-a80aad80-e3d6-11ea-8016-0a2f3afe6f5e.png" width="80%"/>
</p>
## Histogram--Distribution of Tensors ## Histogram--Distribution of Tensors
### Introduction ### Introduction
......
...@@ -511,49 +511,119 @@ visualdl --logdir ./log --port 8080 ...@@ -511,49 +511,119 @@ visualdl --logdir ./log --port 8080
## Graph--网络结构组件 ## Graph--网络结构组件
### 介绍 ### 介绍
Graph组件一键可视化模型的网络结构。用于查看模型属性、节点信息、节点输入输出等,并进行节点搜索,协助开发者们快速分析模型结构与了解数据流向。 Graph组件一键可视化模型的网络结构。用于查看模型属性、节点信息、节点输入输出等,并进行节点搜索,协助开发者们快速分析模型结构与了解数据流向,覆盖动态图与静态图两种格式。
### 记录接口
Graph组件的记录接口如下:
```python
add_graph(model, input_spec, verbose=False):
```
接口参数说明如下:
| 参数 | 格式 | 含义 |
| -------------- | --------------------- | ------------------------------------------- |
| model | paddle.nn.Layer | Paddle的动态图模型 |
| input_spec | list\[paddle.static.InputSpec\|Tensor\] | 用于描述模型[输入的参数](https://www.paddlepaddle.org.cn/documentation/docs/zh/api/paddle/static/InputSpec_cn.html) |
| verbose | bool | 是否在终端打印模型的节点统计信息 |
**注意**
使用add_graph接口需要安装飞桨paddlepaddle, 安装步骤请参考[飞桨官方网站](https://www.paddlepaddle.org.cn/install/quick?docurl=/documentation/docs/zh/install/pip/linux-pip.html)
### Demo ### Demo
共有两种启动方式: 下面展示了使用 Graph 组件记录飞桨动态图模型的示例,代码见[Graph组件](https://github.com/PaddlePaddle/VisualDL/blob/develop/demo/components/graph_test.py)
- 前端启动Graph: ```python
import paddle
import paddle.nn as nn
import paddle.nn.functional as F
- 如只需使用Graph,无需添加任何参数,在命令行执行`visualdl`后即可启动。 from visualdl import LogWriter
- 如果同时需使用其他功能,在命令行指定日志文件路径(以`./log`为例),即可启动:
```shell
visualdl --logdir ./log --port 8080
```
class MyNet(nn.Layer):
def __init__(self):
super(MyNet, self).__init__()
self.conv1 = nn.Conv2D(
in_channels=1, out_channels=20, kernel_size=5, stride=1, padding=2)
self.max_pool1 = nn.MaxPool2D(kernel_size=2, stride=2)
self.conv2 = nn.Conv2D(
in_channels=20,
out_channels=20,
kernel_size=5,
stride=1,
padding=2)
self.max_pool2 = nn.MaxPool2D(kernel_size=2, stride=2)
self.fc = nn.Linear(in_features=980, out_features=10)
def forward(self, inputs):
x = self.conv1(inputs)
x = F.relu(x)
x = self.max_pool1(x)
x = self.conv2(x)
x = F.relu(x)
x = self.max_pool2(x)
x = paddle.reshape(x, [x.shape[0], -1])
x = self.fc(x)
return x
net = MyNet()
with LogWriter(logdir="./log/graph_test/") as writer:
writer.add_graph(
model=net,
input_spec=[paddle.static.InputSpec([-1, 1, 28, 28], 'float32')],
verbose=True)
```
- 后端启动Graph: 运行上述程序后,在命令行执行
- 在命令行加入参数`--model`并指定**模型文件**路径(非文件夹路径),即可启动: ```shell
visualdl --logdir ./log/graph_test/ --port 8080
```
```shell 接着在浏览器打开`http://127.0.0.1:8080`,即可查看Graph
visualdl --model ./log/model --port 8080
```
*Graph目前只支持可视化网络结构格式的模型文件(如__model__(注意此处为两个下划线'_'))
启动后即可查看网络结构可视化: 启动后即可查看飞桨动态图网络结构可视化:
<p align="center"> <p align="center">
<img src="https://user-images.githubusercontent.com/48054808/84490149-51e20580-acd5-11ea-9663-1f156892c0e0.png" width="80%"/> <img src="https://user-images.githubusercontent.com/22424850/175808777-5bc24d7a-9115-44d0-8709-a82a3e341b3d.gif" width="80%"/>
</p> </p>
### 功能操作说明 **注意**
VisualDL之前的版本支持通过--model参数直接指定模型结构文件,现在仍然保持这一选项,
通过`add_graph`接口导出的动态图模型文件(文件名包含"vdlgraph"), 在动态图页面展示,
并在页面中以'manual_input_model'来表示通过该参数指定的模型。其余所支持的文件格式在静态图页面中展示。
例如
```shell
visualdl --model ./log/model.pdmodel --port 8080
```
将展示在静态图页面。
```shell
visualdl --model ./log/vdlgraph.1655783158.log --port 8080
```
将展示在动态图页面。
- 一键上传模型
- 支持模型格式:PaddlePaddle、ONNX、Keras、Core ML、Caffe、Caffe2、Darknet、MXNet、ncnn、TensorFlow Lite
- 实验性支持模型格式:TorchScript、PyTorch、Torch、 ArmNN、BigDL、Chainer、CNTK、Deeplearning4j、MediaPipe、ML.NET、MNN、OpenVINO、Scikit-learn、Tengine、TensorFlow.js、TensorFlow
### 功能操作说明
当前Graph页面分为动态图和静态图两个页面。其中动态图页面用来展示通过add_graph接口导出的飞桨动态图模型结构,静态图页面用来展示飞桨静态图模型结构(通过飞桨的[paddle.jit.save](https://www.paddlepaddle.org.cn/documentation/docs/zh/api/paddle/jit/save_cn.html)导出的后缀名为pdmodel的文件)及其它可支持框架的模型。
<p align="center"> <p align="center">
<img src="https://user-images.githubusercontent.com/48054808/84487396-44c31780-acd1-11ea-831a-1632e636613d.png" width="80%"/> <img src="https://user-images.githubusercontent.com/22424850/175809463-56c0801a-744e-49ed-97bb-66122a351ff7.png" width="80%"/>
</p> </p>
**通用功能**:
- 支持上下左右任意拖拽模型、放大和缩小模型 - 支持上下左右任意拖拽模型、放大和缩小模型
<p align="center"> <p align="center">
...@@ -600,6 +670,44 @@ Graph组件一键可视化模型的网络结构。用于查看模型属性、节 ...@@ -600,6 +670,44 @@ Graph组件一键可视化模型的网络结构。用于查看模型属性、节
<img src="https://user-images.githubusercontent.com/48054808/84487998-27db1400-acd2-11ea-83d7-5d75832ef41d.png" width="25%"/> <img src="https://user-images.githubusercontent.com/48054808/84487998-27db1400-acd2-11ea-83d7-5d75832ef41d.png" width="25%"/>
</p> </p>
**动态图页面特有功能**:
- 展开和折叠指定节点
<p align="center">
<img src="https://user-images.githubusercontent.com/22424850/175809498-753ce2de-fc97-4050-88db-d70f7de47086.png" width="80%"/>
</p>
<p align="center">
<img src="https://user-images.githubusercontent.com/22424850/175809509-04232d9f-572c-4503-a58f-22d5c1e17e5b.png" width="80%"/>
</p>
- 一键全展开和全折叠
<p align="center">
<img src="https://user-images.githubusercontent.com/22424850/175809551-67f67552-59b7-4aca-a1cc-fe5606e7e512.png" width="80%"/>
</p>
<p align="center">
<img src="https://user-images.githubusercontent.com/22424850/175809610-d07c0b15-d1bd-4542-aacd-29b4d2ba8fcb.png" width="80%"/>
</p>
- 飞桨API链接功能
对于使用paddle.nn中的组件搭建的节点,可以使用alt+鼠标点击的方式跳转到官网的API说明文档。
<p align="center">
<img src="https://user-images.githubusercontent.com/22424850/175809570-241db056-0b41-4e3a-828d-376a011a4645.png" width="80%"/>
</p>
<p align="center">
<img src="https://user-images.githubusercontent.com/22424850/175809574-8c173b88-ed27-4c29-a3e1-a3887d2d3107.png" width="80%"/>
</p>
**静态图页面特有功能**:
- 一键上传模型
- 支持模型格式:PaddlePaddle、ONNX、Keras、Core ML、Caffe、Caffe2、Darknet、MXNet、ncnn、TensorFlow Lite
- 实验性支持模型格式:TorchScript、PyTorch、Torch、 ArmNN、BigDL、Chainer、CNTK、Deeplearning4j、MediaPipe、ML.NET、MNN、OpenVINO、Scikit-learn、Tengine、TensorFlow.js、TensorFlow
<p align="center">
<img src="https://user-images.githubusercontent.com/48054808/84487396-44c31780-acd1-11ea-831a-1632e636613d.png" width="80%"/>
</p>
## Histogram--直方图组件 ## Histogram--直方图组件
### 介绍 ### 介绍
......
...@@ -14,7 +14,6 @@ ...@@ -14,7 +14,6 @@
# ======================================================================= # =======================================================================
import json import json
import os import os
import tempfile
from visualdl.component.graph import analyse_model from visualdl.component.graph import analyse_model
from visualdl.component.graph import Model from visualdl.component.graph import Model
...@@ -56,7 +55,6 @@ class GraphReader(object): ...@@ -56,7 +55,6 @@ class GraphReader(object):
self.runs2displayname = {} self.runs2displayname = {}
self.graph_buffer = {} self.graph_buffer = {}
self.walks_buffer = {} self.walks_buffer = {}
self.tempfile = None
@property @property
def logdir(self): def logdir(self):
...@@ -64,11 +62,8 @@ class GraphReader(object): ...@@ -64,11 +62,8 @@ class GraphReader(object):
def get_all_walk(self): def get_all_walk(self):
flush_walks = {} flush_walks = {}
if 'manual_input_model' in self.walks:
flush_walks['manual_input_model'] = [
self.walks['manual_input_model']
]
for dir in self.dir: for dir in self.dir:
dir = os.path.realpath(dir)
for root, dirs, files in bfile.walk(dir): for root, dirs, files in bfile.walk(dir):
flush_walks.update({root: files}) flush_walks.update({root: files})
return flush_walks return flush_walks
...@@ -99,7 +94,10 @@ class GraphReader(object): ...@@ -99,7 +94,10 @@ class GraphReader(object):
def runs(self, update=True): def runs(self, update=True):
self.graphs(update=update) self.graphs(update=update)
return list(self.walks.keys()) graph_runs = list(self.walks.keys(
)) if 'manual_input_model' not in self.graph_buffer else list(
self.walks.keys()) + ['manual_input_model']
return sorted(graph_runs)
def get_graph(self, def get_graph(self,
run, run,
...@@ -108,6 +106,12 @@ class GraphReader(object): ...@@ -108,6 +106,12 @@ class GraphReader(object):
keep_state=False, keep_state=False,
expand_all=False, expand_all=False,
refresh=False): refresh=False):
if run == 'manual_input_model' and run in self.graph_buffer:
graph_model = self.graph_buffer[run]
if nodeid is not None:
graph_model.adjust_visible(nodeid, expand, keep_state)
return graph_model.make_graph(
refresh=refresh, expand_all=expand_all)
if run in self.walks: if run in self.walks:
if run in self.walks_buffer: if run in self.walks_buffer:
if self.walks[run] == self.walks_buffer[run]: if self.walks[run] == self.walks_buffer[run]:
...@@ -118,6 +122,9 @@ class GraphReader(object): ...@@ -118,6 +122,9 @@ class GraphReader(object):
refresh=refresh, expand_all=expand_all) refresh=refresh, expand_all=expand_all)
data = bfile.BFile(bfile.join(run, self.walks[run]), 'rb').read() data = bfile.BFile(bfile.join(run, self.walks[run]), 'rb').read()
if 'pdmodel' in self.walks[run]:
graph_model = Model(analyse_model(data))
else:
graph_model = Model(json.loads(data.decode())) graph_model = Model(json.loads(data.decode()))
self.graph_buffer[run] = graph_model self.graph_buffer[run] = graph_model
self.walks_buffer[run] = self.walks[run] self.walks_buffer[run] = self.walks[run]
...@@ -127,6 +134,11 @@ class GraphReader(object): ...@@ -127,6 +134,11 @@ class GraphReader(object):
refresh=refresh, expand_all=expand_all) refresh=refresh, expand_all=expand_all)
def search_graph_node(self, run, nodeid, keep_state=False, is_node=True): def search_graph_node(self, run, nodeid, keep_state=False, is_node=True):
if run == 'manual_input_model' and run in self.graph_buffer:
graph_model = self.graph_buffer[run]
graph_model.adjust_search_node_visible(
nodeid, keep_state=keep_state, is_node=is_node)
return graph_model.make_graph(refresh=False, expand_all=False)
if run in self.walks: if run in self.walks:
if run in self.walks_buffer: if run in self.walks_buffer:
if self.walks[run] == self.walks_buffer[run]: if self.walks[run] == self.walks_buffer[run]:
...@@ -137,6 +149,9 @@ class GraphReader(object): ...@@ -137,6 +149,9 @@ class GraphReader(object):
refresh=False, expand_all=False) refresh=False, expand_all=False)
data = bfile.BFile(bfile.join(run, self.walks[run]), 'rb').read() data = bfile.BFile(bfile.join(run, self.walks[run]), 'rb').read()
if 'pdmodel' in self.walks[run]:
graph_model = Model(analyse_model(data))
else:
graph_model = Model(json.loads(data.decode())) graph_model = Model(json.loads(data.decode()))
self.graph_buffer[run] = graph_model self.graph_buffer[run] = graph_model
self.walks_buffer[run] = self.walks[run] self.walks_buffer[run] = self.walks[run]
...@@ -145,6 +160,9 @@ class GraphReader(object): ...@@ -145,6 +160,9 @@ class GraphReader(object):
return graph_model.make_graph(refresh=False, expand_all=False) return graph_model.make_graph(refresh=False, expand_all=False)
def get_all_nodes(self, run): def get_all_nodes(self, run):
if run == 'manual_input_model' and run in self.graph_buffer:
graph_model = self.graph_buffer[run]
return graph_model.get_all_leaf_nodes()
if run in self.walks: if run in self.walks:
if run in self.walks_buffer: if run in self.walks_buffer:
if self.walks[run] == self.walks_buffer[run]: if self.walks[run] == self.walks_buffer[run]:
...@@ -152,6 +170,9 @@ class GraphReader(object): ...@@ -152,6 +170,9 @@ class GraphReader(object):
return graph_model.get_all_leaf_nodes() return graph_model.get_all_leaf_nodes()
data = bfile.BFile(bfile.join(run, self.walks[run]), 'rb').read() data = bfile.BFile(bfile.join(run, self.walks[run]), 'rb').read()
if 'pdmodel' in self.walks[run]:
graph_model = Model(analyse_model(data))
else:
graph_model = Model(json.loads(data.decode())) graph_model = Model(json.loads(data.decode()))
self.graph_buffer[run] = graph_model self.graph_buffer[run] = graph_model
self.walks_buffer[run] = self.walks[run] self.walks_buffer[run] = self.walks[run]
...@@ -167,10 +188,6 @@ class GraphReader(object): ...@@ -167,10 +188,6 @@ class GraphReader(object):
def __exit__(self, exc_type, exc_val, exc_tb): def __exit__(self, exc_type, exc_val, exc_tb):
pass pass
def __del__(self):
if self.tempfile:
os.unlink(self.tempfile.name)
def set_input_graph(self, content, file_type='pdmodel'): def set_input_graph(self, content, file_type='pdmodel'):
if isinstance(content, str): if isinstance(content, str):
if not is_VDLGraph_file(content): if not is_VDLGraph_file(content):
...@@ -184,23 +201,10 @@ class GraphReader(object): ...@@ -184,23 +201,10 @@ class GraphReader(object):
if file_type == 'pdmodel': if file_type == 'pdmodel':
data = analyse_model(content) data = analyse_model(content)
self.graph_buffer['manual_input_model'] = Model(data) self.graph_buffer['manual_input_model'] = Model(data)
temp = tempfile.NamedTemporaryFile(suffix='.pdmodel', delete=False)
temp.write(json.dumps(data).encode())
temp.close()
elif file_type == 'vdlgraph': elif file_type == 'vdlgraph':
self.graph_buffer['manual_input_model'] = Model( self.graph_buffer['manual_input_model'] = Model(
json.loads(content.decode())) json.loads(content.decode()))
temp = tempfile.NamedTemporaryFile(
suffix='.log', prefix='vdlgraph.', delete=False)
temp.write(content)
temp.close()
else: else:
return return
if self.tempfile:
os.unlink(self.tempfile.name)
self.tempfile = temp
self.walks['manual_input_model'] = temp.name
self.walks_buffer['manual_input_model'] = temp.name
...@@ -73,8 +73,9 @@ class Api(object): ...@@ -73,8 +73,9 @@ class Api(object):
self._graph_reader = GraphReader(logdir) self._graph_reader = GraphReader(logdir)
self._graph_reader.set_displayname(self._reader) self._graph_reader.set_displayname(self._reader)
if model: if model:
self._reader.model = model if 'vdlgraph' in model:
self._graph_reader.set_input_graph(model) self._graph_reader.set_input_graph(model)
self._reader.model = model
self.model_name = os.path.basename(model) self.model_name = os.path.basename(model)
else: else:
self.model_name = '' self.model_name = ''
...@@ -104,8 +105,7 @@ class Api(object): ...@@ -104,8 +105,7 @@ class Api(object):
def graph_runs(self): def graph_runs(self):
client_ip = request.remote_addr client_ip = request.remote_addr
graph_reader = self.graph_reader_client_manager.get_data(client_ip) graph_reader = self.graph_reader_client_manager.get_data(client_ip)
return self._get_with_reader('data/graph_runs', lib.get_graph_runs, return lib.get_graph_runs(graph_reader)
graph_reader)
@result() @result()
def tags(self): def tags(self):
...@@ -280,6 +280,13 @@ class Api(object): ...@@ -280,6 +280,13 @@ class Api(object):
key = os.path.join('data/plugin/roc_curves/steps', run) key = os.path.join('data/plugin/roc_curves/steps', run)
return self._get_with_retry(key, lib.get_roc_curve_step, run) return self._get_with_retry(key, lib.get_roc_curve_step, run)
@result('application/octet-stream', lambda s: {
"Content-Disposition": 'attachment; filename="%s"' % s.model_name
} if len(s.model_name) else None)
def graph_static_graph(self):
key = os.path.join('data/plugin/graphs/static_graph')
return self._get_with_retry(key, lib.get_static_graph)
@result() @result()
def graph_graph(self, run, expand_all, refresh): def graph_graph(self, run, expand_all, refresh):
client_ip = request.remote_addr client_ip = request.remote_addr
...@@ -395,6 +402,7 @@ def create_api_call(logdir, model, cache_timeout): ...@@ -395,6 +402,7 @@ def create_api_call(logdir, model, cache_timeout):
'embedding/metadata': (api.embedding_metadata, ['name']), 'embedding/metadata': (api.embedding_metadata, ['name']),
'histogram/list': (api.histogram_list, ['run', 'tag']), 'histogram/list': (api.histogram_list, ['run', 'tag']),
'graph/graph': (api.graph_graph, ['run', 'expand_all', 'refresh']), 'graph/graph': (api.graph_graph, ['run', 'expand_all', 'refresh']),
'graph/static_graph': (api.graph_static_graph, []),
'graph/upload': (api.graph_upload, []), 'graph/upload': (api.graph_upload, []),
'graph/search': (api.graph_search, 'graph/search': (api.graph_search,
['run', 'nodeid', 'keep_state', 'is_node']), ['run', 'nodeid', 'keep_state', 'is_node']),
......
...@@ -563,6 +563,14 @@ def get_histogram(log_reader, run, tag): ...@@ -563,6 +563,14 @@ def get_histogram(log_reader, run, tag):
return results return results
def get_static_graph(log_reader):
result = b""
if log_reader.model:
with bfile.BFile(log_reader.model, 'rb') as bfp:
result = bfp.read_file(log_reader.model)
return result
def get_graph(graph_reader, def get_graph(graph_reader,
run, run,
nodeid=None, nodeid=None,
......
...@@ -13,4 +13,4 @@ ...@@ -13,4 +13,4 @@
# limitations under the License. # limitations under the License.
# ======================================================================= # =======================================================================
vdl_version = '2.2.3' vdl_version = '2.3.0'
Markdown is supported
0% .
You are about to add 0 people to the discussion. Proceed with caution.
先完成此消息的编辑!
想要评论请 注册