diff --git a/examples/erniesage/README.en.md b/examples/erniesage/README.en.md new file mode 100644 index 0000000000000000000000000000000000000000..8a491a1ac3045e235ee24bcc82dd8cf0b2883a2f --- /dev/null +++ b/examples/erniesage/README.en.md @@ -0,0 +1,62 @@ +# ERNIESage in PGL + +[中文版 README](./README.md) + + +## Introduction +In many industrial applications, there is often a special graph shown below: Text Graph. As the name implies, the node attributes of such graph consist of text, and the edges provide structural information. Take the search scenario for example, nodes can be expressed by search query, web page titles, and web page content, while the edges are constructed by user feedback or hyperlink information. + +Text Graph + +**ERNIESage** (abbreviation of ERNIE SAmple aggreGatE), a model proposed by the PGL team, effectively improves the performance on text graph by simultaneously modeling text semantics and graph structure information. It's worth mentioning that [**ERNIE**](https://github.com/PaddlePaddle/ERNIE) in **ERNIESage** is a continual pre-training framework for language understanding launched by Baidu. + +**ERNIESage** is an aggregation of ERNIE and GraphSAGE. Its structure is shown in the figure below. The main idea is to use ERNIE as an aggregation function (Aggregators) to model the semantic and structural relationship between its own nodes and neighbor nodes. In addition, for the position-independent characteristics of neighbor nodes, attention mask and independent position embedding mechanism for neighbor blindness are designed. + +ERNIESage + +GraphSAGE with ID feature can only model the graph structure information, while ERNIE can only deal with the text. With the help of PGL, the proposed **ERNIESage** model can combine the advantages of both models. Take the following recommendation example of text graph, we can see that **ERNIESage** achieves the best performance when compared to single ERNIE model or GraphSAGE model. + +ERNIESage_result + +Thanks to the flexibility and usability of PGL, **ERNIESage** can be quickly implemented under PGL's Message Passing paradigm. Acutally, there are four PGL version of ERNIESage: + +- **ERNIESage v1**: ERNIE is applied to the NODE of the text graph; +- **ERNIESage v2**: ERNIE is applied to the EDGE of the text graph; +- **ERNIESage v3**: ERNIE is applied to the first order neighbors and center node; +- **ERNIESage v4**: ERNIE is applied to the N-order neighbors and center node. + +ERNIESage_v1_4 + +## Dependencies +- paddlepaddle>=1.7 +- pgl>=1.1 + +## Dataformat + +## How to run + +We adopt [PaddlePaddle Fleet](https://github.com/PaddlePaddle/Fleet) as our distributed training frameworks ```config/*.yaml``` are some example config files for hyperparameters. + +```sh +# train ERNIESage in distributed gpu mode. +sh local_run.sh config/enriesage_v1_gpu.yaml + +# train ERNIESage in distributed cpu mode. +sh local_run.sh config/enriesage_v1_cpu.yaml +``` + +## Hyperparamters + +- learner_type: `gpu` or `cpu`; gpu use fleet Collective mode, cpu use fleet Transpiler mode. + +## Citation +``` +@misc{ERNIESage, + author = {PGL Team}, + title = {ERNIESage: ERNIE SAmple aggreGatE}, + year = {2020}, + publisher = {GitHub}, + journal = {GitHub repository}, + howpublished = {\url{https://github.com/PaddlePaddle/PGL/tree/master/examples/erniesage}, +} +``` diff --git a/examples/erniesage/README.md b/examples/erniesage/README.md new file mode 100644 index 0000000000000000000000000000000000000000..e4c06786a026604233d6249e1170e7ef4258d40b --- /dev/null +++ b/examples/erniesage/README.md @@ -0,0 +1,62 @@ +# 使用PGL实现ERNIESage + +[ENG Readme](./README.en.md) + +## 背景介绍 + +在很多工业应用中,往往出现如下图所示的一种特殊的图:Text Graph。顾名思义,图的节点属性由文本构成,而边的构建提供了结构信息。如搜索场景下的Text Graph,节点可由搜索词、网页标题、网页正文来表达,用户反馈和超链信息则可构成边关系。 + +Text Graph + +**ERNIESage** 由PGL团队提出,是ERNIE SAmple aggreGatE的简称,该模型可以同时建模文本语义与图结构信息,有效提升 Text Graph 的应用效果。其中 [**ERNIE**](https://github.com/PaddlePaddle/ERNIE) 是百度推出的基于知识增强的持续学习语义理解框架, + +**ERNIESage** 是 ERNIE 与 GraphSAGE 碰撞的结果,是 ERNIE SAmple aggreGatE 的简称,它的结构如下图所示,主要思想是通过 ERNIE 作为聚合函数(Aggregators),建模自身节点和邻居节点的语义与结构关系。ERNIESage 对于文本的建模是构建在邻居聚合的阶段,中心节点文本会与所有邻居节点文本进行拼接;然后通过预训练的 ERNIE 模型进行消息汇聚,捕捉中心节点以及邻居节点之间的相互关系;最后使用 ERNIESage 搭配独特的邻居互相看不见的 Attention Mask 和独立的 Position Embedding 体系,就可以轻松构建 TextGraph 中句子之间以及词之间的关系。 + +ERNIESage + +使用ID特征的GraphSAGE只能够建模图的结构信息,而单独的ERNIE只能处理文本信息。通过PGL搭建的图与文本的桥梁,**ERNIESage**能够很简单的把GraphSAGE以及ERNIE的优点结合一起。以下面TextGraph的场景,**ERNIESage**的效果能够比单独的ERNIE以及GraphSAGE模型都要好。 + +ERNIESage_result + +**ERNIESage**可以很轻松地在PGL中的消息传递范式中进行实现,目前PGL提供了4个版本的ERNIESage模型: + +- **ERNIESage v1**: ERNIE 作用于text graph节点上; +- **ERNIESage v2**: ERNIE 作用在text graph的边上; +- **ERNIESage v3**: ERNIE 作用于一阶邻居及起边上; +- **ERNIESage v4**: ERNIE 作用于N阶邻居及边上; + +ERNIESage_v1_4 + +## 环境依赖 +- paddlepaddle>=1.7 +- pgl>=1.1 + +## Dataformat + +## How to run + +我们采用了[PaddlePaddle Fleet](https://github.com/PaddlePaddle/Fleet)作为我们的分布式训练框架,在```config/*.yaml```中,由部分用于训练ERNIESage的配置。 + +```sh +# 分布式GPU模式或单机模式ERNIESage +sh local_run.sh config/enriesage_v1_gpu.yaml + +# 分布式CPU模式训练ERNIESage +sh local_run.sh config/enriesage_v1_cpu.yaml +``` + +## Hyperparamters + +- learner_type: `gpu` or `cpu`; gpu 使用fleet Collective 模式, cpu 使用fleet Transpiler 模式. + +## Citation +``` +@misc{ERNIESage, + author = {PGL Team}, + title = {ERNIESage: ERNIE SAmple aggreGatE}, + year = {2020}, + publisher = {GitHub}, + journal = {GitHub repository}, + howpublished = {\url{https://github.com/PaddlePaddle/PGL/tree/master/examples/erniesage}, +} +``` diff --git a/examples/erniesage/docs/source/_static/ERNIESage_result.png b/examples/erniesage/docs/source/_static/ERNIESage_result.png new file mode 100644 index 0000000000000000000000000000000000000000..75c28240e1ec58e1603583df465a73815c97a2f8 Binary files /dev/null and b/examples/erniesage/docs/source/_static/ERNIESage_result.png differ diff --git a/examples/erniesage/docs/source/_static/ERNIESage_v1_4.png b/examples/erniesage/docs/source/_static/ERNIESage_v1_4.png new file mode 100644 index 0000000000000000000000000000000000000000..2b5869ab33464eb23511b759e2bb3263d8297004 Binary files /dev/null and b/examples/erniesage/docs/source/_static/ERNIESage_v1_4.png differ diff --git a/pgl/__init__.py b/pgl/__init__.py index 535632ea7afb3a157a4d92bbb08cb6445b21a125..93375e9ec5c7913334b02a05f6681de3f1ee4069 100644 --- a/pgl/__init__.py +++ b/pgl/__init__.py @@ -13,7 +13,7 @@ # limitations under the License. """Generate pgl apis """ -__version__ = "1.0.2" +__version__ = "1.1.0" from pgl import layers from pgl import graph_wrapper from pgl import graph