add rank model BST (#134)

* add rank model BST

* update readme

README.md

@@ -56,6 +56,7 @@
     |         Rank          |                  [xDeepFM](models/rank/xdeepfm/model.py)                  |   ✓   |    x    |        ✓         |     x     | [KDD 2018][xDeepFM: Combining Explicit and Implicit Feature Interactions for Recommender Systems](https://dl.acm.org/doi/pdf/10.1145/3219819.3220023)                                                       |
     |         Rank          |                      [DIN](models/rank/din/model.py)                      |   ✓   |    x    |        ✓         |     x     | [KDD 2018][Deep Interest Network for Click-Through Rate Prediction](https://dl.acm.org/doi/pdf/10.1145/3219819.3219823)                                                                                     |
     |   Rank   |                      [DIEN](models/rank/dien/model.py)                      |    ✓    |    x    |     ✓     |     x     | [AAAI 2019][Deep Interest Evolution Network for Click-Through Rate Prediction](https://www.aaai.org/ojs/index.php/AAAI/article/view/4545/4423)                                                                                     |
+    |   Rank   |                      [BST](models/rank/BST/model.py)                      |    ✓    |    x    |     ✓     |     x     | [DLP-KDD 2019][Behavior Sequence Transformer for E-commerce Recommendation in Alibaba](https://arxiv.org/pdf/1905.06874v1.pdf)                                                                                     |
     |   Rank   |                      [AutoInt](models/rank/AutoInt/model.py)                      |    ✓    |    x    |     ✓     |     x     | [CIKM 2019][AutoInt: Automatic Feature Interaction Learning via Self-Attentive Neural Networks](https://arxiv.org/pdf/1810.11921.pdf)                                                                                     |
     |         Rank          |                [Wide&Deep](models/rank/wide_deep/model.py)                |   ✓   |    x    |        ✓         |     x     | [DLRS 2016][Wide & Deep Learning for Recommender Systems](https://dl.acm.org/doi/pdf/10.1145/2988450.2988454)                                                                                               |
     |         Rank          |                    [FGCNN](models/rank/fgcnn/model.py)                    |   ✓   |    ✓    |        ✓         |     ✓     | [WWW 2019][Feature Generation by Convolutional Neural Network for Click-Through Rate Prediction](https://arxiv.org/pdf/1904.04447.pdf)                                                                      |

README_CN.md

@@ -61,6 +61,7 @@
     |   排序   |                  [xDeepFM](models/rank/xdeepfm/model.py)                  |    ✓    |    x    |     ✓     |     x     | [KDD 2018][xDeepFM: Combining Explicit and Implicit Feature Interactions for Recommender Systems](https://dl.acm.org/doi/pdf/10.1145/3219819.3220023)                                                       |
     |   排序   |                      [DIN](models/rank/din/model.py)                      |    ✓    |    x    |     ✓     |     x     | [KDD 2018][Deep Interest Network for Click-Through Rate Prediction](https://dl.acm.org/doi/pdf/10.1145/3219819.3219823)                                                                                     |
     |   排序   |                      [DIEN](models/rank/dien/model.py)                      |    ✓    |    x    |     ✓     |     x     | [AAAI 2019][Deep Interest Evolution Network for Click-Through Rate Prediction](https://www.aaai.org/ojs/index.php/AAAI/article/view/4545/4423)                                                                                     |
+    |   排序   |                      [BST](models/rank/BST/model.py)                      |    ✓    |    x    |     ✓     |     x     | [DLP_KDD 2019][Behavior Sequence Transformer for E-commerce Recommendation in Alibaba](https://arxiv.org/pdf/1905.06874v1.pdf)                                                                                     |
     |   排序   |                      [AutoInt](models/rank/AutoInt/model.py)                      |    ✓    |    x    |     ✓     |     x     | [CIKM 2019][AutoInt: Automatic Feature Interaction Learning via Self-Attentive Neural Networks](https://arxiv.org/pdf/1810.11921.pdf)                                                                                     |
     |   排序   |                [Wide&Deep](models/rank/wide_deep/model.py)                |    ✓    |    x    |     ✓     |     x     | [DLRS 2016][Wide & Deep Learning for Recommender Systems](https://dl.acm.org/doi/pdf/10.1145/2988450.2988454)                                                                                               |
     |   排序   |                    [FGCNN](models/rank/fgcnn/model.py)                    |    ✓    |    ✓    |     ✓     |     ✓     | [WWW 2019][Feature Generation by Convolutional Neural Network for Click-Through Rate Prediction](https://arxiv.org/pdf/1904.04447.pdf)                                                                      |

models/rank/BST/__init__.py

+# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.

models/rank/BST/config.yaml

+# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# global settings 
+debug: false
+workspace: "paddlerec.models.rank.BST"
+
+dataset:
+- name: sample_1
+  type: DataLoader
+  batch_size: 5
+  data_path: "{workspace}/data/train_data"
+  sparse_slots: "label history cate position target target_cate target_position"
+- name: infer_sample
+  type: DataLoader
+  batch_size: 5
+  data_path: "{workspace}/data/train_data"
+  sparse_slots: "label history cate position target target_cate target_position"
+
+hyper_parameters:
+    optimizer:
+        class: SGD
+        learning_rate: 0.0001
+    use_DataLoader: True
+    item_emb_size: 96
+    cat_emb_size: 96
+    position_emb_size: 96
+    is_sparse: False
+    item_count: 63001
+    cat_count: 801
+    position_count: 5001
+    n_encoder_layers: 1
+    d_model: 288
+    d_key: 48
+    d_value: 48
+    n_head: 6
+    dropout_rate: 0
+    postprocess_cmd: "da"
+    prepostprocess_dropout: 0
+    d_inner_hid: 512
+    relu_dropout: 0.0
+    act: "relu"
+    fc_sizes: [1024, 512, 256]
+
+
+mode: train_runner
+
+runner:
+  - name: train_runner
+    class: train
+    epochs: 1
+    device: cpu
+    init_model_path: ""
+    save_checkpoint_interval: 1
+    save_inference_interval: 1
+    save_checkpoint_path: "increment_BST"
+    save_inference_path: "inference_BST"
+    print_interval: 1
+  - name: infer_runner
+    class: infer
+    device: cpu
+    init_model_path: "increment_BST/0"
+    print_interval: 1
+    
+phase:
+- name: phase1
+  model: "{workspace}/model.py"
+  dataset_name: sample_1
+  thread_num: 1
+#- name: infer_phase
+#  model: "{workspace}/model.py"
+#  dataset_name: infer_sample
+#  thread_num: 1

models/rank/BST/data/build_dataset.py

+# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from __future__ import print_function
+import random
+import pickle
+
+random.seed(1234)
+
+print("read and process data")
+
+with open('./raw_data/remap.pkl', 'rb') as f:
+    reviews_df = pickle.load(f)
+    cate_list = pickle.load(f)
+    user_count, item_count, cate_count, example_count = pickle.load(f)
+
+train_set = []
+test_set = []
+for reviewerID, hist in reviews_df.groupby('reviewerID'):
+    pos_list = hist['asin'].tolist()
+    time_list = hist['unixReviewTime'].tolist()
+
+    def gen_neg():
+        neg = pos_list[0]
+        while neg in pos_list:
+            neg = random.randint(0, item_count - 1)
+        return neg
+
+    neg_list = [gen_neg() for i in range(len(pos_list))]
+
+    for i in range(1, len(pos_list)):
+        hist = pos_list[:i]
+        # set maximum position value
+        time_seq = [
+            min(int((time_list[i] - time_list[j]) / (3600 * 24)), 5000)
+            for j in range(i)
+        ]
+        if i != len(pos_list) - 1:
+            train_set.append((reviewerID, hist, pos_list[i], 1, time_seq))
+            train_set.append((reviewerID, hist, neg_list[i], 0, time_seq))
+        else:
+            label = (pos_list[i], neg_list[i])
+            test_set.append((reviewerID, hist, label, time_seq))
+
+random.shuffle(train_set)
+random.shuffle(test_set)
+
+assert len(test_set) == user_count
+
+
+def print_to_file(data, fout, slot):
+    if not isinstance(data, list):
+        data = [data]
+    for i in range(len(data)):
+        fout.write(slot + ":" + str(data[i]))
+        fout.write(' ')
+
+
+print("make train data")
+with open("paddle_train.txt", "w") as fout:
+    for line in train_set:
+        history = line[1]
+        target = line[2]
+        label = line[3]
+        position = line[4]
+        cate = [cate_list[x] for x in history]
+        print_to_file(history, fout, "history")
+        print_to_file(cate, fout, "cate")
+        print_to_file(position, fout, "position")
+        print_to_file(target, fout, "target")
+        print_to_file(cate_list[target], fout, "target_cate")
+        print_to_file(0, fout, "target_position")
+        print_to_file(label, fout, "label")
+        fout.write("\n")
+
+print("make test data")
+with open("paddle_test.txt", "w") as fout:
+    for line in test_set:
+        history = line[1]
+        target = line[2]
+        position = line[3]
+        cate = [cate_list[x] for x in history]
+
+        print_to_file(history, fout, "history")
+        print_to_file(cate, fout, "cate")
+        print_to_file(position, fout, "position")
+        print_to_file(target[0], fout, "target")
+        print_to_file(cate_list[target[0]], fout, "target_cate")
+        print_to_file(0, fout, "target_position")
+        fout.write("label:1\n")
+
+        print_to_file(history, fout, "history")
+        print_to_file(cate, fout, "cate")
+        print_to_file(position, fout, "position")
+        print_to_file(target[0], fout, "target")
+        print_to_file(cate_list[target[1]], fout, "target_cate")
+        print_to_file(0, fout, "target_position")
+        fout.write("label:0\n")
+
+print("make config data")
+with open('config.txt', 'w') as f:
+    f.write(str(user_count) + "\n")
+    f.write(str(item_count) + "\n")
+    f.write(str(cate_count) + "\n")
+    f.wrire(str(50000) + "\n")

models/rank/BST/data/convert_pd.py

+# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from __future__ import print_function
+import pickle
+import pandas as pd
+
+
+def to_df(file_path):
+    with open(file_path, 'r') as fin:
+        df = {}
+        i = 0
+        for line in fin:
+            df[i] = eval(line)
+            i += 1
+        df = pd.DataFrame.from_dict(df, orient='index')
+        return df
+
+
+print("start to analyse reviews_Electronics_5.json")
+reviews_df = to_df('./raw_data/reviews_Electronics_5.json')
+with open('./raw_data/reviews.pkl', 'wb') as f:
+    pickle.dump(reviews_df, f, pickle.HIGHEST_PROTOCOL)
+
+print("start to analyse meta_Electronics.json")
+meta_df = to_df('./raw_data/meta_Electronics.json')
+meta_df = meta_df[meta_df['asin'].isin(reviews_df['asin'].unique())]
+meta_df = meta_df.reset_index(drop=True)
+with open('./raw_data/meta.pkl', 'wb') as f:
+    pickle.dump(meta_df, f, pickle.HIGHEST_PROTOCOL)

models/rank/BST/data/data_process.sh

+#! /bin/bash
+
+set -e
+echo "begin download data"
+mkdir raw_data
+cd raw_data
+wget -c http://snap.stanford.edu/data/amazon/productGraph/categoryFiles/reviews_Electronics_5.json.gz
+gzip -d reviews_Electronics_5.json.gz
+wget -c http://snap.stanford.edu/data/amazon/productGraph/categoryFiles/meta_Electronics.json.gz
+gzip -d meta_Electronics.json.gz
+echo "download data successfully"
+
+cd ..
+python convert_pd.py
+python remap_id.py

models/rank/BST/data/remap_id.py

+# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+from __future__ import print_function
+import random
+import pickle
+import numpy as np
+
+random.seed(1234)
+
+with open('./raw_data/reviews.pkl', 'rb') as f:
+    reviews_df = pickle.load(f)
+    reviews_df = reviews_df[['reviewerID', 'asin', 'unixReviewTime']]
+with open('./raw_data/meta.pkl', 'rb') as f:
+    meta_df = pickle.load(f)
+    meta_df = meta_df[['asin', 'categories']]
+    meta_df['categories'] = meta_df['categories'].map(lambda x: x[-1][-1])
+
+
+def build_map(df, col_name):
+    key = sorted(df[col_name].unique().tolist())
+    m = dict(zip(key, range(len(key))))
+    df[col_name] = df[col_name].map(lambda x: m[x])
+    return m, key
+
+
+asin_map, asin_key = build_map(meta_df, 'asin')
+cate_map, cate_key = build_map(meta_df, 'categories')
+revi_map, revi_key = build_map(reviews_df, 'reviewerID')
+
+user_count, item_count, cate_count, example_count =\
+    len(revi_map), len(asin_map), len(cate_map), reviews_df.shape[0]
+print('user_count: %d\titem_count: %d\tcate_count: %d\texample_count: %d' %
+      (user_count, item_count, cate_count, example_count))
+
+meta_df = meta_df.sort_values('asin')
+meta_df = meta_df.reset_index(drop=True)
+reviews_df['asin'] = reviews_df['asin'].map(lambda x: asin_map[x])
+reviews_df = reviews_df.sort_values(['reviewerID', 'unixReviewTime'])
+reviews_df = reviews_df.reset_index(drop=True)
+reviews_df = reviews_df[['reviewerID', 'asin', 'unixReviewTime']]
+
+cate_list = [meta_df['categories'][i] for i in range(len(asin_map))]
+cate_list = np.array(cate_list, dtype=np.int32)
+
+with open('./raw_data/remap.pkl', 'wb') as f:
+    pickle.dump(reviews_df, f, pickle.HIGHEST_PROTOCOL)  # uid, iid
+    pickle.dump(cate_list, f, pickle.HIGHEST_PROTOCOL)  # cid of iid line
+    pickle.dump((user_count, item_count, cate_count, example_count), f,
+                pickle.HIGHEST_PROTOCOL)
+    pickle.dump((asin_key, cate_key, revi_key), f, pickle.HIGHEST_PROTOCOL)

models/rank/BST/model.py

+#   Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import math
+from functools import partial
+
+import numpy as np
+import paddle.fluid as fluid
+import paddle.fluid.layers as layers
+
+from paddlerec.core.utils import envs
+from paddlerec.core.model import ModelBase
+
+
+def positionwise_feed_forward(x, d_inner_hid, d_hid, dropout_rate):
+    """
+    Position-wise Feed-Forward Networks.
+    This module consists of two linear transformations with a ReLU activation
+    in between, which is applied to each position separately and identically.
+    """
+    hidden = layers.fc(input=x,
+                       size=d_inner_hid,
+                       num_flatten_dims=2,
+                       act="relu")
+    if dropout_rate:
+        hidden = layers.dropout(
+            hidden,
+            dropout_prob=dropout_rate,
+            seed=dropout_seed,
+            is_test=False)
+    out = layers.fc(input=hidden, size=d_hid, num_flatten_dims=2)
+    return out
+
+
+def pre_post_process_layer(prev_out, out, process_cmd, dropout_rate=0.):
+    """
+    Add residual connection, layer normalization and droput to the out tensor
+    optionally according to the value of process_cmd.
+    This will be used before or after multi-head attention and position-wise
+    feed-forward networks.
+    """
+    for cmd in process_cmd:
+        if cmd == "a":  # add residual connection
+            out = out + prev_out if prev_out else out
+        elif cmd == "n":  # add layer normalization
+            out = layers.layer_norm(
+                out,
+                begin_norm_axis=len(out.shape) - 1,
+                param_attr=fluid.initializer.Constant(1.),
+                bias_attr=fluid.initializer.Constant(0.))
+        elif cmd == "d":  # add dropout
+            if dropout_rate:
+                out = layers.dropout(
+                    out,
+                    dropout_prob=dropout_rate,
+                    seed=dropout_seed,
+                    is_test=False)
+    return out
+
+
+pre_process_layer = partial(pre_post_process_layer, None)
+post_process_layer = pre_post_process_layer
+
+
+class Model(ModelBase):
+    def __init__(self, config):
+        ModelBase.__init__(self, config)
+
+    def _init_hyper_parameters(self):
+        self.item_emb_size = envs.get_global_env(
+            "hyper_parameters.item_emb_size", 64)
+        self.cat_emb_size = envs.get_global_env(
+            "hyper_parameters.cat_emb_size", 64)
+        self.position_emb_size = envs.get_global_env(
+            "hyper_parameters.position_emb_size", 64)
+        self.act = envs.get_global_env("hyper_parameters.act", "sigmoid")
+        self.is_sparse = envs.get_global_env("hyper_parameters.is_sparse",
+                                             False)
+        # significant for speeding up the training process
+        self.use_DataLoader = envs.get_global_env(
+            "hyper_parameters.use_DataLoader", False)
+        self.item_count = envs.get_global_env("hyper_parameters.item_count",
+                                              63001)
+        self.cat_count = envs.get_global_env("hyper_parameters.cat_count", 801)
+        self.position_count = envs.get_global_env(
+            "hyper_parameters.position_count", 5001)
+        self.n_encoder_layers = envs.get_global_env(
+            "hyper_parameters.n_encoder_layers", 1)
+        self.d_model = envs.get_global_env("hyper_parameters.d_model", 96)
+        self.d_key = envs.get_global_env("hyper_parameters.d_key", None)
+        self.d_value = envs.get_global_env("hyper_parameters.d_value", None)
+        self.n_head = envs.get_global_env("hyper_parameters.n_head", None)
+        self.dropout_rate = envs.get_global_env(
+            "hyper_parameters.dropout_rate", 0.0)
+        self.postprocess_cmd = envs.get_global_env(
+            "hyper_parameters.postprocess_cmd", "da")
+        self.preprocess_cmd = envs.get_global_env(
+            "hyper_parameters.postprocess_cmd", "n")
+        self.prepostprocess_dropout = envs.get_global_env(
+            "hyper_parameters.prepostprocess_dropout", 0.0)
+        self.d_inner_hid = envs.get_global_env("hyper_parameters.d_inner_hid",
+                                               512)
+        self.relu_dropout = envs.get_global_env(
+            "hyper_parameters.relu_dropout", 0.0)
+        self.layer_sizes = envs.get_global_env("hyper_parameters.fc_sizes",
+                                               None)
+
+    def multi_head_attention(self, queries, keys, values, d_key, d_value,
+                             d_model, n_head, dropout_rate):
+        keys = queries if keys is None else keys
+        values = keys if values is None else values
+        if not (len(queries.shape) == len(keys.shape) == len(values.shape) == 3
+                ):
+            raise ValueError(
+                "Inputs: quries, keys and values should all be 3-D tensors.")
+
+        def __compute_qkv(queries, keys, values, n_head, d_key, d_value):
+            """
+            Add linear projection to queries, keys, and values.
+            """
+            q = fluid.layers.fc(input=queries,
+                                size=d_key * n_head,
+                                bias_attr=False,
+                                num_flatten_dims=2)
+            k = fluid.layers.fc(input=keys,
+                                size=d_key * n_head,
+                                bias_attr=False,
+                                num_flatten_dims=2)
+            v = fluid.layers.fc(input=values,
+                                size=d_value * n_head,
+                                bias_attr=False,
+                                num_flatten_dims=2)
+            return q, k, v
+
+        def __split_heads_qkv(queries, keys, values, n_head, d_key, d_value):
+            """
+            Reshape input tensors at the last dimension to split multi-heads 
+            and then transpose. Specifically, transform the input tensor with shape
+            [bs, max_sequence_length, n_head * hidden_dim] to the output tensor
+            with shape [bs, n_head, max_sequence_length, hidden_dim].
+            """
+            # The value 0 in shape attr means copying the corresponding dimension
+            # size of the input as the output dimension size.
+            reshaped_q = fluid.layers.reshape(
+                x=queries, shape=[0, 0, n_head, d_key], inplace=True)
+            # permuate the dimensions into:
+            # [batch_size, n_head, max_sequence_len, hidden_size_per_head]
+            q = fluid.layers.transpose(x=reshaped_q, perm=[0, 2, 1, 3])
+            # For encoder-decoder attention in inference, insert the ops and vars
+            # into global block to use as cache among beam search.
+            reshaped_k = fluid.layers.reshape(
+                x=keys, shape=[0, 0, n_head, d_key], inplace=True)
+            k = fluid.layers.transpose(x=reshaped_k, perm=[0, 2, 1, 3])
+            reshaped_v = fluid.layers.reshape(
+                x=values, shape=[0, 0, n_head, d_value], inplace=True)
+            v = fluid.layers.transpose(x=reshaped_v, perm=[0, 2, 1, 3])
+
+            return q, k, v
+
+        def scaled_dot_product_attention(q, k, v, d_key, dropout_rate):
+            """
+            Scaled Dot-Product Attention
+            """
+            product = fluid.layers.matmul(
+                x=q, y=k, transpose_y=True, alpha=d_key**-0.5)
+
+            weights = fluid.layers.softmax(product)
+            if dropout_rate:
+                weights = fluid.layers.dropout(
+                    weights,
+                    dropout_prob=dropout_rate,
+                    seed=None,
+                    is_test=False)
+            out = fluid.layers.matmul(weights, v)
+            return out
+
+        def __combine_heads(x):
+            """
+            Transpose and then reshape the last two dimensions of inpunt tensor x
+            so that it becomes one dimension, which is reverse to __split_heads.
+            """
+            if len(x.shape) != 4:
+                raise ValueError("Input(x) should be a 4-D Tensor.")
+
+            trans_x = fluid.layers.transpose(x, perm=[0, 2, 1, 3])
+            # The value 0 in shape attr means copying the corresponding dimension
+            # size of the input as the output dimension size.
+            return fluid.layers.reshape(
+                x=trans_x,
+                shape=[0, 0, trans_x.shape[2] * trans_x.shape[3]],
+                inplace=True)
+
+        q, k, v = __compute_qkv(queries, keys, values, n_head, d_key, d_value)
+        q, k, v = __split_heads_qkv(q, k, v, n_head, d_key, d_value)
+
+        ctx_multiheads = scaled_dot_product_attention(q, k, v, d_model,
+                                                      dropout_rate)
+
+        out = __combine_heads(ctx_multiheads)
+
+        proj_out = fluid.layers.fc(input=out,
+                                   size=d_model,
+                                   bias_attr=False,
+                                   num_flatten_dims=2)
+
+        return proj_out
+
+    def encoder_layer(self, x):
+        attention_out = self.multi_head_attention(
+            pre_process_layer(x, self.preprocess_cmd,
+                              self.prepostprocess_dropout), None, None,
+            self.d_key, self.d_value, self.d_model, self.n_head,
+            self.dropout_rate)
+        attn_output = post_process_layer(x, attention_out,
+                                         self.postprocess_cmd,
+                                         self.prepostprocess_dropout)
+        ffd_output = positionwise_feed_forward(
+            pre_process_layer(attn_output, self.preprocess_cmd,
+                              self.prepostprocess_dropout), self.d_inner_hid,
+            self.d_model, self.relu_dropout)
+        return post_process_layer(attn_output, ffd_output,
+                                  self.postprocess_cmd,
+                                  self.prepostprocess_dropout)
+
+    def net(self, inputs, is_infer=False):
+
+        init_value_ = 0.1
+
+        hist_item_seq = self._sparse_data_var[1]
+        hist_cat_seq = self._sparse_data_var[2]
+        position_seq = self._sparse_data_var[3]
+        target_item = self._sparse_data_var[4]
+        target_cat = self._sparse_data_var[5]
+        target_position = self._sparse_data_var[6]
+        self.label = self._sparse_data_var[0]
+
+        item_emb_attr = fluid.ParamAttr(name="item_emb")
+        cat_emb_attr = fluid.ParamAttr(name="cat_emb")
+        position_emb_attr = fluid.ParamAttr(name="position_emb")
+
+        hist_item_emb = fluid.embedding(
+            input=hist_item_seq,
+            size=[self.item_count, self.item_emb_size],
+            param_attr=item_emb_attr,
+            is_sparse=self.is_sparse)
+
+        hist_cat_emb = fluid.embedding(
+            input=hist_cat_seq,
+            size=[self.cat_count, self.cat_emb_size],
+            param_attr=cat_emb_attr,
+            is_sparse=self.is_sparse)
+
+        hist_position_emb = fluid.embedding(
+            input=hist_cat_seq,
+            size=[self.position_count, self.position_emb_size],
+            param_attr=position_emb_attr,
+            is_sparse=self.is_sparse)
+
+        target_item_emb = fluid.embedding(
+            input=target_item,
+            size=[self.item_count, self.item_emb_size],
+            param_attr=item_emb_attr,
+            is_sparse=self.is_sparse)
+
+        target_cat_emb = fluid.embedding(
+            input=target_cat,
+            size=[self.cat_count, self.cat_emb_size],
+            param_attr=cat_emb_attr,
+            is_sparse=self.is_sparse)
+
+        target_position_emb = fluid.embedding(
+            input=target_position,
+            size=[self.position_count, self.position_emb_size],
+            param_attr=position_emb_attr,
+            is_sparse=self.is_sparse)
+
+        item_sequence_target = fluid.layers.reduce_sum(
+            fluid.layers.sequence_concat([hist_item_emb, target_item_emb]),
+            dim=1)
+        cat_sequence_target = fluid.layers.reduce_sum(
+            fluid.layers.sequence_concat([hist_cat_emb, target_cat_emb]),
+            dim=1)
+        position_sequence_target = fluid.layers.reduce_sum(
+            fluid.layers.sequence_concat(
+                [hist_position_emb, target_position_emb]),
+            dim=1)
+
+        whole_embedding_withlod = fluid.layers.concat(
+            [
+                item_sequence_target, cat_sequence_target,
+                position_sequence_target
+            ],
+            axis=1)
+        pad_value = fluid.layers.assign(input=np.array(
+            [0.0], dtype=np.float32))
+        whole_embedding, _ = fluid.layers.sequence_pad(whole_embedding_withlod,
+                                                       pad_value)
+
+        for _ in range(self.n_encoder_layers):
+            enc_output = self.encoder_layer(whole_embedding)
+            enc_input = enc_output
+        enc_output = pre_process_layer(enc_output, self.preprocess_cmd,
+                                       self.prepostprocess_dropout)
+
+        dnn_input = fluid.layers.reduce_sum(enc_output, dim=1)
+
+        for s in self.layer_sizes:
+            dnn_input = fluid.layers.fc(
+                input=dnn_input,
+                size=s,
+                act=self.act,
+                param_attr=fluid.ParamAttr(
+                    initializer=fluid.initializer.TruncatedNormalInitializer(
+                        loc=0.0, scale=init_value_ / math.sqrt(float(10)))),
+                bias_attr=fluid.ParamAttr(
+                    initializer=fluid.initializer.TruncatedNormalInitializer(
+                        loc=0.0, scale=init_value_)))
+
+        y_dnn = fluid.layers.fc(input=dnn_input, size=1, act=None)
+
+        self.predict = fluid.layers.sigmoid(y_dnn)
+        cost = fluid.layers.log_loss(
+            input=self.predict, label=fluid.layers.cast(self.label, "float32"))
+        avg_cost = fluid.layers.reduce_sum(cost)
+
+        self._cost = avg_cost
+
+        predict_2d = fluid.layers.concat([1 - self.predict, self.predict], 1)
+        label_int = fluid.layers.cast(self.label, 'int64')
+        auc_var, batch_auc_var, _ = fluid.layers.auc(input=predict_2d,
+                                                     label=label_int,
+                                                     slide_steps=0)
+        self._metrics["AUC"] = auc_var
+        self._metrics["BATCH_AUC"] = batch_auc_var
+        if is_infer:
+            self._infer_results["AUC"] = auc_var

models/rank/readme.md

@@ -37,6 +37,7 @@
 | xDeepFM | xDeepFM | [xDeepFM: Combining Explicit and Implicit Feature Interactions for Recommender Systems](https://dl.acm.org/doi/pdf/10.1145/3219819.3220023)(2018) |
 | DIN | Deep Interest Network | [Deep Interest Network for Click-Through Rate Prediction](https://dl.acm.org/doi/pdf/10.1145/3219819.3219823)(2018) |
 | DIEN | Deep Interest Evolution Network | [Deep Interest Evolution Network for Click-Through Rate Prediction](https://www.aaai.org/ojs/index.php/AAAI/article/view/4545/4423)(2019) |
+| BST | transformer in user behavior sequence for rank | [Behavior Sequence Transformer for E-commerce Recommendation in Alibaba](https://arxiv.org/pdf/1905.06874v1.pdf)(2019) |
 | FGCNN | Feature Generation by CNN | [Feature Generation by Convolutional Neural Network for Click-Through Rate Prediction](https://arxiv.org/pdf/1904.04447.pdf)(2019) |
 | FIBINET | Combining Feature Importance and Bilinear feature Interaction | [《FiBiNET: Combining Feature Importance and Bilinear feature Interaction for Click-Through Rate Prediction》]( https://arxiv.org/pdf/1905.09433.pdf)(2019) |
 | FLEN | Leveraging Field for Scalable CTR Prediction | [《FLEN: Leveraging Field for Scalable CTR Prediction》]( https://arxiv.org/pdf/1911.04690.pdf)（2019） |