Merge branch 'develop' into psgan

modules/image/Image_gan/gan/styleganv2_mixing/README.md

+# styleganv2_mixing
+
+|模型名称|styleganv2_mixing|
+| :--- | :---: |
+|类别|图像 - 图像生成|
+|网络|StyleGAN V2|
+|数据集|-|
+|是否支持Fine-tuning|否|
+|模型大小|190MB|
+|最新更新日期|2021-12-23|
+|数据指标|-|
+
+
+## 一、模型基本信息  
+
+- ### 应用效果展示
+  - 样例结果示例：
+    <p align="center">
+    <img src="https://user-images.githubusercontent.com/22424850/147241001-3babb1bd-98d4-4a9c-a61d-2298fca041e1.jpg"  width = "40%"  hspace='10'/>
+    <br />
+    输入图像1
+    <br />
+    <img src="https://user-images.githubusercontent.com/22424850/147241006-0bc2cda8-d271-4cfd-8a0d-e6feea7bf167.jpg"  width = "40%"  hspace='10'/>
+    <br />
+    输入图像2
+    <br />
+    <img src="https://user-images.githubusercontent.com/22424850/147241020-f4420729-c489-4661-b43f-c929c62c0ce7.png"  width = "40%"  hspace='10'/>
+    <br />
+    输出图像
+     <br />
+    </p>
+
+- ### 模型介绍
+
+  - StyleGAN V2 的任务是使用风格向量进行image generation，而Mixing模块则是利用其风格向量实现两张生成图像不同层次不同比例的混合。
+
+
+
+## 二、安装
+
+- ### 1、环境依赖  
+  - paddlepaddle >= 2.1.0
+  - paddlehub >= 2.1.0    | [如何安装PaddleHub](../../../../docs/docs_ch/get_start/installation.rst)
+
+- ### 2、安装
+
+  - ```shell
+    $ hub install styleganv2_mixing
+    ```
+  - 如您安装时遇到问题，可参考：[零基础windows安装](../../../../docs/docs_ch/get_start/windows_quickstart.md)
+ | [零基础Linux安装](../../../../docs/docs_ch/get_start/linux_quickstart.md) | [零基础MacOS安装](../../../../docs/docs_ch/get_start/mac_quickstart.md)
+
+## 三、模型API预测
+
+- ### 1、命令行预测
+
+  - ```shell
+    # Read from a file
+    $ hub run styleganv2_mixing --image1 "/PATH/TO/IMAGE1" --image2 "/PATH/TO/IMAGE2"
+    ```
+  - 通过命令行方式实现人脸融合模型的调用，更多请见 [PaddleHub命令行指令](../../../../docs/docs_ch/tutorial/cmd_usage.rst)
+
+- ### 2、预测代码示例
+
+  - ```python
+    import paddlehub as hub
+
+    module = hub.Module(name="styleganv2_mixing")
+    input_path = ["/PATH/TO/IMAGE"]
+    # Read from a file
+    module.generate(paths=input_path, direction_name = 'age', direction_offset = 5, output_dir='./editing_result/', use_gpu=True)  
+    ```
+
+- ### 3、API
+
+  - ```python
+    generate(self, images=None, paths=None, weights = [0.5] * 18, output_dir='./mixing_result/', use_gpu=False, visualization=True)
+    ```
+    - 人脸融合生成API。
+
+    - **参数**
+      - images (list[dict]): data of images, 每一个元素都为一个 dict，有关键字 image1, image2, 相应取值为：
+          - image1 (numpy.ndarray): 待融合的图片1，shape 为 \[H, W, C\]，BGR格式；<br/>
+          - image2 (numpy.ndarray) : 待融合的图片2，shape为 \[H, W, C\]，BGR格式；<br/>
+      - paths (list[str]): paths to images, 每一个元素都为一个dict, 有关键字 image1, image2, 相应取值为：
+          - image1 (str): 待融合的图片1的路径；<br/>
+          - image2 (str) : 待融合的图片2的路径；<br/>
+      - weights (list(float)): 融合的权重
+      - images (list\[numpy.ndarray\]): 图片数据 <br/>
+      - paths (list\[str\]): 图片路径；<br/>
+      - output\_dir (str): 结果保存的路径； <br/>
+      - use\_gpu (bool): 是否使用 GPU；<br/>
+      - visualization(bool): 是否保存结果到本地文件夹
+
+
+## 四、服务部署
+
+- PaddleHub Serving可以部署一个在线人脸融合服务。
+
+- ### 第一步：启动PaddleHub Serving
+
+  - 运行启动命令：
+  - ```shell
+    $ hub serving start -m styleganv2_mixing
+    ```
+
+  - 这样就完成了一个人脸融合的在线服务API的部署，默认端口号为8866。
+
+  - **NOTE:** 如使用GPU预测，则需要在启动服务之前，请设置CUDA\_VISIBLE\_DEVICES环境变量，否则不用设置。
+
+- ### 第二步：发送预测请求
+
+  - 配置好服务端，以下数行代码即可实现发送预测请求，获取预测结果
+
+  - ```python
+    import requests
+    import json
+    import cv2
+    import base64
+
+
+    def cv2_to_base64(image):
+      data = cv2.imencode('.jpg', image)[1]
+      return base64.b64encode(data.tostring()).decode('utf8')
+
+    # 发送HTTP请求
+    data = {'images':[{'image1': cv2_to_base64(cv2.imread("/PATH/TO/IMAGE1")),'image2': cv2_to_base64(cv2.imread("/PATH/TO/IMAGE2"))}]}
+    headers = {"Content-type": "application/json"}
+    url = "http://127.0.0.1:8866/predict/styleganv2_mixing"
+    r = requests.post(url=url, headers=headers, data=json.dumps(data))
+
+    # 打印预测结果
+    print(r.json()["results"])
+
+## 五、更新历史
+
+* 1.0.0
+
+  初始发布
+
+  - ```shell
+    $ hub install styleganv2_mixing==1.0.0
+    ```

modules/image/Image_gan/gan/styleganv2_mixing/basemodel.py

+#   Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserve.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import os
+import random
+import numpy as np
+import paddle
+from ppgan.models.generators import StyleGANv2Generator
+from ppgan.utils.download import get_path_from_url
+from ppgan.utils.visual import make_grid, tensor2img, save_image
+
+model_cfgs = {
+    'ffhq-config-f': {
+        'model_urls': 'https://paddlegan.bj.bcebos.com/models/stylegan2-ffhq-config-f.pdparams',
+        'size': 1024,
+        'style_dim': 512,
+        'n_mlp': 8,
+        'channel_multiplier': 2
+    },
+    'animeface-512': {
+        'model_urls': 'https://paddlegan.bj.bcebos.com/models/stylegan2-animeface-512.pdparams',
+        'size': 512,
+        'style_dim': 512,
+        'n_mlp': 8,
+        'channel_multiplier': 2
+    }
+}
+
+
+@paddle.no_grad()
+def get_mean_style(generator):
+    mean_style = None
+
+    for i in range(10):
+        style = generator.mean_latent(1024)
+
+        if mean_style is None:
+            mean_style = style
+
+        else:
+            mean_style += style
+
+    mean_style /= 10
+    return mean_style
+
+
+@paddle.no_grad()
+def sample(generator, mean_style, n_sample):
+    image = generator(
+        [paddle.randn([n_sample, generator.style_dim])],
+        truncation=0.7,
+        truncation_latent=mean_style,
+    )[0]
+
+    return image
+
+
+@paddle.no_grad()
+def style_mixing(generator, mean_style, n_source, n_target):
+    source_code = paddle.randn([n_source, generator.style_dim])
+    target_code = paddle.randn([n_target, generator.style_dim])
+
+    resolution = 2**((generator.n_latent + 2) // 2)
+
+    images = [paddle.ones([1, 3, resolution, resolution]) * -1]
+
+    source_image = generator([source_code], truncation_latent=mean_style, truncation=0.7)[0]
+    target_image = generator([target_code], truncation_latent=mean_style, truncation=0.7)[0]
+
+    images.append(source_image)
+
+    for i in range(n_target):
+        image = generator(
+            [target_code[i].unsqueeze(0).tile([n_source, 1]), source_code],
+            truncation_latent=mean_style,
+            truncation=0.7,
+        )[0]
+        images.append(target_image[i].unsqueeze(0))
+        images.append(image)
+
+    images = paddle.concat(images, 0)
+
+    return images
+
+
+class StyleGANv2Predictor:
+    def __init__(self,
+                 output_path='output_dir',
+                 weight_path=None,
+                 model_type=None,
+                 seed=None,
+                 size=1024,
+                 style_dim=512,
+                 n_mlp=8,
+                 channel_multiplier=2):
+        self.output_path = output_path
+
+        if weight_path is None:
+            if model_type in model_cfgs.keys():
+                weight_path = get_path_from_url(model_cfgs[model_type]['model_urls'])
+                size = model_cfgs[model_type].get('size', size)
+                style_dim = model_cfgs[model_type].get('style_dim', style_dim)
+                n_mlp = model_cfgs[model_type].get('n_mlp', n_mlp)
+                channel_multiplier = model_cfgs[model_type].get('channel_multiplier', channel_multiplier)
+                checkpoint = paddle.load(weight_path)
+            else:
+                raise ValueError('Predictor need a weight path or a pretrained model type')
+        else:
+            checkpoint = paddle.load(weight_path)
+
+        self.generator = StyleGANv2Generator(size, style_dim, n_mlp, channel_multiplier)
+        self.generator.set_state_dict(checkpoint)
+        self.generator.eval()
+
+        if seed is not None:
+            paddle.seed(seed)
+            random.seed(seed)
+            np.random.seed(seed)
+
+    def run(self, n_row=3, n_col=5):
+        os.makedirs(self.output_path, exist_ok=True)
+        mean_style = get_mean_style(self.generator)
+
+        img = sample(self.generator, mean_style, n_row * n_col)
+        save_image(tensor2img(make_grid(img, nrow=n_col)), f'{self.output_path}/sample.png')
+
+        for j in range(2):
+            img = style_mixing(self.generator, mean_style, n_col, n_row)
+            save_image(tensor2img(make_grid(img, nrow=n_col + 1)), f'{self.output_path}/sample_mixing_{j}.png')

modules/image/Image_gan/gan/styleganv2_mixing/model.py

+#   Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserve.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import os
+import cv2
+import numpy as np
+import paddle
+
+from .basemodel import StyleGANv2Predictor
+
+
+def make_image(tensor):
+    return (((tensor.detach() + 1) / 2 * 255).clip(min=0, max=255).transpose((0, 2, 3, 1)).numpy().astype('uint8'))
+
+
+class StyleGANv2MixingPredictor(StyleGANv2Predictor):
+    @paddle.no_grad()
+    def run(self, latent1, latent2, weights=[0.5] * 18):
+
+        latent1 = paddle.to_tensor(latent1).unsqueeze(0)
+        latent2 = paddle.to_tensor(latent2).unsqueeze(0)
+        assert latent1.shape[1] == latent2.shape[1] == len(
+            weights), 'latents and their weights should have the same level nums.'
+        mix_latent = []
+        for i, weight in enumerate(weights):
+            mix_latent.append(latent1[:, i:i + 1] * weight + latent2[:, i:i + 1] * (1 - weight))
+        mix_latent = paddle.concat(mix_latent, 1)
+        latent_n = paddle.concat([latent1, latent2, mix_latent], 0)
+        generator = self.generator
+        img_gen, _ = generator([latent_n], input_is_latent=True, randomize_noise=False)
+        imgs = make_image(img_gen)
+        src_img1 = imgs[0]
+        src_img2 = imgs[1]
+        dst_img = imgs[2]
+
+        return src_img1, src_img2, dst_img

modules/image/Image_gan/gan/styleganv2_mixing/module.py

+# Copyright (c) 2021 PaddlePaddle Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import os
+import argparse
+import copy
+
+import paddle
+import paddlehub as hub
+from paddlehub.module.module import moduleinfo, runnable, serving
+import numpy as np
+import cv2
+from skimage.io import imread
+from skimage.transform import rescale, resize
+
+from .model import StyleGANv2MixingPredictor
+from .util import base64_to_cv2
+
+
+@moduleinfo(
+    name="styleganv2_mixing",
+    type="CV/style_transfer",
+    author="paddlepaddle",
+    author_email="",
+    summary="",
+    version="1.0.0")
+class styleganv2_mixing:
+    def __init__(self):
+        self.pretrained_model = os.path.join(self.directory, "stylegan2-ffhq-config-f.pdparams")
+        self.network = StyleGANv2MixingPredictor(weight_path=self.pretrained_model, model_type='ffhq-config-f')
+        self.pixel2style2pixel_module = hub.Module(name='pixel2style2pixel')
+
+    def generate(self,
+                 images=None,
+                 paths=None,
+                 weights=[0.5] * 18,
+                 output_dir='./mixing_result/',
+                 use_gpu=False,
+                 visualization=True):
+        '''
+        images (list[dict]): data of images, each element is a dict，the keys are as below：
+          - image1 (numpy.ndarray): image1 to be mixed，shape is \[H, W, C\]，BGR format；<br/>
+          - image2 (numpy.ndarray) : image2 to be mixed，shape is \[H, W, C\]，BGR format；<br/>
+        paths (list[str]): paths to images, each element is a dict，the keys are as below：
+          - image1 (str): path to image1；<br/>
+          - image2 (str) : path to image2；<br/>
+        weights (list(float)): weight for mixing
+        output_dir: the dir to save the results
+        use_gpu: if True, use gpu to perform the computation, otherwise cpu.
+        visualization: if True, save results in output_dir.
+        '''
+        results = []
+        paddle.disable_static()
+        place = 'gpu:0' if use_gpu else 'cpu'
+        place = paddle.set_device(place)
+        if images == None and paths == None:
+            print('No image provided. Please input an image or a image path.')
+            return
+        if images != None:
+            for image_dict in images:
+                image1 = image_dict['image1'][:, :, ::-1]
+                image2 = image_dict['image2'][:, :, ::-1]
+                _, latent1 = self.pixel2style2pixel_module.network.run(image1)
+                _, latent2 = self.pixel2style2pixel_module.network.run(image2)
+                results.append(self.network.run(latent1, latent2, weights))
+
+        if paths != None:
+            for path_dict in paths:
+                path1 = path_dict['image1']
+                path2 = path_dict['image2']
+                image1 = cv2.imread(path1)[:, :, ::-1]
+                image2 = cv2.imread(path2)[:, :, ::-1]
+                _, latent1 = self.pixel2style2pixel_module.network.run(image1)
+                _, latent2 = self.pixel2style2pixel_module.network.run(image2)
+                results.append(self.network.run(latent1, latent2, weights))
+
+        if visualization == True:
+            if not os.path.exists(output_dir):
+                os.makedirs(output_dir, exist_ok=True)
+            for i, out in enumerate(results):
+                if out is not None:
+                    cv2.imwrite(os.path.join(output_dir, 'src_{}_image1.png'.format(i)), out[0][:, :, ::-1])
+                    cv2.imwrite(os.path.join(output_dir, 'src_{}_image2.png'.format(i)), out[1][:, :, ::-1])
+                    cv2.imwrite(os.path.join(output_dir, 'dst_{}.png'.format(i)), out[2][:, :, ::-1])
+
+        return results
+
+    @runnable
+    def run_cmd(self, argvs: list):
+        """
+        Run as a command.
+        """
+        self.parser = argparse.ArgumentParser(
+            description="Run the {} module.".format(self.name),
+            prog='hub run {}'.format(self.name),
+            usage='%(prog)s',
+            add_help=True)
+
+        self.arg_input_group = self.parser.add_argument_group(title="Input options", description="Input data. Required")
+        self.arg_config_group = self.parser.add_argument_group(
+            title="Config options", description="Run configuration for controlling module behavior, not required.")
+        self.add_module_config_arg()
+        self.add_module_input_arg()
+        self.args = self.parser.parse_args(argvs)
+        results = self.generate(
+            paths=[{
+                'image1': self.args.image1,
+                'image2': self.args.image2
+            }],
+            weights=self.args.weights,
+            output_dir=self.args.output_dir,
+            use_gpu=self.args.use_gpu,
+            visualization=self.args.visualization)
+        return results
+
+    @serving
+    def serving_method(self, images, **kwargs):
+        """
+        Run as a service.
+        """
+        images_decode = copy.deepcopy(images)
+        for image in images_decode:
+            image['image1'] = base64_to_cv2(image['image1'])
+            image['image2'] = base64_to_cv2(image['image2'])
+        results = self.generate(images_decode, **kwargs)
+        tolist = [result.tolist() for result in results]
+        return tolist
+
+    def add_module_config_arg(self):
+        """
+        Add the command config options.
+        """
+        self.arg_config_group.add_argument('--use_gpu', action='store_true', help="use GPU or not")
+
+        self.arg_config_group.add_argument(
+            '--output_dir', type=str, default='mixing_result', help='output directory for saving result.')
+        self.arg_config_group.add_argument('--visualization', type=bool, default=False, help='save results or not.')
+
+    def add_module_input_arg(self):
+        """
+        Add the command input options.
+        """
+        self.arg_input_group.add_argument('--image1', type=str, help="path to input image1.")
+        self.arg_input_group.add_argument('--image2', type=str, help="path to input image2.")
+        self.arg_input_group.add_argument(
+            "--weights",
+            type=float,
+            nargs="+",
+            default=[0.5] * 18,
+            help="different weights at each level of two latent codes")

modules/image/Image_gan/gan/styleganv2_mixing/requirements.txt

+ppgan

modules/image/Image_gan/gan/styleganv2_mixing/util.py

+import base64
+import cv2
+import numpy as np
+
+
+def base64_to_cv2(b64str):
+    data = base64.b64decode(b64str.encode('utf8'))
+    data = np.fromstring(data, np.uint8)
+    data = cv2.imdecode(data, cv2.IMREAD_COLOR)
+    return data