README.md 8.0 KB
Newer Older
littletomatodonkey's avatar
littletomatodonkey 已提交
1
English | [简体中文](README_ch.md)
W
weishengyu 已提交
2

littletomatodonkey's avatar
fix doc  
littletomatodonkey 已提交
3 4
## Style Text

littletomatodonkey's avatar
littletomatodonkey 已提交
5 6 7
### Contents
- [1. Introduction](#Introduction)
- [2. Preparation](#Preparation)
littletomatodonkey's avatar
fix doc  
littletomatodonkey 已提交
8 9
- [3. Quick Start](#Quick_Start)
- [4. Applications](#Applications)
littletomatodonkey's avatar
littletomatodonkey 已提交
10
- [5. Code Structure](#Code_structure)
W
weishengyu 已提交
11 12


littletomatodonkey's avatar
littletomatodonkey 已提交
13 14
<a name="Introduction"></a>
### Introduction
W
weishengyu 已提交
15

littletomatodonkey's avatar
littletomatodonkey 已提交
16 17 18 19 20
<div align="center">
    <img src="doc/images/3.png" width="800">
</div>

<div align="center">
littletomatodonkey's avatar
fix doc  
littletomatodonkey 已提交
21
    <img src="doc/images/9.png" width="600">
littletomatodonkey's avatar
littletomatodonkey 已提交
22 23 24
</div>


D
dyning 已提交
25
The Style-Text data synthesis tool is a tool based on Baidu and HUST cooperation research work, "Editing Text in the Wild" [https://arxiv.org/abs/1908.03047](https://arxiv.org/abs/1908.03047).
littletomatodonkey's avatar
littletomatodonkey 已提交
26 27 28 29 30 31

Different from the commonly used GAN-based data synthesis tools, the main framework of Style-Text includes:
* (1) Text foreground style transfer module.
* (2) Background extraction module.
* (3) Fusion module.

littletomatodonkey's avatar
littletomatodonkey 已提交
32
After these three steps, you can quickly realize the image text style transfer. The following figure is some results of the data synthesis tool.
littletomatodonkey's avatar
littletomatodonkey 已提交
33 34

<div align="center">
littletomatodonkey's avatar
fix doc  
littletomatodonkey 已提交
35
    <img src="doc/images/10.png" width="1000">
littletomatodonkey's avatar
littletomatodonkey 已提交
36 37 38 39
</div>


<a name="Preparation"></a>
W
weishengyu 已提交
40 41
#### Preparation

W
weishengyu 已提交
42
1. Please refer the [QUICK INSTALLATION](../doc/doc_en/installation_en.md) to install PaddlePaddle. Python3 environment is strongly recommended.
W
weishengyu 已提交
43 44 45
2. Download the pretrained models and unzip:

```bash
littletomatodonkey's avatar
littletomatodonkey 已提交
46 47
cd StyleText
wget https://paddleocr.bj.bcebos.com/dygraph_v2.0/style_text/style_text_models.zip
W
weishengyu 已提交
48 49 50
unzip style_text_models.zip
```

littletomatodonkey's avatar
littletomatodonkey 已提交
51
If you save the model in another location, please modify the address of the model file in `configs/config.yml`, and you need to modify these three configurations at the same time:
W
weishengyu 已提交
52 53 54 55 56 57 58 59 60 61 62 63

```
bg_generator:
  pretrain: style_text_rec/bg_generator
...
text_generator:
  pretrain: style_text_models/text_generator
...
fusion_generator:
  pretrain: style_text_models/fusion_generator
```

littletomatodonkey's avatar
fix doc  
littletomatodonkey 已提交
64 65
<a name="Quick_Start"></a>
### Quick Start
W
weishengyu 已提交
66

littletomatodonkey's avatar
littletomatodonkey 已提交
67
#### Synthesis single image
W
weishengyu 已提交
68

littletomatodonkey's avatar
fix doc  
littletomatodonkey 已提交
69
1. You can run `tools/synth_image` and generate the demo image, which is saved in the current folder.
W
weishengyu 已提交
70

littletomatodonkey's avatar
littletomatodonkey 已提交
71
```python
72
python3 tools/synth_image.py -c configs/config.yml --style_image examples/style_images/2.jpg --text_corpus PaddleOCR --language en
W
weishengyu 已提交
73 74
```

75
* Note 1: The language options is correspond to the corpus. Currently, the tool only supports English, Simplified Chinese and Korean.
76
* Note 2: Synth-Text is mainly used to generate images for OCR recognition models.
77
  So the height of style images should be around 32 pixels. Images in other sizes may behave poorly.
78
* Note 3: You can modify `use_gpu` in `configs/config.yml` to determine whether to use GPU for prediction.
79 80


littletomatodonkey's avatar
fix doc  
littletomatodonkey 已提交
81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101

For example, enter the following image and corpus `PaddleOCR`.

<div align="center">
    <img src="examples/style_images/2.jpg" width="300">
</div>

The result `fake_fusion.jpg` will be generated.

<div align="center">
    <img src="doc/images/4.jpg" width="300">
</div>

What's more, the medium result `fake_bg.jpg` will also be saved, which is the background output.

<div align="center">
    <img src="doc/images/7.jpg" width="300">
</div>


`fake_text.jpg` * `fake_text.jpg` is the generated image with the same font style as `Style Input`.
W
weishengyu 已提交
102 103


littletomatodonkey's avatar
fix doc  
littletomatodonkey 已提交
104 105 106
<div align="center">
    <img src="doc/images/8.jpg" width="300">
</div>
W
weishengyu 已提交
107 108


littletomatodonkey's avatar
littletomatodonkey 已提交
109
#### Batch synthesis
W
weishengyu 已提交
110

littletomatodonkey's avatar
fix doc  
littletomatodonkey 已提交
111
In actual application scenarios, it is often necessary to synthesize pictures in batches and add them to the training set. StyleText can use a batch of style pictures and corpus to synthesize data in batches. The synthesis process is as follows:
W
weishengyu 已提交
112 113

1. The referenced dataset can be specifed in `configs/dataset_config.yml`:
littletomatodonkey's avatar
littletomatodonkey 已提交
114

littletomatodonkey's avatar
fix doc  
littletomatodonkey 已提交
115 116 117 118 119 120 121 122 123
   * `Global`
     * `output_dir:`:Output synthesis data path.
   * `StyleSampler`
     * `image_home`:style images' folder.
     * `label_file`:Style images' file list. If label is provided, then it is the label file path.
     * `with_label`:Whether the `label_file` is label file list.
   * `CorpusGenerator`
     * `method`:Method of CorpusGenerator,supports `FileCorpus` and `EnNumCorpus`. If `EnNumCorpus` is used,No other configuration is needed,otherwise you need to set `corpus_file` and `language`.
     * `language`:Language of the corpus.
W
Wei Shengyu 已提交
124
     * `corpus_file`: Filepath of the corpus. Corpus file should be a text file which will be split by line-endings('\n'). Corpus generator samples one line each time.
W
Wei Shengyu 已提交
125 126 127


Example of corpus file: 
W
Wei Shengyu 已提交
128 129 130
```
PaddleOCR
飞桨文字识别
W
Wei Shengyu 已提交
131 132
StyleText
风格文本图像数据合成
W
Wei Shengyu 已提交
133
```
W
weishengyu 已提交
134

littletomatodonkey's avatar
littletomatodonkey 已提交
135
We provide a general dataset containing Chinese, English and Korean (50,000 images in all) for your trial ([download link](https://paddleocr.bj.bcebos.com/dygraph_v2.0/style_text/chkoen_5w.tar)), some examples are given below :
littletomatodonkey's avatar
littletomatodonkey 已提交
136 137 138 139 140

<div align="center">
     <img src="doc/images/5.png" width="800">
</div>

W
weishengyu 已提交
141 142 143
2. You can run the following command to start synthesis task:

   ``` bash
144
   python3 tools/synth_dataset.py -c configs/dataset_config.yml
W
weishengyu 已提交
145
   ```
E
Evezerest 已提交
146

147
We also provide example corpus and images in `examples` folder.
148 149 150 151 152 153 154 155 156 157 158
    <div align="center">
        <img src="examples/style_images/1.jpg" width="300">
        <img src="examples/style_images/2.jpg" width="300">
    </div>
If you run the code above directly, you will get example output data in `output_data` folder.
You will get synthesis images and labels as below:
   <div align="center">
       <img src="doc/images/12.png" width="800">
   </div>
There will be some cache under the `label` folder. If the program exit unexpectedly, you can find cached labels there.
When the program finish normally, you will find all the labels in `label.txt` which give the final results.
littletomatodonkey's avatar
littletomatodonkey 已提交
159

littletomatodonkey's avatar
fix doc  
littletomatodonkey 已提交
160 161
<a name="Applications"></a>
### Applications
littletomatodonkey's avatar
littletomatodonkey 已提交
162 163 164
We take two scenes as examples, which are metal surface English number recognition and general Korean recognition, to illustrate practical cases of using StyleText to synthesize data to improve text recognition. The following figure shows some examples of real scene images and composite images:

<div align="center">
littletomatodonkey's avatar
fix doc  
littletomatodonkey 已提交
165
    <img src="doc/images/11.png" width="800">
littletomatodonkey's avatar
littletomatodonkey 已提交
166 167 168 169 170
</div>


After adding the above synthetic data for training, the accuracy of the recognition model is improved, which is shown in the following table:

L
littletomatodonkey 已提交
171

littletomatodonkey's avatar
littletomatodonkey 已提交
172
| Scenario | Characters | Raw Data | Test Data | Only Use Raw Data</br>Recognition Accuracy | New Synthetic Data | Simultaneous Use of Synthetic Data</br>Recognition Accuracy | Index Improvement |
L
littletomatodonkey 已提交
173 174 175
| -------- | ---------- | -------- | -------- | -------------------------- | ------------ | ---------------------- | -------- |
| Metal surface | English and numbers | 2203     | 650      | 0.5938                     | 20000        | 0.7546                 | 16%      |
| Random background | Korean       | 5631     | 1230     | 0.3012                     | 100000       | 0.5057                 | 20%      |
littletomatodonkey's avatar
littletomatodonkey 已提交
176 177 178 179


<a name="Code_structure"></a>
### Code Structure
littletomatodonkey's avatar
fix doc  
littletomatodonkey 已提交
180

littletomatodonkey's avatar
littletomatodonkey 已提交
181
```
littletomatodonkey's avatar
littletomatodonkey 已提交
182
StyleText
littletomatodonkey's avatar
fix doc  
littletomatodonkey 已提交
183
|-- arch                        // Network module files.
littletomatodonkey's avatar
littletomatodonkey 已提交
184 185 186 187 188
|   |-- base_module.py
|   |-- decoder.py
|   |-- encoder.py
|   |-- spectral_norm.py
|   `-- style_text_rec.py
littletomatodonkey's avatar
fix doc  
littletomatodonkey 已提交
189
|-- configs                     // Config files.
littletomatodonkey's avatar
littletomatodonkey 已提交
190 191
|   |-- config.yml
|   `-- dataset_config.yml
littletomatodonkey's avatar
fix doc  
littletomatodonkey 已提交
192 193 194 195 196 197 198 199
|-- engine                      // Synthesis engines.
|   |-- corpus_generators.py    // Sample corpus from file or generate random corpus.
|   |-- predictors.py           // Predict using network.
|   |-- style_samplers.py       // Sample style images.
|   |-- synthesisers.py         // Manage other engines to synthesis images.
|   |-- text_drawers.py         // Generate standard input text images.
|   `-- writers.py              // Write synthesis images and labels into files.
|-- examples                    // Example files.
littletomatodonkey's avatar
littletomatodonkey 已提交
200 201 202 203 204 205
|   |-- corpus
|   |   `-- example.txt
|   |-- image_list.txt
|   `-- style_images
|       |-- 1.jpg
|       `-- 2.jpg
littletomatodonkey's avatar
fix doc  
littletomatodonkey 已提交
206
|-- fonts                       // Font files.
littletomatodonkey's avatar
littletomatodonkey 已提交
207 208 209
|   |-- ch_standard.ttf
|   |-- en_standard.ttf
|   `-- ko_standard.ttf
littletomatodonkey's avatar
fix doc  
littletomatodonkey 已提交
210
|-- tools                       // Program entrance.
littletomatodonkey's avatar
littletomatodonkey 已提交
211
|   |-- __init__.py
littletomatodonkey's avatar
fix doc  
littletomatodonkey 已提交
212 213 214
|   |-- synth_dataset.py        // Synthesis dataset.
|   `-- synth_image.py          // Synthesis image.
`-- utils                       // Module of basic functions.
littletomatodonkey's avatar
littletomatodonkey 已提交
215 216 217 218 219 220
    |-- config.py
    |-- load_params.py
    |-- logging.py
    |-- math_functions.py
    `-- sys_funcs.py
```