> * 原文地址：[(CAN’T) PICTURE THIS: An Analysis of Image Filtering on WeChat Moments — Part 1](https://citizenlab.ca/2018/08/cant-picture-this-an-analysis-of-image-filtering-on-wechat-moments/)
> * 原文作者：[Jeffrey Knockel](https://citizenlab.ca/author/jknockel/), [Lotus Ruan](https://citizenlab.ca/author/lotus/), [Masashi Crete-Nishihata](https://citizenlab.ca/author/masashi/), and [Ron Deibert](https://citizenlab.ca/author/profd/)
> * 译文出自：[掘金翻译计划](https://github.com/xitu/gold-miner)
> * 本文永久链接：[https://github.com/xitu/gold-miner/blob/master/TODO1/cant-picture-this-an-analysis-of-image-filtering-on-wechat-moments-1.md](https://github.com/xitu/gold-miner/blob/master/TODO1/cant-picture-this-an-analysis-of-image-filtering-on-wechat-moments-1.md)
> * 译者：
> * 校对者：

# (CAN’T) PICTURE THIS: An Analysis of Image Filtering on WeChat Moments — Part 1

> * [(CAN’T) PICTURE THIS: An Analysis of Image Filtering on WeChat Moments — Part 1](https://github.com/xitu/gold-miner/blob/master/TODO1/cant-picture-this-an-analysis-of-image-filtering-on-wechat-moments-1.md)
> * [(CAN’T) PICTURE THIS: An Analysis of Image Filtering on WeChat Moments — Part 2](https://github.com/xitu/gold-miner/blob/master/TODO1/cant-picture-this-an-analysis-of-image-filtering-on-wechat-moments-2.md)
> * [(CAN’T) PICTURE THIS: An Analysis of Image Filtering on WeChat Moments — Part 3](https://github.com/xitu/gold-miner/blob/master/TODO1/cant-picture-this-an-analysis-of-image-filtering-on-wechat-moments-3.md)
> * [(CAN’T) PICTURE THIS: An Analysis of Image Filtering on WeChat Moments — Part 4](https://github.com/xitu/gold-miner/blob/master/TODO1/cant-picture-this-an-analysis-of-image-filtering-on-wechat-moments-4.md)

## Key findings

*   WeChat (the most popular chat app in China) uses two different algorithms to filter images in Moments: an OCR-based one that filters images containing sensitive text and a visual-based one that filters images that are visually similar to those on an image blacklist
*   We discovered that the OCR-based algorithm has implementation details common to many OCR algorithms in that it converts images to grayscale and uses blob merging to consolidate characters
*   We found that the visual-based algorithm is not based on any machine learning approach that uses high level classification of an image to determine whether it is sensitive or not; however, we found that the algorithm does possess other surprising properties
*   For both the OCR- and visual-based algorithms, we uncovered multiple implementation details that informed techniques to evade the filter
*   By analyzing and understanding how both the OCR- and visual-based filtering algorithms operate, we are able to discover weaknesses in both algorithms that allow one to upload images perceptually similar to those prohibited but that evade filtering

## Introduction

WeChat, (_Weixin_ 微信 in Chinese), is the dominant chat application in China and fourth largest in the world. In February 2018, WeChat reportedly [hit](http://www.scmp.com/tech/apps-gaming/article/2135690/tencents-wechat-hits-1-billion-milestone-lunar-new-year-boosts) one billion monthly active users during the Chinese Lunar New Year. The application is owned and operated by Tencent, one of China’s largest technology companies. In the past two years, WeChat has transformed beyond a commercial social media platform and become part of China’s e-governance initiatives. China’s Ministry of Public Security has been [collaborating](http://www.g.com.cn/tech/21472402/) with Tencent to implement the country’s national identification card system on WeChat.

Chinese users spend [a third](http://www.economist.com/news/business/21703428-chinas-wechat-shows-way-social-medias-future-wechats-world) of their mobile online time on WeChat and typically return to the app ten times a day or more. Among WeChat’s many functions, the [most frequently used](http://tech.qq.com/a/20160321/030364.htm) feature is [WeChat Moments](http://blog.wechat.com/2015/06/12/tech-tip-your-guide-to-wechat-moments/) (朋友圈), which resembles Facebook’s Timeline and allows users to share images, videos, and articles. Moments has a relatively high level of intimacy, because a user’s updates on Moments can only be seen by friends who have been verified or selected by the user, and a user can only see interactions of people who are already on their WeChat contact list. Because of such perceived privacy, users [reported](https://t.qianzhan.com/caijing/detail/170424-8f9569e1.html) that they frequently share details of their daily life and express personal opinions on Moments.

Operating a chat application in China requires following laws and regulations on content control and monitoring. Previous Citizen Lab research [uncovered](https://citizenlab.ca/2016/11/wechat-china-censorship-one-app-two-systems/) that WeChat censors content–both text and images–and [demonstrates](https://citizenlab.ca/2017/11/managing-message-censorship-19th-national-communist-party-congress-wechat/) that censorship is heightened around sensitive events. Our previous work [found](https://citizenlab.ca/2017/04/we-cant-chat-709-crackdown-discussions-blocked-on-weibo-and-wechat/) that WeChat uses a hash-based system to filter images in one-to-one and group chats. However, image censorship on Moments is more complex: an image is filtered according to its content in a way that is tolerant to some modifications to the image.

In this report, we present our findings studying the implementation of image filtering on WeChat Moments. We found two different algorithms that WeChat Moments uses to filter images: an OCR-based one that filters images containing sensitive text and a visual-based one that filters images that are visually similar to those on an image blacklist. We found that the OCR-based algorithm has similarities to many common OCR algorithms in that it converts images to grayscale and uses blob merging to consolidate characters. We also found that the visual-based algorithm is not based on any machine learning approach that uses high level classification of an image to determine whether it is sensitive or not; however, we found that the algorithm does possess other surprising properties. By understanding how both the OCR- and visual-based algorithms work, we are able to discover weaknesses in both algorithms that allow one to upload images perceptually similar to those blacklisted but that evade filtering.

Through our findings we provide a better understanding of how image filtering is implemented on an application with over one billion users. We hope that our methods can be used as a road map for future research studying image filtering on other platforms.

## Regulatory Environment in China

WeChat thrives on the huge user base it has amassed in China, but the Chinese market carries unique challenges. As Chinese social media applications continue to gain popularity, authorities have introduced tighter content controls.

Any Internet company operating in China is subject to laws and regulations that hold companies legally responsible for content on their platforms. Companies are expected to invest in staff and filtering technologies to moderate content and stay in compliance with government regulations. [Failure to comply](http://www.wsj.com/articles/china-threatens-sina-corp-over-insufficient-censorship-1428743575) can lead to fines or revocation of operating licenses. This environment creates a system of “intermediary liability” where responsibility of content control is pushed down to companies.

In 2010, China’s State Council Information Office (SCIO) published a major government-issued document on its Internet policy. It includes [a list of prohibited topics](http://www.humanrights.cn/cn/rqlt/rqwj/rqbps/t20100608_605154_1.htm) that are vaguely defined, including “disrupting social order and stability” and “damaging state honor and interests.” Control over the Internet in China has tightened since 2012 following the establishment of the Cyberspace Administration of China (CAC). The CAC has [become](http://www.xinhuanet.com/english/2017-05/03/c_136251798.htm) the new regulator of online news services replacing the SCIO. Chinese President Xi Jinping directly heads the CAC, which signals a dramatic change of the leadership’s attitudes towards Internet management: It is [a matter of national security](http://www.cac.gov.cn/2014-02/27/c_133148354.htm) and that the (CPC) [must](https://news.qq.com/a/20160321/020121.htm) control the Internet just as how it controls traditional media.

Recent regulations push content control liability down to the user level. In 2014, the CAC introduced regulations informally referred to as the “[WeChat Ten Doctrines](http://paper.people.com.cn/xaq/html/2014-11/01/content_1522983.htm)”, which emphasizes the implementation of a real-name registration system and a prohibition against activities that violate the “seven baselines” of observing laws and regulations, the Socialist system, the national interest, citizens’ lawful rights and interests, public order, social morality, and truthfulness of information. In 2017, the CAC released four major regulations on Internet management, ranging from strengthening real-name registration requirements on [Internet forums](http://www.cac.gov.cn/2017-08/25/c_1121541921.htm) and [online comments](http://www.cac.gov.cn/2017-08/25/c_1121541842.htm) to making individuals who host public accounts and moderate chat groups [liable](http://www.cac.gov.cn/2017-09/07/c_1121624269.htm) for content on the platforms.

Under the CAC, WeChat, along with other Chinese social media platforms, face much higher penalties than fines if they fail to moderate content. On April 9, 2018, the CAC [ordered](http://www.bjnews.com.cn/finance/2018/04/09/482425.html) all Chinese app stores to remove the four most popular news aggregation applications for weeks because they failed to “maintain the lawful order of information sharing.” A day later, authorities [demanded](http://www.xinhuanet.com/english/2018-04/10/c_137100672.htm) Toutiao, China’s top news aggregation website, and WeChat permanently shut down an account that featured parody and jokes due to “publishing vulgar and improper content.” In the same month, Tencent [suspended](https://www.thepaper.cn/newsDetail_forward_2070142) all video playing functions on WeChat and QQ if the URL of the video was an external link.

To handle increased government pressures, companies are investing more heavily in filtering technologies and human resources to moderate content. Global Times, a Chinese state media outlet affiliated with People’s Daily, [reported](http://www.globaltimes.cn/content/1098173.shtml) that tech companies are expanding their human censor team and developing artificial intelligence tools to review “trillions of posts, voice messages, photos and videos every day” to make sure their content is in line with laws and regulations. However, authorities still think that “these platforms are not fully performing their duties.”

In September 2016, Chinese authorities issued [new regulations](http://news.sohu.com/20160920/n468794222.shtml) that explicitly state that messages and comments on social media products like WeChat Moments can be collected and used as “electronic data” in legal proceedings. Martin Lau Chi-ping, a senior manager at Tencent, [said](http://www.scmp.com/tech/social-gadgets/article/2138249/tencent-profit-doubles-strong-smartphone-games-business) the following:

> “We are very concerned about user data security. It is top of our concerns… In a law enforcement situation, of course, any company has to comply with the regulations and laws within the country.”

Recently, WeChat users have been arrested for “[insulting police](http://d.youth.cn/sk/201702/t20170224_9165131.htm)” or “[threatening to blow up a government building](http://www.kejilie.com/ifeng/article/veqIby.html)” on Moments, which indicates that the feature may be subject to monitoring by the authorities or the company.

## Previous Examples of WeChat Image Filtering

In [previous](https://citizenlab.ca/2017/04/we-cant-chat-709-crackdown-discussions-blocked-on-weibo-and-wechat/) [Citizen Lab](https://citizenlab.ca/2017/04/we-cant-chat-709-crackdown-discussions-blocked-on-weibo-and-wechat/) [research](https://citizenlab.ca/2017/04/we-cant-chat-709-crackdown-discussions-blocked-on-weibo-and-wechat/), we showed that image censorship occurs in both WeChat’s chat function and WeChat Moments. Similar to keyword-based text filtering, censorship of images is only enabled for users with accounts registered to mainland China phone numbers. The filtering is also non-transparent in that no notice is given to a user if the image they have sent is blocked. Censorship of an image is concealed from the user who posted the censored image.

Figure 1 shows a user with an international account successfully posting a censored image: the image is visible to users with international accounts, but the post is hidden from users with China accounts.

[![](https://citizenlab.ca/wp-content/uploads/2018/08/f1-1024x603.png)](https://citizenlab.ca/wp-content/uploads/2018/08/f1.png)

Figure 1: Image censorship on WeChat moments. A user with a China account (on the left) attempts to send an [image](https://www.chrlawyers.hk/sites/default/files/33.png) related to the 709 Crackdown and is hidden from other China account’s Moments feed (on the right). The image is visible in the user’s own feed as well as to an international account (in the middle).

In January 2017, we [discovered](https://citizenlab.ca/2017/04/we-cant-chat-709-crackdown-discussions-blocked-on-weibo-and-wechat/) that a number of images related to the “[709 Crackdown](https://chinachange.org/tag/709-arrest-of-lawyers/)” (referring to a crackdown on human rights lawyers and their families in China) are blocked in group chat when using an account registered to a mainland China phone number. The censorship was found when we were performing keyword testing of news articles. When we copied and pasted the image accompanying certain news articles about the 709 Crackdown, the image itself was filtered. In subsequent sample testing, we found 58 images related to the event censored on Moments, most of which are infographics related to the 709 Crackdown, profile sketches of the affected lawyers and their relatives, or images of people holding the slogan “oppose torture, pay attention to Xie Yang” (“反对酷刑，关注谢阳”). See Figure 2 for an example of the image filtering.

[![](https://citizenlab.ca/wp-content/uploads/2018/08/f2-1024x906.png)](https://citizenlab.ca/wp-content/uploads/2018/08/f2.png)

Figure 2: Image censorship in a WeChat group Chat. A user with a China account (on the left) attempts to send an image of the cover of a report on the 709 Crackdown and is blocked.

We [documented](https://citizenlab.ca/2017/07/analyzing-censorship-of-the-death-of-liu-xiaobo-on-wechat-and-weibo/) similar instances of image censorship on WeChat following the death of Liu Xiaobo in July 2017. The scope of censorship was wider and more intensive compared to the case of the 709 Crackdown. Not only were images censored on WeChat’s group chat and Moments, but we also documented image filtering on WeChat’s one-to-one chat function for the first time (see Figure 3).

In the wake of Liu Xiaobo’s death, we again found that images blocked in one-to-one chat messages were also blocked on group chat and WeChat Moments. Images blocked in chat functions were always blocked on WeChat Moments. The greater attention to WeChat Moments and group chat may be due to the semi-public nature of the two features. Messages in these functions can reach a larger audience than one-to-one chat, potentially making these features subject to a higher level of scrutiny. However, the blocking of images on one-to-one chat shows an effort to restrict content across semi-public and private chat functions, demonstrating the sensitivity of Liu Xiaobo’s death.

[![](https://citizenlab.ca/wp-content/uploads/2018/08/f3-1024x877.png)](https://citizenlab.ca/wp-content/uploads/2018/08/f3.png)

Figure 3: Image censorship in a WeChat one-to-one chat. A user with an international account (on the left) attempts to send an image of a cartoon of an empty chair symbolizing Nobel Laureate Liu Xiaobo to a China account. The image is not received by the China account.

In both cases, our tests showed that an image on Moments is filtered according to that image’s content in a way that is tolerant to some modifications to the image; however, until this study it was unclear the algorithms used by WeChat to filter and which kinds of image modifications evaded filtering and which did not. In this report, we conduct a systematic analysis of WeChat’s filtering mechanisms to understand how WeChat implements image filtering. This understanding informs weaknesses in WeChat’s algorithms and techniques for evading WeChat’s image filtering.

> * 下一篇：[(CAN’T) PICTURE THIS: An Analysis of Image Filtering on WeChat Moments — Part 2](https://github.com/xitu/gold-miner/blob/master/TODO1/cant-picture-this-an-analysis-of-image-filtering-on-wechat-moments-2.md)

> 如果发现译文存在错误或其他需要改进的地方，欢迎到 [掘金翻译计划](https://github.com/xitu/gold-miner) 对译文进行修改并 PR，也可获得相应奖励积分。文章开头的 **本文永久链接** 即为本文在 GitHub 上的 MarkDown 链接。


---

> [掘金翻译计划](https://github.com/xitu/gold-miner) 是一个翻译优质互联网技术文章的社区，文章来源为 [掘金](https://juejin.im) 上的英文分享文章。内容覆盖 [Android](https://github.com/xitu/gold-miner#android)、[iOS](https://github.com/xitu/gold-miner#ios)、[前端](https://github.com/xitu/gold-miner#前端)、[后端](https://github.com/xitu/gold-miner#后端)、[区块链](https://github.com/xitu/gold-miner#区块链)、[产品](https://github.com/xitu/gold-miner#产品)、[设计](https://github.com/xitu/gold-miner#设计)、[人工智能](https://github.com/xitu/gold-miner#人工智能)等领域，想要查看更多优质译文请持续关注 [掘金翻译计划](https://github.com/xitu/gold-miner)、[官方微博](http://weibo.com/juejinfanyi)、[知乎专栏](https://zhuanlan.zhihu.com/juejinfanyi)。
