minor docs improvements (#3906)

* CLICKHOUSE-4063: less manual html @ index.md

* CLICKHOUSE-4063: recommend markdown="1" in README.md

* CLICKHOUSE-4003: manually purge custom.css for now

* CLICKHOUSE-4064: expand <details> before any print (including to pdf)

* CLICKHOUSE-3927: rearrange interfaces/formats.md a bit

* CLICKHOUSE-3306: add few http headers

* Remove copy-paste introduced in #3392

* Hopefully better chinese fonts #3392

* get rid of tabs @ custom.css

* Apply comments and patch from #3384

* Add jdbc.md to ToC and some translation, though it still looks badly incomplete

* minor punctuation

* Add some backlinks to official website from mirrors that just blindly take markdown sources

* Do not make fonts extra light

* find . -name '*.md' -type f | xargs -I{} perl -pi -e 's//g' {}

* find . -name '*.md' -type f | xargs -I{} perl -pi -e 's/ sql/g' {}

* Remove outdated stuff from roadmap.md

* Not so light font on front page too

* Refactor Chinese formats.md to match recent changes in other languages

* Update some links on front page

* Remove some outdated comment

* Add twitter link to front page

* More front page links tuning

* Add Amsterdam meetup link

* Smaller font to avoid second line

* Add Amsterdam link to README.md

* Proper docs nav translation

* Back to 300 font-weight except Chinese

* fix docs build

* Update Amsterdam link

* remove symlinks

* more zh punctuation

* apply lost comment by @zhang2014

* Apply comments by @zhang2014 from #3417

* Remove Beijing link

* rm incorrect symlink

* restore content of docs/zh/operations/table_engines/index.md

* CLICKHOUSE-3751: stem terms while searching docs

* CLICKHOUSE-3751: use English stemmer in non-English docs too

* CLICKHOUSE-4135 fix

* Remove past meetup link

* Add blog link to top nav

* Add ContentSquare article link

* Add form link to front page + refactor some texts

* couple markup fixes

* minor

* Introduce basic ODBC driver page in docs

* More verbose 3rd party libs disclaimer

* Put third-party stuff into a separate folder

* Separate third-party stuff in ToC too

* Update links

* Move stuff that is not really (only) a client library into a separate page

* Add clickhouse-hdfs-loader link

* Some introduction for "interfaces" section

* Rewrite tcp.md

* http_interface.md -> http.md

* fix link

* Remove unconvenient error for now

* try to guess anchor instead of failing

* remove symlink

* Remove outdated info from introduction

* remove ru roadmap.md

* replace ru roadmap.md with symlink

* Update roadmap.md

* lost file

* Title case in toc_en.yml

* Sync "Functions" ToC section with en

* Remove reference to pretty old ClickHouse release from docs

* couple lost symlinks in fa

* Close quote in proper place

* Rewrite en/getting_started/index.md

* Sync en<>ru getting_started/index.md

* minor changes

* Some gui.md refactoring

* Translate DataGrip section to ru

* Translate DataGrip section to zh

* Translate DataGrip section to fa

* Translate DBeaver section to fa

* Translate DBeaver section to zh

* Split third-party GUI to open-source and commercial

* Mention some RDBMS integrations + ad-hoc translation fixes

* Add rel="external nofollow" to outgoing links from docs

* Lost blank lines

* Fix class name

* More rel="external nofollow"

* Apply suggestions by @sundy-li

* Mobile version of front page improvements

* test

* test 2

* test 3

* Update LICENSE

* minor docs fix

* Highlight current article as suggested by @sundy-li

* fix link destination

* Introduce backup.md (only "en" for now)

* Mention INSERT+SELECT in backup.md

* Some improvements for replication.md

* Add backup.md to toc

* Mention clickhouse-backup tool

* Mention LightHouse in third-party GUI list

* Introduce interfaces/third-party/proxy.md

* Add clickhouse-bulk to proxy.md

* Major extension of integrations.md contents

* fix link target

* remove unneeded file

* better toc item name

* fix markdown

* better ru punctuation

* Add yet another possible backup approach

* Simplify copying permalinks to headers

* Support non-eng link anchors in docs + update some deps

* Generate anchors for single-page mode automatically

* Remove anchors to top of pages

* Remove anchors that nobody links to

* build fixes

* fix few links

* restore css

* fix some links

* restore gifs

* fix lost words

* more docs fixes

* docs fixes

* NULL anchor

* update urllib3 dependency

* more fixes

* Remove excessive content from print version

* Try short license again

* Back to long license for now

* Introduce anchor integrity checks for single-page docs

* Add --save-raw-single-page option to build.py (helps to debug incorrect anchors)

* fix kafka engine links

* fix one class of broken anchors

* fix some broken links

* Add https://github.com/hatarist/clickhouse-cli to third-party section (in gui.md for now, maybe will add cli.md later)

* fix one more class of links to nowhere

* less duplicate anchors

* get rid of weird anchors

* fix anchor

* fix link

* fix couple links

* rearrange integrations.md a bit + sync zh version

* Mention nagios plugin in other languages

* port summingmergetree.md fix to zh

* Make doc links to nowhere fatal

* additional check in markdown extension

* add option to skip pdf

docs/en/interfaces/third-party/integrations.md

@@ -19,6 +19,9 @@
 - Object storages
     - [S3](https://en.wikipedia.org/wiki/Amazon_S3)
         - [clickhouse-backup](https://github.com/AlexAkulov/clickhouse-backup)
+- Configuration management
+    - [puppet](https://puppet.com)
+        - [innogames/clickhouse](https://forge.puppet.com/innogames/clickhouse)
 - Monitoring
     - [Graphite](https://graphiteapp.org)
         - [graphouse](https://github.com/yandex/graphouse)
@@ -33,9 +36,6 @@
 - Logging
     - [fluentd](https://www.fluentd.org)
         - [loghouse](https://github.com/flant/loghouse) (for [Kubernetes](https://kubernetes.io))
-- Configuration management
-    - [puppet](https://puppet.com)
-        - [innogames/clickhouse](https://forge.puppet.com/innogames/clickhouse)
 
 ## Programming Language Ecosystems
 

docs/fa/interfaces/third-party/integrations.md

@@ -21,6 +21,9 @@
 - فروشگاه شی
     - [S3](https://en.wikipedia.org/wiki/Amazon_S3)
         - [clickhouse-backup](https://github.com/AlexAkulov/clickhouse-backup)
+- مدیریت تنظیمات
+    - [puppet](https://puppet.com)
+        - [innogames/clickhouse](https://forge.puppet.com/innogames/clickhouse)
 - نظارت بر
     - [Graphite](https://graphiteapp.org)
         - [graphouse](https://github.com/yandex/graphouse)
@@ -30,12 +33,11 @@
     - [Prometheus](https://prometheus.io/)
         - [clickhouse_exporter](https://github.com/f1yegor/clickhouse_exporter)
         - [PromHouse](https://github.com/Percona-Lab/PromHouse)
+    - [Nagios](https://www.nagios.org/)
+        - [check_clickhouse](https://github.com/exogroup/check_clickhouse/)
 - ثبت نام
     - [fluentd](https://www.fluentd.org)
         - [loghouse](https://github.com/flant/loghouse) (برای [Kubernetes](https://kubernetes.io))
-- مدیریت تنظیمات
-    - [puppet](https://puppet.com)
-        - [innogames/clickhouse](https://forge.puppet.com/innogames/clickhouse)
 
 ## اکوسیستم زبان برنامه نویسی
 

docs/ru/interfaces/third-party/integrations.md

@@ -18,6 +18,9 @@
 - Хранилища объектов
     - [S3](https://en.wikipedia.org/wiki/Amazon_S3)
         - [clickhouse-backup](https://github.com/AlexAkulov/clickhouse-backup)
+- Системы управления конфигурацией
+    - [puppet](https://puppet.com)
+        - [innogames/clickhouse](https://forge.puppet.com/innogames/clickhouse)
 - Мониторинг
     - [Graphite](https://graphiteapp.org)
         - [graphouse](https://github.com/yandex/graphouse)
@@ -27,12 +30,11 @@
     - [Prometheus](https://prometheus.io/)
         - [clickhouse_exporter](https://github.com/f1yegor/clickhouse_exporter)
         - [PromHouse](https://github.com/Percona-Lab/PromHouse)
+    - [Nagios](https://www.nagios.org/)
+        - [check_clickhouse](https://github.com/exogroup/check_clickhouse/)
 - Логирование
     - [fluentd](https://www.fluentd.org)
         - [loghouse](https://github.com/flant/loghouse) (для [Kubernetes](https://kubernetes.io))
-- Системы управления конфигурацией
-    - [puppet](https://puppet.com)
-        - [innogames/clickhouse](https://forge.puppet.com/innogames/clickhouse)
 
 ## Экосистемы вокруг языков программирования
 

docs/tools/build.py

@@ -178,11 +178,12 @@ def build_single_page_version(lang, args, cfg):
                     single_page_output_path
                 )
 
-                single_page_index_html = os.path.abspath(os.path.join(single_page_output_path, 'index.html'))
-                single_page_pdf = single_page_index_html.replace('index.html', 'clickhouse_%s.pdf' % lang)
-                create_pdf_command = ['wkhtmltopdf', '--print-media-type', single_page_index_html, single_page_pdf]
-                logging.debug(' '.join(create_pdf_command))
-                subprocess.check_call(' '.join(create_pdf_command), shell=True)
+                if not args.skip_pdf:
+                    single_page_index_html = os.path.abspath(os.path.join(single_page_output_path, 'index.html'))
+                    single_page_pdf = single_page_index_html.replace('index.html', 'clickhouse_%s.pdf' % lang)
+                    create_pdf_command = ['wkhtmltopdf', '--print-media-type', single_page_index_html, single_page_pdf]
+                    logging.debug(' '.join(create_pdf_command))
+                    subprocess.check_call(' '.join(create_pdf_command), shell=True)
 
                 with temp_dir() as test_dir:
                     cfg.load_dict({
@@ -229,6 +230,7 @@ if __name__ == '__main__':
     arg_parser.add_argument('--theme-dir', default='mkdocs-material-theme')
     arg_parser.add_argument('--output-dir', default='build')
     arg_parser.add_argument('--skip-single-page', action='store_true')
+    arg_parser.add_argument('--skip-pdf', action='store_true')
     arg_parser.add_argument('--save-raw-single-page', type=str)
     arg_parser.add_argument('--verbose', action='store_true')
 

docs/tools/mdx_clickhouse.py

@@ -43,9 +43,10 @@ class ClickHouseLinkPattern(ClickHouseLinkMixin, markdown.inlinepatterns.LinkPat
 
 class ClickHousePreprocessor(markdown.util.Processor):
     def run(self, lines):
-        for line in lines:
-            if '<!--hide-->' not in line:
-                yield line
+        if os.getenv('QLOUD_TOKEN'):
+            for line in lines:
+                if '<!--hide-->' not in line:
+                    yield line
 
 
 class ClickHouseMarkdown(markdown.extensions.Extension):

docs/tools/test.py

@@ -34,6 +34,7 @@ def test_single_page(input_path, lang):
             logging.error('Found %d duplicate anchor points' % duplicate_anchor_points)
         if links_to_nowhere:
             logging.error('Found %d links to nowhere' % links_to_nowhere)
+            sys.exit(10)
 
         assert len(anchor_points) > 10, 'Html parsing is probably broken'
                 

docs/zh/interfaces/third-party/integrations.md

@@ -18,6 +18,9 @@
 - 对象存储
     - [S3](https://en.wikipedia.org/wiki/Amazon_S3)
         - [clickhouse-backup](https://github.com/AlexAkulov/clickhouse-backup)
+- 配置管理
+    - [puppet](https://puppet.com)
+        - [innogames/clickhouse](https://forge.puppet.com/innogames/clickhouse)
 - 监控
     - [Graphite](https://graphiteapp.org)
         - [graphouse](https://github.com/yandex/graphouse)
@@ -27,6 +30,8 @@
     - [Prometheus](https://prometheus.io/)
         - [clickhouse_exporter](https://github.com/f1yegor/clickhouse_exporter)
         - [PromHouse](https://github.com/Percona-Lab/PromHouse)
+    - [Nagios](https://www.nagios.org/)
+        - [check_clickhouse](https://github.com/exogroup/check_clickhouse/)
 - 记录
     - [fluentd](https://www.fluentd.org)
         - [loghouse](https://github.com/flant/loghouse) (对于 [Kubernetes](https://kubernetes.io))

docs/zh/operations/table_engines/summingmergetree.md

@@ -13,7 +13,7 @@ CREATE TABLE [IF NOT EXISTS] [db.]table_name [ON CLUSTER cluster]
     name1 [type1] [DEFAULT|MATERIALIZED|ALIAS expr1],
     name2 [type2] [DEFAULT|MATERIALIZED|ALIAS expr2],
     ...
-) ENGINE = MergeTree()
+) ENGINE = SummingMergeTree()
 [PARTITION BY expr]
 [ORDER BY expr]
 [SAMPLE BY expr]