Add a script to add versions to sync configuration

Add a new python script that can be used to add a new version to
the configuration of a component.
This script can be used by various projects as part of their
release process to keep the version on the website up-to-date.
Signed-off-by: NAndrea Frittoli <andrea.frittoli@gmail.com>

sync/README.md

-# Sync
+# Sync and Version
 
-This directory includes a helper script for synchronizing contents
-from specified Tekton repositories to this repository.
+This directory includes scripts to manage multiple versions of
+documentation from the different Tekton projects.
+The `sync` script pulls content from a project, the `versions`
+script adds a new version to the `sync` config.
 
-To run this script locally, set up a Python 3 environment with appropriate
-Google Cloud Platform credentials, and execute the following command:
+## `sync`
 
-**Note:** This is a [link](../DEVELOPMENT.md) for the steps to run the entire website locally.
+This `sync` script allows synchronizing contents from specified Tekton
+repositories to this repository.
+
+To run this script locally, set up a Python 3 environment and execute
+the script:
 
 ```bash
 python3 -m venv tkn_web_env
 source tkn_web_env/bin/activate    
 pip3 install -r requirements.txt
-python3 sync/sync.py
+./sync/sync.py
 ```
 
-## Usage
+**Note:** Follow [these steps](../DEVELOPMENT.md) to run the entire website locally.
+
+### Usage
 
 ```bash
        USAGE: sync.py [flags]
@@ -28,7 +35,7 @@ sync.py:
 Try --helpfull to get a list of all flags.
 ```
 
-## Configuring Directories
+### Configuring Directories
 
 The config directory should include the configuration for syncing/curating contents from
 specific Tekton repositories.
@@ -42,7 +49,7 @@ The YAML files here are used by the scripts in `../sync`.
 The yaml sync file requires the following schema
 
 ```yaml
-# Each YAML file under sync/ configures how helper/helper.py synchronizes
+# Each YAML file under sync/ configures how sync/sync.py synchronizes
 # contents of various versions from its source of truth (usually a GitHub
 # repository of a Tekton component, such as tektoncd/pipelines) to
 # content/ (for the lastest version) and vault/ (for earlier versions).
@@ -56,10 +63,17 @@ displayOrder: 0
 repository: https://github.com/tektoncd/foobar
 # The directory in the GitHub repository where contents reside.
 docDirectory: docs
+# The link to the GitHub tag page.
+archive: https://github.com/tektoncd/foobar/tags
 # The tags (versions) of contents to sync.
 # Note that sync.py and related script reads tags in the order specified in
 # the following list; the first entry in tags will automatically become the
 # latest version of contents.
+# To add a new version, append to the list as below
+#- name: v0.8.2
+#  displayName: v0.8.x
+#  files:
+#  - myfiles.md: myfiles.md
 tags:
   # The name of the tag in the GitHub repository.
 - name: master
@@ -69,17 +83,10 @@ tags:
   # Key-value pairs of files to sync, where the key is the original filename
   # and the value is the new filename.
   files:
-  - foo.md : bar.md
-# To add a new version, append to the list as below
-#- name: v0.8.2
-#  displayName: v0.8.x
-#  files:
-#  - myfiles.md: myfiles.md
-# The link to the GitHub tag page.
-archive: https://github.com/tektoncd/foobar/tags
+    foo.md : bar.md
 ```
 
-## Mental Model
+### Mental Model
 
 This is a quick diagram that will help you develop a mental model on how the sync works.
 
@@ -94,4 +101,37 @@ To build the docker file
 ```bash
 # You must cd into the correct directory to build the image
 docker build -t tekton/web sync/.
-```
\ No newline at end of file
+```
+
+## `versions`
+
+The `versions` script can be used to add a new version to the `sync` configurations.
+It was designed to be integrated in the release process of Tekton projects.
+
+To run this script locally, set up a Python 3 environment and execute
+the script:
+
+```bash
+python3 -m venv tkn_web_env
+source tkn_web_env/bin/activate    
+pip3 install -r requirements.txt
+./sync/versions.py -p <project-name> -r <version-name>
+```
+
+**Note:** Follow [these steps](../DEVELOPMENT.md) to run the entire website locally.
+
+## Usage
+
+```bash
+USAGE: ./sync/versions.py [flags]
+
+flags:
+
+./sync/versions.py:
+  -p,--project: Name of the component
+    (default: 'pipeline')
+  -r,--version: Version of the component
+  -c,--config: Config directory
+
+Try --helpfull to get a list of all flags.
+```

sync/config/cli.yaml

@@ -4,7 +4,7 @@
 # content/ (for the lastest version) and vault/ (for earlier versions).
 
 # The name of the component.
-# helper.py will use this value to build directories in content/ and vault/.
+# sync.py will use this value to build directories in content/ and vault/.
 component: CLI
 # The order of the component.
 displayOrder: 2
@@ -12,10 +12,17 @@ displayOrder: 2
 repository: https://github.com/tektoncd/cli
 # The directory in the GitHub repository where contents reside.
 docDirectory: docs
+# The link to the GitHub tag page.
+archive: https://github.com/tektoncd/cli/tags
 # The tags (versions) of contents to sync.
 # Note that helper.py and related script reads tags in the order specified in
 # the following list; the first entry in tags will automatically become the
 # latest version of contents.
+# To add a new version, append to the list as below
+#- name: v0.8.2
+#  displayName: v0.8.x
+#  files:
+#  - myfiles.md: myfiles.md
 tags:
   # The name of the tag in the GitHub repository.
 - name: master
@@ -26,10 +33,3 @@ tags:
   # and the value is the new filename.
   files:
     README.md: _index.md
-# To add a new version, append to the list as below
-#- name: v0.8.2
-#  displayName: v0.8.x
-#  files:
-#  - myfiles.md: myfiles.md
-# The link to the GitHub tag page.
-archive: https://github.com/tektoncd/cli/tags

sync/config/pipelines.yaml

@@ -12,15 +12,21 @@ displayOrder: 0
 repository: https://github.com/tektoncd/pipeline
 # The directory in the GitHub repository where contents reside.
 docDirectory: docs
+# The link to the GitHub tag page.
+archive: https://github.com/tektoncd/pipeline/tags
 # The tags (versions) of contents to sync.
-# Note that helper.py and related script reads tags in the order specified in
+# Note that sync.py and related script reads tags in the order specified in
 # the following list; the first entry in tags will automatically become the
 # latest version of contents.
+# To add a new version, append to the list as below
+#- name: v0.8.2
+#  displayName: v0.8.x
+#  files:
+#  - myfiles.md: myfiles.md
 tags:
-  # The name of the tag in the GitHub repository.
 - name: master
   # The name to display on tekton.dev.
-  # helper.py will use this value in the version switcher and other places.
+  # sync.py will use this value in the version switcher and other places.
   displayName: master
   # Key-value pairs of files to sync, where the key is the original filename
   # and the value is the new filename.
@@ -43,10 +49,3 @@ tags:
     variables.md: variables.md
     workspaces.md: workspaces.md
     events.md: events.md
-# To add a new version, append to the list as below
-#- name: v0.8.2
-#  displayName: v0.8.x
-#  files:
-#  - myfiles.md: myfiles.md
-# The link to the GitHub tag page.
-archive: https://github.com/tektoncd/pipeline/tags

sync/config/triggers.yaml

@@ -4,7 +4,7 @@
 # content/ (for the lastest version) and vault/ (for earlier versions).
 
 # The name of the component.
-# helper.py will use this value to build directories in content/ and vault/.
+# sync.py will use this value to build directories in content/ and vault/.
 component: Triggers
 # The order of the component.
 displayOrder: 1
@@ -12,6 +12,8 @@ displayOrder: 1
 repository: https://github.com/tektoncd/triggers
 # The directory in the GitHub repository where contents reside.
 docDirectory: docs
+# The link to the GitHub tag page.
+archive: https://github.com/tektoncd/triggers/tags
 # The tags (versions) of contents to sync.
 # Note that helper.py and related script reads tags in the order specified in
 # the following list; the first entry in tags will automatically become the
@@ -35,5 +37,3 @@ tags:
     cel_expressions.md: cel_expressions.md
     getting-started/create-ingress.yaml: create-ingress.yaml
     getting-started/create-webhook.yaml: create-webhook.yaml
-# The link to the GitHub tag page.
-archive: https://github.com/tektoncd/triggers/tags

sync/requirements.txt

 wget==3.2
-PyYAML==5.1.2
+ruamel.yaml==0.16.12
 google-cloud-storage==1.23.0
 Jinja2==2.11.1
 google-auth==1.14.0
@@ -8,4 +8,4 @@ urlopen==1.0.0
 markdown==3.1.1
 lxml==4.5.2
 coverage==5.3
-flake8==3.8.3
\ No newline at end of file
+flake8==3.8.3

sync/sync.py

+#!/usr/bin/env python
+
+# Copyright 2020 The Tekton Authors
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
 # This script helps synchronize contents from their respective sources of
 # truth (usually GitHub repositories of each Tekton
 # components, such as tektoncd/pipelines) to tektoncd/website.
@@ -5,22 +21,23 @@
 import fileinput
 import json
 import logging
-import markdown
 import os
 import os.path
+from pathlib import Path
 import re
 import shutil
+from urllib.request import urlopen
+from urllib.error import HTTPError
+from urllib.error import URLError
 import wget
-import yaml
 
 from absl import app
 from absl import flags
+import markdown
 from jinja2 import Environment
 from jinja2 import FileSystemLoader
 from lxml import etree
-from urllib.request import urlopen
-from urllib.error import HTTPError
-from urllib.error import URLError
+from ruamel.yaml import YAML
 
 
 FLAGS = flags.FLAGS
@@ -30,7 +47,7 @@ FLAGS = flags.FLAGS
 # If there is a conflict, we'll get an error at import time.
 flags.DEFINE_string(
     'config',
-    os.path.dirname(os.path.abspath(__file__)) + '/config',
+    os.path.join(os.path.dirname(os.path.abspath(__file__)), 'config'),
     'Config directory', short_name='c')
 
 CONTENT_DIR = './content/en/docs'
@@ -39,9 +56,6 @@ TEMPLATE_DIR = './templates'
 VAULT_DIR = './content/en/vault'
 BUCKET_NAME = 'tekton-website-assets'
 
-GCP_NETLIFY_ENV_CRED = os.environ.get('GCP_CREDENTIAL_JSON')
-GCP_PROJECT = os.environ.get('GCP_PROJECT')
-
 LINKS_RE = r'\[([^\]]*)\]\((?!.*://|/)([^)]*).md(#[^)]*)?\)'
 
 jinja_env = Environment(loader=FileSystemLoader(TEMPLATE_DIR))
@@ -183,22 +197,32 @@ def get_files(path, file_type):
     return file_list
 
 
-def yaml_files_to_dic_list(files):
-    """ return a list of yaml files to a sorted
-     list based on a field called displayOrder """
-
+def load_config(files):
+    """ return a list of yaml files sorted based on a field called displayOrder """
+    yaml = YAML()
     dic_list = []
 
     for file in files:
         with open(file, 'r') as text:
             # get the paths from the config file
-            dic_list.append(yaml.load(text, Loader=yaml.FullLoader))
+            dic_list.append({
+                "filename": file,
+                "content": yaml.load(text)
+            })
 
-    dic_list.sort(key=lambda x: x['displayOrder'])
+    dic_list.sort(key=lambda x: x['content']['displayOrder'])
 
     return dic_list
 
 
+def save_config(config):
+    """ save config files back to yaml """
+    yaml = YAML()
+    for c in config:
+        out = Path(c['filename'])
+        yaml.dump(c['content'], out)
+
+
 def get_tags(sync_config):
     """ return a list of tags with, there name, and displayName """
     tags = []
@@ -239,13 +263,14 @@ def sync(argv):
     """ fetch all the files and sync it to the website """
     # get the path of the urls needed
     config_files = get_files(f'{FLAGS.config}', ".yaml")
-    config = yaml_files_to_dic_list(config_files)
+    config = [x["content"] for x in load_config(config_files)]
     # download resources
     download_resources_to_project(config)
+    versions = get_versions(config)
     # create version switcher script
-    create_resource(JS_ASSET_DIR, "version-switcher.js", get_versions(config))
+    create_resource(JS_ASSET_DIR, "version-switcher.js", versions)
     # create index for vault
-    create_resource(VAULT_DIR, "_index.md", get_versions(config))
+    create_resource(VAULT_DIR, "_index.md", versions)
 
 
 if __name__ == '__main__':

sync/test_sync.py

+#!/usr/bin/env python
+
+# Copyright 2020 The Tekton Authors
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
 import unittest
 import tempfile
 import shutil
@@ -11,7 +27,8 @@ from sync import is_ref
 from sync import remove_ending_forward_slash
 from sync import get_tags
 from sync import download_files
-from sync import yaml_files_to_dic_list
+from sync import load_config
+from sync import save_config
 from sync import get_files
 
 
@@ -90,7 +107,7 @@ class TestSync(unittest.TestCase):
         )
         shutil.rmtree(dirpath)
 
-    def test_yaml_files_to_dic_list(self):
+    def test_load_save_config(self):
         """ convert a list of files into a list of dictionaries """
         # create a tmp file with yaml txt
         text = "{displayOrder: 1}"
@@ -101,11 +118,20 @@ class TestSync(unittest.TestCase):
             tmp_name = tmp.name
             tmp.write(text.strip().encode())
 
-        expected = [{'displayOrder': 1}]
-        actual = yaml_files_to_dic_list([tmp_name])
-        self.read_and_delete_file(tmp_name)
+        expected = [{'content': {'displayOrder': 1},
+                     'filename': tmp_name}]
+        actual = load_config([tmp_name])
         self.assertEqual(actual, expected)
 
+        mod_config = actual
+        mod_config[0]['content']['displayOrder'] = 2
+        expected = [{'content': {'displayOrder': 2},
+                     'filename': tmp_name}]
+        save_config(mod_config)
+        actual = load_config([tmp_name])
+        self.assertEqual(actual, expected)
+        self.read_and_delete_file(tmp_name)
+
     def test_get_files(self):
         """ create a list of files within a
         directory that contain a valid extension"""

sync/test_versions.py

+#!/usr/bin/env python
+
+# Copyright 2020 The Tekton Authors
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import unittest
+
+from ruamel.yaml import YAML
+
+import versions
+
+test_config_string = """
+# This is a test config
+component: test
+displayOrder: 0
+repository: https://foo.bar
+docDirectory: docs
+archive: https://foo.bar/tags
+tags:
+- name: foo
+  displayName: foo
+  files:
+    FOO.md: _index.md
+    bar.md: bar.md
+- name: bar
+  displayName: bar
+  files:
+    FOO.md: _index.md
+    bar.md: bar.md
+"""
+
+test_config_string_new = """
+# This is a test config
+component: test
+displayOrder: 0
+repository: https://foo.bar
+docDirectory: docs
+archive: https://foo.bar/tags
+tags:
+- name: new
+  displayName: new
+  files:
+    FOO.md: _index.md
+    bar.md: bar.md
+- name: foo
+  displayName: foo
+  files:
+    FOO.md: _index.md
+    bar.md: bar.md
+- name: bar
+  displayName: bar
+  files:
+    FOO.md: _index.md
+    bar.md: bar.md
+"""
+
+yaml = YAML()
+test_config = [{
+    'filename': 'foo.md',
+    'content': yaml.load(test_config_string)
+}]
+test_config_new = [{
+    'filename': 'foo.md',
+    'content': yaml.load(test_config_string_new)
+}]
+
+
+class TestAddVersion(unittest.TestCase):
+
+    def test_add_version(self):
+        expected = test_config_new
+        actual, updated = versions.add_version(test_config, 'test', 'new')
+        self.assertTrue(updated)
+        self.assertEqual(actual, expected)
+
+    def test_add_version_missing_component(self):
+        expected = test_config
+        actual, updated = versions.add_version(test_config, 'missing', 'new')
+        self.assertFalse(updated)
+        self.assertEqual(actual, expected)
+

sync/versions.py

+#!/usr/bin/env python
+
+# Copyright 2020 The Tekton Authors
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import copy
+import logging
+import sys
+
+from absl import app
+from absl import flags
+
+import sync
+
+FLAGS = flags.FLAGS
+
+flags.DEFINE_string(
+    'project', 'pipeline', 'Name of the component', short_name='p')
+flags.DEFINE_string(
+    'version', None, 'Version of the component', short_name='r')
+
+def add_version(config, component, version):
+    """ add a new version for the component in the config """
+    updated = False
+    for idx, c in enumerate(config):
+        if c['content']['repository'].endswith(f'/{component}'):
+            updated = True
+            tags = c['content']['tags']
+            new_tag = copy.deepcopy(tags[0])
+            new_tag['name'] = version
+            new_tag['displayName'] = version
+            config[idx]['content']['tags'] = [new_tag]
+            config[idx]['content']['tags'].extend(tags)
+    return config, updated
+
+
+def main(argv):
+    """ add a new version to a component """
+    # load the configs
+    config_files = sync.get_files(f'{FLAGS.config}', ".yaml")
+    config = sync.load_config(config_files)
+    # add a version to the config
+    new_config, updated = add_version(config, FLAGS.project, FLAGS.version)
+    if updated:
+        sync.save_config(new_config)
+    else:
+        logging.error(f'Could not find any config file for {FLAGS.project} to update')
+        sys.exit(1)
+
+
+if __name__ == '__main__':
+    app.run(main)