From d3886f9d41a90f68578e70985caac6afbfe12747 Mon Sep 17 00:00:00 2001 From: Kamil Trzcinski Date: Wed, 16 Sep 2015 10:58:33 +0200 Subject: [PATCH] Added migration docs and updated installation documentation --- doc/README.md | 1 + doc/migrate/README.md | 0 doc/migrate_ci_to_ce/README.md | 244 +++++++++++++++++++++++++++++++++ doc/update/7.14-to-8.0.md | 2 +- lib/support/nginx/gitlab_ci | 29 ++++ 5 files changed, 275 insertions(+), 1 deletion(-) delete mode 100644 doc/migrate/README.md create mode 100644 doc/migrate_ci_to_ce/README.md create mode 100644 lib/support/nginx/gitlab_ci diff --git a/doc/README.md b/doc/README.md index 632ddefeb2c..f5f1f56b1e2 100644 --- a/doc/README.md +++ b/doc/README.md @@ -47,6 +47,7 @@ - [Update](update/README.md) Update guides to upgrade your installation. - [Welcome message](customization/welcome_message.md) Add a custom welcome message to the sign-in page. - [Reply by email](reply_by_email/README.md) Allow users to comment on issues and merge requests by replying to notification emails. +- [Migrate GitLab CI to CE/EE](migrate_ci_to_ce/README.md) Follow this guide to migrate your existing GitLab CI data to GitLab CE/EE. ### Administrator documentation diff --git a/doc/migrate/README.md b/doc/migrate/README.md deleted file mode 100644 index e69de29bb2d..00000000000 diff --git a/doc/migrate_ci_to_ce/README.md b/doc/migrate_ci_to_ce/README.md new file mode 100644 index 00000000000..262346e8ae4 --- /dev/null +++ b/doc/migrate_ci_to_ce/README.md @@ -0,0 +1,244 @@ +## Migrate GitLab CI to GitLab CE/EE + +## Notice + +**You need to have working GitLab CI 7.14 to perform migration. +The older versions are not supported and will most likely break migration procedure.** + +This migration can't be done online and takes significant amount of time. +Make sure to plan it ahead. + +If you are running older version please follow the upgrade guide first: +https://gitlab.com/gitlab-org/gitlab-ci/blob/master/doc/update/7.13-to-7.14.md + +The migration is done in two parts: +1. **[CI]** You will be making a changes to GitLab CI instance. +1. **[CE]** You will be making a changes to GitLab CE/EE instance. + +### 1. Stop CI server [CI] + + sudo service gitlab_ci stop + +### 2. Backup [CI] + +**The migration procedure is database breaking. +You need to create backup if you still want to access CI data in case of failure.** + +```bash +cd /home/gitlab_ci/gitlab-ci +sudo -u gitlab_ci -H bundle exec backup:create RAILS_ENV=production +``` + +### 3. Prepare GitLab CI database to migration [CI] + +Copy and paste the command in terminal to rename all tables. +This also breaks your database structure disallowing you to use it anymore. + + cat < gitlab_ci.sql + +#### b. Dump PostgreSQL + + pg_dump -h DB_HOSTNAME -U DB_USERNAME -p DB_PORT --data-only GITLAB_CI_DATABASE -t "ci_*" > gitlab_ci.sql + +#### c. Dump MySQL and migrate to PostgreSQL + + # Dump existing MySQL database first + mysqldump --default-character-set=utf8 --compatible=postgresql --complete-insert \ + --host=DB_USERNAME --port=DB_PORT --user=DB_HOSTNAME -p + GITLAB_CI_DATABASE \ + ci_application_settings ci_builds ci_commits ci_events ci_jobs ci_projects \ + ci_runner_projects ci_runners ci_services ci_tags ci_taggings ci_trigger_requests \ + ci_triggers ci_variables ci_web_hooks > gitlab_ci.sql.tmp + + # Convert database to be compatible with PostgreSQL + git clone https://github.com/gitlabhq/mysql-postgresql-converter.git -b gitlab + python mysql-postgresql-converter/db_converter.py gitlab_ci.sql.tmp gitlab_ci.sql.tmp2 + ed -s gitlab_ci.sql.tmp2 < mysql-postgresql-converter/move_drop_indexes.ed + + # Filter to only include INSERT statements + grep "^\(START\|SET\|INSERT\|COMMIT\)" gitlab_ci.sql.tmp2 > gitlab_ci.sql + +### 5. Make sure that your GitLab CE/EE is 8.0 [CE] + +Please verify that you use GitLab CE/EE 8.0. +If not, please follow update guide: https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/update/7.14-to-8.0.md + +### 6. Stop GitLab CE/EE [CE] + +Before you can migrate actual data you need to stop GitLab CE/EE first. + + sudo service gitlab stop + +### 7. Backup GitLab CE/EE [CE] + +This migration poses a **significant risk** of breaking your GitLab CE/EE. +You should create a backup before doing it. + + cd /home/git/gitlab + sudo -u git -H bundle exec rake gitlab:backup:create RAILS_ENV=production + +### 8. Copy secret tokens [CE] + +The `secrets.yml` file stores encryption keys for secure variables. + +You need to copy the content of `config/secrets.yml` to the same file in GitLab CE. + + sudo cp /home/gitlab_ci/gitlab-ci/config/secrets.yml /home/git/gitlab/config/secrets.yml + sudo chown git:git /home/git/gitlab/config/secrets.yml + sudo chown 0600 /home/git/gitlab/config/secrets.yml + +### 9. Copy build logs [CE] + +You need to copy the contents of `builds/` to the same directory in GitLab CE/EE. + + sudo rsync -av /home/gitlab_ci/gitlab-ci/builds /home/git/gitlab/builds + sudo chown -R git:git /home/git/gitlab/builds + +The build traces are usually quite big so it will take a significant amount of time. + +### 10. Import GitLab CI database [CE] + +The one of the last steps is to import existing GitLab CI database. + + sudo mv /home/gitlab_ci/gitlab-ci/gitlab_ci.sql /home/git/gitlab/gitlab_ci.sql + sudo chown git:git /home/git/gitlab/gitlab_ci.sql + sudo -u git -H bundle exec rake ci:migrate CI_DUMP=/home/git/gitlab/gitlab_ci.sql RAILS_ENV=production + +This will take a significant amount of time. The GitLab CE/EE task does: +1. Deletes data from all existing CI tables +1. Import database data +1. Fixes database auto increments +1. Fixes tags assigned to Builds and Runners +1. Fixes services used by CI + +### 11. Start GitLab [CE] + + sudo service gitlab start + +### 12. Update nginx [CI] + +Now get back to GitLab CI and update **Nginx** configuration in order to: +1. Have all existing runners able to communicate with GitLab. +1. Have GitLab able send build triggers to CI address specified in Project's settings -> Services -> GitLab CI. + +You need to edit `/etc/nginx/sites-available/gitlab_ci` and paste: + + # GITLAB CI + server { + listen 80 default_server; # e.g., listen 192.168.1.1:80; + server_name YOUR_CI_SERVER_FQDN; # e.g., server_name source.example.com; + + access_log /var/log/nginx/gitlab_ci_access.log; + error_log /var/log/nginx/gitlab_ci_error.log; + + # expose API to fix runners + location /api { + proxy_read_timeout 300; + proxy_connect_timeout 300; + proxy_redirect off; + proxy_set_header X-Real-IP $remote_addr; + + # You need to specify your DNS servers that are able to resolve YOUR_GITLAB_SERVER_FQDN + resolver 8.8.8.8 8.8.4.4; + proxy_pass $scheme://YOUR_GITLAB_SERVER_FQDN/ci$request_uri; + } + + # redirect all other CI requests + location / { + return 301 $scheme://YOUR_GITLAB_SERVER_FQDN/ci$request_uri; + } + + # adjust this to match the largest build log your runners might submit, + # set to 0 to disable limit + client_max_body_size 10m; + } + +Make sure to fill the blanks to match your setup: +1. **YOUR_CI_SERVER_FQDN**: The existing public facing address of GitLab CI, eg. ci.gitlab.com. +1. **YOUR_GITLAB_SERVER_FQDN**: The public facing address of GitLab CE/EE, eg. gitlab.com. + +**Make sure to not remove the `/ci$request_uri`. This is required to properly forward the requests.** + +## Check your configuration + + sudo nginx -t + +## Restart nginx + + sudo /etc/init.d/nginx restart + +### Done! + +If everything went OK you should be able to access all your GitLab CI data by pointing your browser to: +https://gitlab.example.com/ci/. + +The GitLab CI should also work when using the previous address, redirecting you to the GitLab CE/EE. + +**Enjoy!** diff --git a/doc/update/7.14-to-8.0.md b/doc/update/7.14-to-8.0.md index b81b6a59980..59415e98782 100644 --- a/doc/update/7.14-to-8.0.md +++ b/doc/update/7.14-to-8.0.md @@ -155,7 +155,7 @@ git diff origin/7-14-stable:lib/support/nginx/gitlab origin/8-0-stable:lib/suppo ### 9. Migrate GitLab CI to GitLab CE/EE Now, GitLab CE and EE has CI integrated. However, migrations don't happen automatically and you need to do it manually. -Please follow the following guide [to migrate](../migrate/README.md) your GitLab CI instance to GitLab CE/EE. +Please follow the following guide [to migrate](../migrate_ci_to_ce/README.md) your GitLab CI instance to GitLab CE/EE. ### 10. Start application diff --git a/lib/support/nginx/gitlab_ci b/lib/support/nginx/gitlab_ci new file mode 100644 index 00000000000..bf05edfd780 --- /dev/null +++ b/lib/support/nginx/gitlab_ci @@ -0,0 +1,29 @@ +# GITLAB CI +server { + listen 80 default_server; # e.g., listen 192.168.1.1:80; + server_name YOUR_CI_SERVER_FQDN; # e.g., server_name source.example.com; + + access_log /var/log/nginx/gitlab_ci_access.log; + error_log /var/log/nginx/gitlab_ci_error.log; + + # expose API to fix runners + location /api { + proxy_read_timeout 300; + proxy_connect_timeout 300; + proxy_redirect off; + proxy_set_header X-Real-IP $remote_addr; + + # You need to specify your DNS servers that are able to resolve YOUR_GITLAB_SERVER_FQDN + resolver 8.8.8.8 8.8.4.4; + proxy_pass $scheme://YOUR_GITLAB_SERVER_FQDN/ci$request_uri; + } + + # redirect all other CI requests + location / { + return 301 $scheme://YOUR_GITLAB_SERVER_FQDN/ci$request_uri; + } + + # adjust this to match the largest build log your runners might submit, + # set to 0 to disable limit + client_max_body_size 10m; +} \ No newline at end of file -- GitLab