diff --git a/concourse/README.md b/concourse/README.md index ac043db2600892b08660a309aa18d4311700d29e..12da0080c25edb5d84bd5255a44e8a8410ac8592 100644 --- a/concourse/README.md +++ b/concourse/README.md @@ -1,4 +1,19 @@ -# Concourse Directory Contract +## Concourse pipeline configuration and scripts + +This directory contains pipelines, task files, and scripts to run various sets +of builds and tests on +[our Concourse continuous integration server](http://gpdb.ci.pivotalci.info/). + +[Learn more about Concourse overall](http://concourse.ci/) + +We use Concourse because: + +- its builds and test runs are on generic container images, and thus repeatable + and reproducible if we encounter test failures +- its configuration is fully encoded and repeatable if necessary +- its interface encourages us all to react to test failures + +## Concourse Directory Contract ### Directory Structure The Concourse directory should contain this README and three sub-directories only: @@ -7,32 +22,36 @@ The Concourse directory should contain this README and three sub-directories onl * tasks * scripts -##### Pipelines Directory -There should be seven pipelines in this directory: +#### Pipelines Directory +There should be just a handful of pipelines in this directory: -* `pipeline.yml` the pipeline that compiles, tests, and produces installers from the master branch of gpdb. -* `dev_generate_installer.yml` which compiles and generates an installer for the given source and saves it to a dev bucket. +* `pipeline.yml` the pipeline that compiles, tests, and produces installers + from the master branch of gpdb. +* `dev_generate_installer.yml` which compiles and generates an installer for + the given source and saves it to a dev bucket. +* `gpcloud_pipeline.yml` run gpcloud tests against developers specified branch. * `pr_pipeline.yml` which compiles and tests pull requests. * `concourse-upgrade.yml` which uses Concourse to upgrade itself. -* `pipeline_tinc.yml` run TINC tests against gpdb master branch. -* `gpcloud_pipeline.yml` run gpcloud tests against developers specified branch. +* `pipeline_coverity.yml` run a build with coverity flags, separate so that it + isn't accidentally triggered -##### Tasks Directory +#### Tasks Directory All task yamls should live in this directory. If a task file is not referenced in the `pipelines` directory it is considered abandoned and can be removed. There are some exceptions to this rule. Please do not create any more exceptions, and remove these as the occasion arises: -* `package_tarball.yml` is being used by [a gporca pipeline ](https://github.com/greenplum-db/gporca/blob/master/concourse/pipeline.yml) -* `test_with_planner.yml` is being used by [a gporca pipeline ](https://github.com/greenplum-db/gporca/blob/master/concourse/pipeline.yml) -* `test_with_orca.yml` is being used by [a gporca pipeline ](https://github.com/greenplum-db/gporca/blob/master/concourse/pipeline.yml) -* `build_with_orca.yml` is being used by [a gporca pipeline ](https://github.com/greenplum-db/gporca/blob/master/concourse/pipeline.yml) -* `gpcheckcloud_tests_gpcloud.yml` is being used by [a gpcloud pipeline ](https://github.com/greenplum-db/gporca/blob/master/concourse/pipeline_gpcloud.yml) -* `regression_tests_gpcloud.yml` is being used by the main pipeline and [a gpcloud pipeline ](https://github.com/greenplum-db/gporca/blob/master/concourse/pipeline_gpcloud.yml) -* `unit_tests_gpcloud.yml` is being used by [a gpcloud pipeline ](https://github.com/greenplum-db/gporca/blob/master/concourse/pipeline_gpcloud.yml) +* `package_tarball.yml` is being used by + [a gporca pipeline](https://github.com/greenplum-db/gporca/blob/master/concourse/pipeline.yml) +* `test_with_planner.yml` is being used by + [a gporca pipeline](https://github.com/greenplum-db/gporca/blob/master/concourse/pipeline.yml) +* `test_with_orca.yml` is being used by + [a gporca pipeline](https://github.com/greenplum-db/gporca/blob/master/concourse/pipeline.yml) +* `build_with_orca.yml` is being used by + [a gporca pipeline](https://github.com/greenplum-db/gporca/blob/master/concourse/pipeline.yml) -##### Scripts Directory +#### Scripts Directory All script files should live in this directory. If a script file is not referenced in any of the directories it is considered abandoned and can be removed. @@ -40,21 +59,25 @@ There are some exceptions to this rule. Please do not create any more exceptions, and remove these as the occasion arises: * `cpplint.py` is being used by codegen -* `package_tarball.bash` is being used by [a gporca pipeline ](https://github.com/greenplum-db/gporca/blob/master/concourse/pipeline.yml) -* `builds/` is being used by [a gporca pipeline ](https://github.com/greenplum-db/gporca/blob/master/concourse/pipeline.yml) -* `test_gpdb.py` is being used by [a gporca pipeline ](https://github.com/greenplum-db/gporca/blob/master/concourse/pipeline.yml) -* `build_gpdb.py` is being used by [a gporca pipeline ](https://github.com/greenplum-db/gporca/blob/master/concourse/pipeline.yml) -* `gpcheckcloud_tests_gpcloud.bash` is being used by [a gpcloud pipeline ](https://github.com/greenplum-db/gporca/blob/master/concourse/tasks/gpcheckcloud_test_gpcloud.yml) -* `regression_tests_gpcloud.bash` is being used by [a gpcloud pipeline ](https://github.com/greenplum-db/gporca/blob/master/concourse/tasks/regression_tests_gpcloud.yml) -* `unit_tests_gpcloud.bash` is being used by [a gpcloud pipeline ](https://github.com/greenplum-db/gporca/blob/master/concourse/tasks/unit_tests_gpcloud.yml) +* `package_tarball.bash` is being used by + [a gporca pipeline](https://github.com/greenplum-db/gporca/blob/master/concourse/pipeline.yml) +* `builds/` is being used by + [a gporca pipeline](https://github.com/greenplum-db/gporca/blob/master/concourse/pipeline.yml) +* `test_gpdb.py` is being used by + [a gporca pipeline](https://github.com/greenplum-db/gporca/blob/master/concourse/pipeline.yml) +* `build_gpdb.py` is being used by + [a gporca pipeline](https://github.com/greenplum-db/gporca/blob/master/concourse/pipeline.yml) ### The Concourse Deployment -There is a `gpdb` team in the [Concourse instance](http://gpdb.ci.pivotalci.info/). +There is a `gpdb` team in the +[Concourse instance](http://gpdb.ci.pivotalci.info/). Use this team to create any pipelines instead of the `main` team. ### Creating Your Own Pipeline Many developers want to create their own copies of the master pipeline. -To accommodate this without naming confusion, workload instability, and artifact collision we have the following solution: + +To accommodate this without naming confusion, workload instability, nor +artifact collision, we have the following solution: 1. Fork or branch the gpdb git repo 1. Create an s3 bucket for your pipeline to use @@ -69,11 +92,15 @@ fly -t gpdb set-pipeline -c concourse/pipelines/pipeline.yml -p NEW_PIPELINE_NAM #### Notes and warnings * Clean up your dev pipelines when you are finished with them. (Use `fly destroy-pipeline`) -* Be sure to use a unique name for your pipeline. Don't blow away the master pipeline by using `gpdb_master` when setting your dev pipeline. -* Please pause the pulse jobs (the rightmost column of builds). If you are working on one, that's fine, but pause all of the others. We've had issues with too many pulse jobs running at once. +* Be sure to use a unique name for your pipeline. Don't blow away the master + pipeline by using `gpdb_master` when setting your dev pipeline. Try to prefix + your pipeline name with `dev:` +* Please pause the pulse jobs (the rightmost column of builds). If you are + working on one, that's fine, but pause all of the others. We've had issues + with too many pulse jobs running at once. ### Updating This README Changes should be proposed to this contract with a PR. -# Triggering pulse projects +### Triggering pulse projects Use `gpdb5-pulse-worker` tag to invoke the job to trigger and monitor Pulse projects