Reorganize pipelines to reduce syncronized job starts

The GPDB5 deployment in the gpdb5-pipeline AWS account has a week of
instability, and to the best of our knowledge this appears to be due to
the "fan out" pattern that is manifested in our pipeline.  After the
compile steps are completed, many, many jobs are kicked off
simultaneaously.  This structure, combined with the many copies of the
master pipeline cause such a large simulaneous workload that concourse
resources like volume creation can not keep up with the demand.

We do not contend that this structure is the correct structure, and we
welcome collaboration on what jobs should live where.  Hopefully this
will keep the deployment humming for the near future.
Signed-off-by: NDivya Bhargov <dbhargov@pivotal.io>

concourse/scripts/validate_pipeline_release_jobs.py

@@ -5,7 +5,11 @@ import re
 import yaml
 
 RELEASE_VALIDATOR_JOB = ['Release_Candidate']
-JOBS_THAT_SHOULD_NOT_BLOCK_RELEASE = ['compile_gpdb_binary_swap_centos6', 'icw_gporca_centos6_gpos_memory'] + RELEASE_VALIDATOR_JOB
+JOBS_THAT_ARE_GATES = ['gate_compile_start', 'gate_compile_end', 'gate_icw_start', 'gate_icw_end',
+        'gate_mm_misc_start', 'gate_mm_misc_end', 'gate_general_misc_start', 'gate_general_misc_end',
+        'gate_filerep_start', 'gate_filerep_end', 'gate_cluster_start', 'gate_cluster_end',
+        'gate_nightly_start', 'gate_nightly_end', 'gate_cs_misc_start', 'gate_cs_misc_end']
+JOBS_THAT_SHOULD_NOT_BLOCK_RELEASE = ['compile_gpdb_binary_swap_centos6', 'icw_gporca_centos6_gpos_memory'] + RELEASE_VALIDATOR_JOB + JOBS_THAT_ARE_GATES
 
 pipeline_raw = open(os.environ['PIPELINE_FILE'],'r').read()
 pipeline_buffer_cleaned = re.sub('{{', '', re.sub('}}', '', pipeline_raw)) # ignore concourse v2.x variable interpolation