This commit updates the `rangefuncs_cdb` tests to use `plpgsql`
instead of `plperlu`; since `plperlu` is not supported by SLES.

We do not need a separate answer file for ORCA because the results
produced by planner and ORCA are the same. Hence this commit also
removes `rangefuncs_cdb_optimizer.out` file.
Signed-off-by: NEkta Khanna <ekhanna@pivotal.io>

* Replace "template1" with "postgres" database

* Move ddboost cleanup step to be last to ensure all files in directory are removed from data domain server

Follow on to #2495

* Updated SLES targets to remove perl and mapreduce

* Removed unneded --disable-mapreduce

Add targets for gpnetbenchClient and gpnetbenchServer to top-level
Makefile so that gpcheckperf runs with the open source build

Remove targets for gpnetbenchClient and gpnetbenchServer from gpAux
Makefile

Add additional debugging print statement to gpcheckperf
Signed-off-by: NMelanie Plageman <mplageman@pivotal.io>

* add "libpq" to list of common APIs

libpq is probably the most common API for PostgreSQL.

* Update documentation to reflect new test target:
"make installcheck-good" -> "make installcheck-world"

Provide alternative answer for gp_metadata test when GPORCA is not compiled in

Also, add retry to check for walsender is gone after walrcv_disconnect() as it
can take sometime to detect the connection drop.

For subselect_gp test, we were removing the distribution policy of a
table to see if it would do a gather motion or not. Since it's
technically a corrupted table, we should delete it after we're done
with it. We also remove a quicklz reference that should not have been
there.

For gppc test, they were using the regression database. This made our
gpcheckcat call at the end of ICW relatively useless since all our
data would have been deleted due to gppc tests recreating the
regression database.

For gpload test, some generated files were previously commited. We
should be actively cautious of this and remove them when we see them.

PL/Perl is an optional component, and the main ICW should not use
it as it may not be present. Move the tests that seem useful to
the plperl test suite instead and remove the ones which we have
ample coverage for elsewhere.

The gpmapreduce application is an optional install included via
the --enable-mapreduce configure option. The tests were however
still in src/test/regress and unconditionally included in the
ICW schedule, thus causing test failures when mapreduce wasn't
configured.

Move all gpmapreduce tests to co-locate them with the mapreduce
code and only test when configured.

Also, add a dependency on Perl for gpmapreduce in autoconf since
it's a required component.

This is part of the effort to get all backup/restore tests using the
same test suite. Since the Netbackup tests take significantly longer, we
only run a subset of the regular test suite. We also tag scenarios to
allow parallel runs on separate hosts in CI.

This suite will take 1h, 40 mins after parallelization, down from the
current 2h, 20 mins.

- Before building Index object (IMDIndex), we build LogicalIndexes
  via calling `gpdb::Plgidx(oidRel)` in which a partition tables is
  traversed and index  information (such as logicalIndexOid,
  nColumns, indexKeys, indPred, indExprs, indIsUnique, partCons,
  defaultLevels) is captured.
- For Indexes which are available on all the partitions partCons
  and defaultLevels are NULL/empty.
- Later in `CTranslatorRelcacheToDXL::PmdindexPartTable` to build
  Index object, we use the derived LogicalIndexes information and
  populates the array  holding the levels on which default
  partitions exists. But since defaultLevels is NIL in this case,
  pdrgpulDefaultLevels is set to empty i,e `default partitions on levels: {}`
- This causes an issue while trying to build the propagation expression,
  as because of wrong number of default partitions on level we mark the
  scan as partial and tries to construct a test propagation expression
  instead of a const propagation expression.
- This patch fixes the issue by marking the default partitions on
  levels for index equal to the default partitions on levels for the
  part relation if the index exists on all the parts.
Signed-off-by: NJemish Patel <jpatel@pivotal.io>

* Clarify `make behave` behavior and remove unused call

* Add flags argument to pass in custom flags to behave

Allows us to use more complex tag combinations for including/excluding
scenarios. This also removes the skip_tags argument as it only worked in
limited circumstances.
Signed-off-by: NNadeem Ghani <nghani@pivotal.io>

Fix MU_check concourse task to print failing unit test on error.

This patch is to fix below defects reported by Coverity Scan.

```
New defect(s) Reported-by: Coverity Scan
Showing 3 of 3 defect(s)

** CID 169611:  Security best practices violations  (TOCTOU)
/tmp/build/0e1b53a0/gpdb_src/src/backend/utils/resgroup/resgroup-ops-cgroup.c:
272 in createDir()

________________________________________________________________________________________________________
*** CID 169611:  Security best practices violations  (TOCTOU)
/tmp/build/0e1b53a0/gpdb_src/src/backend/utils/resgroup/resgroup-ops-cgroup.c:
272 in createDir()
266
267             buildPath(group, comp, "", path, pathsize);
268
269             if (access(path, F_OK))
270             {
271                     /* the dir is not created yet, create it */
>>>     CID 169611:  Security best practices violations  (TOCTOU)
>>>     Calling function "mkdir" that uses "path" after a check
>>>     function. This can cause a time-of-check, time-of-use race
>>>     condition.
272                     if (mkdir(path, 0755) && errno != EEXIST)
273                             return false;
274             }
275
276             return true;
277     }

** CID 169610:  Control flow issues  (NO_EFFECT)
/tmp/build/0e1b53a0/gpdb_src/src/backend/utils/resgroup/resgroup-ops-cgroup.c:
412 in writeData()

________________________________________________________________________________________________________
*** CID 169610:  Control flow issues  (NO_EFFECT)
/tmp/build/0e1b53a0/gpdb_src/src/backend/utils/resgroup/resgroup-ops-cgroup.c:
412 in writeData()
406             size_t ret = write(fd, data, datasize);
407
408             /* save errno before close */
409             int err = errno;
410             close(fd);
411
>>>     CID 169610:  Control flow issues  (NO_EFFECT)
>>>     This less-than-zero comparison of an unsigned value is never
>>>     true. "ret < 0UL".
412             if (ret < 0)
413                     CGROUP_ERROR("can't write data to file '%s':
%s", path, strerror(err));
414             if (ret != datasize)
415                     CGROUP_ERROR("can't write all data to file
'%s'", path);
416     }
417

** CID 169609:  Control flow issues  (NO_EFFECT)
/tmp/build/0e1b53a0/gpdb_src/src/backend/utils/resgroup/resgroup-ops-cgroup.c:
385 in readData()

________________________________________________________________________________________________________
*** CID 169609:  Control flow issues  (NO_EFFECT)
/tmp/build/0e1b53a0/gpdb_src/src/backend/utils/resgroup/resgroup-ops-cgroup.c:
385 in readData()
379             size_t ret = read(fd, data, datasize);
380
381             /* save errno before close() */
382             int err = errno;
383             close(fd);
384
>>>     CID 169609:  Control flow issues  (NO_EFFECT)
>>>     This less-than-zero comparison of an unsigned value is never
>>>     true. "ret < 0UL".
385             if (ret < 0)
386                     CGROUP_ERROR("can't read data from file '%s':
%s", path, strerror(err));
387
388             return ret;
389     }
390
```

Remove duplicate unittest_check_gpdb in compile_gpdb.bash, it's already run by the above "if" block when needed.

Some minor documentation fixes

[ci skip]

`outputdir` here is where to place the converted files, not the
directory named `output`. PostgreSQL places them into `outputdir`
rightly.

    commit 64cdbbc4
    Author: Peter Eisentraut <peter_e@gmx.net>
    Date:   Sat Feb 14 21:33:41 2015 -0500

        pg_regress: Write processed input/*.source into output dir

        Before, it was writing the processed files into the input directory,
        which is incorrect in a vpath build.

This way we are consistent now with the upstream Postgres.

On pulse, this seemed to make no change
On concourse, this added color
Signed-off-by: NLarry Hamel <lhamel@pivotal.io>
Signed-off-by: NNadeem Ghani <nghani@pivotal.io>

* doc updates for variadic function arguments and type modifiers

* david s requested edits

Amazon S3 supports server side encryption to encrypt the uploaded files.
In this commit, gpcloud can support setting S3 server side encryption
(SSE-S3) in writable external table. A new option is added into gpcloud
configuration file, named server_side_encryption. User can add an option
'server_side_encryption = sse-s3' to enable server side encryption.

The default value of server_side_encryption is none if this option is not
appeared in configuration file and server side encryption will be disabled.
Signed-off-by: NHaozhou Wang <hawang@pivotal.io>

ON MASTER feature is not fully supported by built-in protocols other
than s3 yet, disable it with those protocols for now.

1. Add --with-gssapi to sles configurations in gpAux/Makefile to enable kerberos build
2. Add kerberos sbin path to PATH for sles.
3. Disable psql pager to avoid concourse hang.
Signed-off-by: NYuan Zhao <yuzhao@pivotal.io>

- Provide a high-level overview welcome section
- Remove exceptions for gpcloud -- now in this repo
- Newline/return more frequently in the markdown