Now that we use the upstream implementation for window functions, the
'gp_enable_sequential_window_plans' and 'gp_idf_deduplicate' GUCs are no
longer.

HASHJOIN_IS_OUTER already existed in nodeHashJoin.c; but was added again
by commit 321c0529.
This commit removes the redundant copy of the same.

This is to avoid collision of the nonsegwalrep binary.

Long ago, we thought we might need to support multiple mirrors. But we
don't, and don't forsee it coming soon. Simplify the code to only ever
have one mirror, but still allow for the possibility of no mirrors

Author: Shoaib Lari <slari@pivotal.io>
Author: C.J. Jameson <cjameson@pivotal.io>

The gpArray use of GpDB and Segment classes was confusing. This change renames
GpDB to Segment and Segment to SegmentPair to clarify usage. Its a big diff, but
a simple, repeating change.

Author: Shoaib Lari <slari@pivotal.io>
Author: Marbin Tan <mtan@pivotal.io>
Author: C.J. Jameson <cjameson@pivotal.io>

* Have a separate docker file for gpadmin user

* Add indent package for centos6

The temporary file hack to deal with pygresql output oddities didn't
ensure that those files were always removed. Instead of a named file we
don't need, just use a TemporaryFile that will go away as soon as we
release it.

Added a FIXME to revisit this once we upgrade pygresql.

pg_proc_gp.h is generated from pg_proc.sql, but a few recent commits
updated only pg_proc_gp.h, not pg_proc.sql. As a result, if you ran the
perl script, the gp_replication_error() function vanished, and the obsolete
PT verification functions reappeared. Update pg_proc_gp.h, so that
pg_proc_gp.h is reproduced by the perl script again.

Instead of assuming that casts are always binary coercible (and hence that we
could get away with just dropping them), translate casts in ORCA plans into
either a RelabelType or a FuncExpr.
Signed-off-by: NSambitesh Dash <sdash@pivotal.io>

PR for 5X_STABLE
Will be ported to MAIN

* docs: change GUC default: optimizer_join_arity_for_associativity_commutativity=18

* docs: GUC removed ZSTD (zstandard)  from gp_default_storage_options (added by mistake)

The correct order is as follows:
1. modify ~/.bash_profile
2. ssh to localhost
3. make create-demo-cluster

See https://github.com/greenplum-db/gpdb/issues/3704 for more detail

There is a long-standing race condition in BackoffSweeper() which
triggers an error and then triggers another assertion failure for
not reset sweeperInProgress to false.

This commit doesn't resolve the race condition fundamentally with
lock or other implementation, because the whole backoff mechanism
did not ask for accurate control, so skipping some sweeps should
be fine so far. We also downgrade the log level to DEBUG because
a restart of sweeper backend is unnecessary.

Partition tables hard-code the operator '=' lookup to namespace
'pg_catalog', which means that in this test we had to put our
user-defined operator into that special system namespace. This works
fine, until we try to pg_dump the resulting database: pg_catalog is not
dumped by default. That led to an incomplete dump that will fail to
restore.

This commit reinstates the dropping of the schema at the end of `gporca`
test to get the pipelines back to green (broken after c7ab6924 ).
Backpatch to 5X_STABLE.

xlogviewer is disconnected from the build, and hasn't built since
we merged the HOT patch during the PostgreSQL 8.3 merge in the 5.x
development cycle. Since we have contrib/xlogdump, remove rather
than trying to resurrect it.

Discussion: https://groups.google.com/a/greenplum.org/forum/#!topic/gpdb-dev/ov5Sxrd1JL4

Since the docs folder is ignored, it is possible for the gpdb_src to get
to a detatched HEAD state if there was a docs commit after a given
commit. We don't really need the branch to get checkout the commit in
question.
Signed-off-by: NEd Espino <edespino@pivotal.io>
Signed-off-by: NDivya Bhargov <dbhargov@pivotal.io>

The `gporca` regression test suite uses a schema but doesn't really
switch `search_path` to the schema that's meant to encapsulate most of
the objects it uses. This has led to multiple instances where we:
  1. Either used a table from another namespace by accident;
  2. Or we leaked objects into the public namespace that other tests in
  turned accidentally depended on.

As we were about to add a few user-defined types and casts to the test
suite, we want to (at last) ensure that all future additions are scoped
to the namespace.
Signed-off-by: NSambitesh Dash <sdash@pivotal.io>

Closes #4238

Author: C.J. Jameson <cjameson@pivotal.io>

*  docs: pl/container updates - plcontainer command, configuration file

-removed experimental warning.
-replaced plcontainer command with new options.
-updated configuration file - changed element names, added setting element.
-updated default shared volume.

PR for 5X_STABLE
Will be ported to MAIN

* docs: pl/container - updates based on review comments.

* docs: pl/container - more updates based on additional review comments.

* docs: pl/container - minor edit

* docs: pl/container - another minor fix.

* docs - add info about greenplum database release versioning

* misc edits requested by david

* update subnav titles

* clarify that deprecated features removed only in major release

* docs: gpstop new option --host

* docs: gpstop - update update/clarify --host description based on review comments.

* docs: gpstop --host. updates based on review comments.

* docs: gpstop - added information on restoring segments after using --host.

* docs: gpstop --host. corrected name of utility to recover segments : gprecoverseg.

Several \d variants don't put a row count at the end of their tables,
which means that atmsort doesn't stop sorting output until it finds a
row count somewhere later. Some tests are having their diff output
horribly mangled because of this, which makes debugging difficult.

When we see a \d command, try to apply more heuristics for finding the
end of a table. In addition to ending at a row count, end table sorting
whenever we find a line that doesn't have the same number of column
separators as the table header. If we don't have a table header, still
attempt to end table sorting at a blank line.

extprotocol's test output order must be fixed as a result. Put the
"External options" line where psql actually prints it, after "Format
options".

https://github.com/hyperic/sigar is not under active development.

https://github.com/boundary/sigar is a fork that has been somewhat updated. In particular, it has a fix to allow it to compile with GCC 5 and 6.

* Fix 4 (out of 64) windowerr tests that use row_number() to be non-deterministic

* Fix remaining 58 (out of 64) tests.

There was a difference in the results between planner and optimizer due
to different value of row_number assigned.

row_number() is inherently non-deterministic in GPDB. For example, for
the following query:

  select row_number() over (partition by b) from foo;

Let's say that foo was not distributed by b. In this case, to compute
the WindowAgg, we would first have to redistribute the table on b (or
gather all the tuples on master). Thus, for rows having the same b
value, the row_number assigned depends on the order in which they are
received by the WindowAgg - which is non-deterministic.

In qp_olap_windowerr.sql tests, we mitigate this by forcing an order on
ord column, which is unique in this context, making it easier to compare
test results.

* Remove FIXME comment and enable optimizer_trace_fallback
Signed-off-by: NShreedhar Hardikar <shardikar@pivotal.io>

o This moves the validation (subprocess call) of the pipelines release
  jobs into the tool which generates the production pipeline. The
  corresponding task validate_pipeline.yml is removed and
  corresponding job.

o Output production fly commands for both "gpdb_master" and
  "gpdb_master_without_asserts" pipelines. This will help engineers
  update both production pipelines.

o Fix the icw sles jobs in the development pipelines.

o Update README.md with usage examples.

o Remove validate_pipeline from pr_pipeline.yml as this validation is
  moving to gen_pipeline.py.

Signed-off-by: NShreedhar Hardikar <shardikar@pivotal.io>

Passing -K to demo_cluster.sh will create gpdemo without enabling
data checksums, which otherwise is the default. The reason for why
one would want to disable checksums is to facilitate testing of
upgrades from a non-checksum cluster to one with (and vice versa).

When upgrading the cluster, allow the addition or removal of data
checksums on the data pages as they are copied across from the
old cluster. This allows for adding checksums without requiring a
full dump/restore cycle.

The test_gpdb.sh script is extended to support adding/removal of
checksums. In the process, also extend to remove gpdemo config
files generated when setting up the new cluster, and put them in
gitignore as well to keep git status happy during testing.

This also fixes a previously incorrect check for checksum version
and aligns variable names better with upstream.

The new tests added as part of the postgres merge, contain tests of
window functions like: last_value, nth_value without specifying the
ORDER BY clause in the window function. Due to Redistribute Motion
getting added, the order is not deterministic without an explicit ORDER
BY clause within the window function.

This commit updates such tests for the relevant changes in ORCA (https://github.com/greenplum-db/gporca/commit/855ba856fdc59e88923523f1f8b2ead32ae32364).
Signed-off-by: NSambitesh Dash <sdash@pivotal.io>