-Move cast examples to install guide
-Pointer from install guide to admin guide for cast description.

Fixed typo

[ci skip]
Authored-by: NTodd Sedano <tsedano@pivotal.io>

This was left behind to get converted to seconds by commit
7d59d215ec065983c666b80bc2c982d13e476c48. Most likely reason for gpexpand jobs.

This is cherry-pcik of upstream commit c61559ec.
---------------------
Reduce pg_ctl's reaction time when waiting for postmaster start/stop.

pg_ctl has traditionally waited one second between probes for whether
the start or stop request has completed.  That behavior was embodied
in the original shell script written in 1999 (commit 5b912b08) and
I doubt anyone's questioned it since.  Nowadays, machines are a lot
faster, and the shell script is long since replaced by C code, so it's
fair to reconsider how long we ought to wait.

This patch adjusts the coding so that the wait time can be any even
divisor of 1 second, and sets the actual probe rate to 10 per second.
That's based on experimentation with the src/test/recovery TAP tests,
which include a lot of postmaster starts and stops.  This patch alone
reduces the (non-parallelized) runtime of those tests from ~4m30s to
~3m5s on my machine.  Increasing the probe rate further doesn't help
much, so this seems like a good number.

In the real world this probably won't have much impact, since people
don't start/stop production postmasters often, and the shutdown checkpoint
usually takes nontrivial time too.  But it makes development work and
testing noticeably snappier, and that's good enough reason for me.

Also, by reducing the dead time in postmaster restart sequences, this
change has made it easier to reproduce some bugs that have been lurking
for awhile.  Patches for those will follow.

Discussion: https://postgr.es/m/18444.1498428798@sss.pgh.pa.us
---------------------

Authored-by: NJim Doty <jdoty@pivotal.io>

Authored-by: NJim Doty <jdoty@pivotal.io>

This test explicitly creates some tables and verifies that they can be
read from after gpaddmirrors has been run on the cluster. The existing
TINC test that does this will be deleted after all the new gpaddmirrors
tests have been added.

We've also added some code to allow cluster creation on more than one
segment host.
Co-authored-by: NNadeem Ghani <nghani@pivotal.io>
Co-authored-by: NJamie McAtamney <jmcatamney@pivotal.io>

By design, all array types should be hashable by GPDB.
Issue 3741 shows that GPDB suffers from text array type
being not hashable error.

This commit fixes that.

AIX builds are currently failing with the following error message:

    ../../config/install-sh: utils/errcodes.h does not exist.

There is currently a hardcoded list of headers that must be generated
before installing src/include on AIX. errcodes.h came in as part of the
9.1 merge, and it was missed in this list.

For good measure, add this to the Windows build as well (it's not
currently failing there) so that some future unrelated change doesn't
introduce this same build failure.
Co-authored-by: NAsim Praveen <apraveen@pivotal.io>

There are no distributed transactions during binary upgrade, so we can
ignore them.

VACUUM FREEZE must function correctly during binary upgrade so that the
new cluster's catalogs don't contain bogus transaction IDs. Do a simple
check on the QD in our test script, by querying the age of all the rows
in gp_segment_configuration.

As part of the merge with PostgreSQL 9.1, two gpdb/master commits were
incorrectly reverted. This change replays the following commits:

commit dd42f3d4
Author: Todd Sedano <tsedano@pivotal.io>
Date:   Thu May 17 10:47:30 2018 -0700

    Syncing python-dependencies with pythonsrc-ext

    In comparing https://github.com/greenplum-db/pythonsrc-ext
    to python-dependencies.txt, there are several differences.

    pythonsrc-ext is the vendored python repo, so it should be
    in sync with pythonsrc-ext

    argparse is in pythonsrc-ext, adding it to python-dependencies.txt

    enum34, parse-type, ptyprocess, six are not in pythonsrc-ext,
    so moving them to python-developer-dependencies.txt
Authored-by: NTodd Sedano <tsedano@pivotal.io>

commit 7f9f11d3
Author: Daniel Gustafsson <dgustafsson@pivotal.io>
Date:   Fri May 18 09:43:48 2018 +0200

    Fix typo in comment

Add hook framework in signal handler, e.g. StatementCancelHandler or die,
to cancel the slow function in a 3rd party library.

Add PLy_handle_cancel_interrupt hook to cancel plpython, which use
Py_AddPendingCall to cancel embedded python asynchronously.

Many places is code need to check if its row or column oriented storage table,
which means basically is it AppendOptimized table or not. Currently its done by
combination of two macros RelationIsAoRows() and RelationIsAoCols(). Simplify
the same with new macro RelationIsAppendOptimized().

This is the final batch of commits from PostgreSQL 9.1 development, up
to the point where the REL9_1_STABLE branch was created, and 9.2
development started on the PostgreSQL master branch.

Notable upstream changes:

* Changes to "temporary" vs. "permanent" relation handling. Previously,
  relations were classified as either temporary or not-temporary. Now
  there is a concept of "relation persistence", which includes
  un-WAL-logged (but permanent) tables as a third option.

  Additionally, 9.1 adds the backend ID to the relpath of temporary
  tables. Because GPDB does not have a 1:1 relationship between backends
  and temp relations, this has been reverted for now and the concept of
  a special `TempRelBackendId` has been added. We may look into
  backporting pieces of the parallel session support from future
  versions of PostgreSQL to minimize this difference.

  There is currently a fatal flaw in our handling of relfilenodes for
  temporary relations: it is possible (though unlikely) for a temporary
  sequence and a permanent relation (or vice-versa) to point to the same
  shared buffer in the buffer cache. This may lead to corruption of that
  buffer. Our current attempt at a fix for this issue isn't good enough
  yet, so the offending code is marked with FIXMEs. We plan to solve this
  problem with a total overhaul of the way temporary relations are
  stored in the shared buffer cache.

* Collation is now supported per-column. Many plan nodes now have their
  own concept of input and output collation IDs. Notably, since this
  change involves heavy changes to generated plans, ORCA does not yet
  support non-default collation.

* The SERIALIZABLE isolation level was added. This is not currently
  supported in GPDB, because we don't have a way to analyze
  serialization conflicts in a distributed manner; we will "fall back"
  to REPEATABLE READ.

* GROUP BY now allows selection of columns in the grouping query without
  mentioning those columns in the GROUP BY clause, as long as those
  columns are from a table that's being grouped by its primary key. This
  requires some modification to the parallel grouping planner in GPDB.

* Support for INSTEAD OF triggers on views. Notably, this would allow
  "modification" of views, but GPDB currently has no support for this
  concept in the planner. This functionality is temporarily disabled for
  now.

* CTEs (WITH ... AS) can now contain data modification statements
  (INSERT/UPDATE/DELETE). This is not yet supported in GPDB.

* `make -k` is now supported for installcheck[-world] and several other
  Makefile targets.

* Foreign tables. Currently, GPDB does not support planning for
  statements made against foreign tables, so CREATE FOREIGN TABLE has
  been disabled.

* SELinux support and integration has been added with the SECURITY LABEL
  command and the associated sepgsql utility. SECURITY LABEL support is
  currently tested in GPDB via the upstream regression tests, but
  sepgsql is not.

* GUC API changes: assignment hooks are now split into two phases
  (check/assign).

Notable GPDB changes:

* Several pieces of the COPY dispatching code have been heavily
  refactored/rewritten to match upstream more closely. For now,
  compatibility with 5.x external table catalog entries has been
  maintained, though this is likely to change.
Co-authored-by: NAsim Praveen <apraveen@pivotal.io>
Co-authored-by: NBhuvnesh Chaudhary <bchaudhary@pivotal.io>
Co-authored-by: NDhanashree Kashid <dkashid@pivotal.io>
Co-authored-by: NGoutam Tadi <gtadi@pivotal.io>
Co-authored-by: NHeikki Linnakangas <hlinnakangas@pivotal.io>
Co-authored-by: NJacob Champion <pchampion@pivotal.io>
Co-authored-by: NJesse Zhang <sbjesse@gmail.com>
Co-authored-by: NJinbao Chen <jinchen@pivotal.io>
Co-authored-by: NLarry Hamel <lhamel@pivotal.io>
Co-authored-by: NMax Yang <myang@pivotal.io>
Co-authored-by: NMelanie Plageman <mplageman@pivotal.io>
Co-authored-by: NOmer Arap <oarap@pivotal.io>
Co-authored-by: NPaul Guo <paulguo@gmail.com>
Co-authored-by: NRichard Guo <guofenglinux@gmail.com>
Co-authored-by: NSambitesh Dash <sdash@pivotal.io>
Co-authored-by: NShreedhar Hardikar <shardikar@pivotal.io>
Co-authored-by: NTaylor Vesely <tvesely@pivotal.io>
Co-authored-by: NVenkatesh Raghavan <vraghavan@pivotal.io>

In comparing https://github.com/greenplum-db/pythonsrc-ext
to python-dependencies.txt, there are several differences.

pythonsrc-ext is the vendored python repo, so it should be
in sync with pythonsrc-ext

argparse is in pythonsrc-ext, adding it to python-dependencies.txt

enum34, parse-type, ptyprocess, six are not in pythonsrc-ext,
so moving them to python-developer-dependencies.txt
Authored-by: NTodd Sedano <tsedano@pivotal.io>

Authored-by: NTodd Sedano <tsedano@pivotal.io>

The TINC tests in the walrep_1 job have been migrated to Behave. This
commit removes the walrep_1 job and updates the generated master
pipeline.
Co-authored-by: NNadeem Ghani <nghani@pivotal.io>
Co-authored-by: NJamie McAtamney <jmcatamney@pivotal.io>

Issue is that during recovery in progress mirror can take longer than default
timeout to finish promotion. Because the timeout is sometimes too short
gprecoverseg would intermittently throw 'error 'can't start transaction' in
'BEGIN'. This commit leverages GUCS gp_gang_creation_retry_count and
gp_gang_creation_retry_timer to increase the timeout alotted for gang retriable
errors to approximately 30 seconds.
Co-authored-by: NAshwin Agrawal <aagrawal@pivotal.io>

The walrep_1 TINC tests have been migrated to the Behave framework under
gpMgmt top-level directory. We remove the Makefile target but keep the
TINC tests because the walrep_2 target has some test dependencies.
Co-authored-by: NJimmy Yih <jyih@pivotal.io>
Co-authored-by: NNadeem Ghani <nghani@pivotal.io>

We added a job for gpactivatestandby and moved the gpinitstandby job to
the multi-node CLI suite.
Co-authored-by: NNadeem Ghani <nghani@pivotal.io>
Co-authored-by: NJimmy Yih <jyih@pivotal.io>

These scenarios come from the walrep_1 TINC Makefile target.  They have
been designed to run on both singlenode and multinode environments. The
tests have not been removed from TINC yet because some walrep_2 TINC
tests have dependencies.
Co-authored-by: NJimmy Yih <jyih@pivotal.io>
Co-authored-by: NNadeem Ghani <nghani@pivotal.io>

These scenarios come from the walrep_1 TINC Makefile target. They have
been designed to run on both singlenode and multinode environments.
The tests have not been removed from TINC yet because some walrep_2 TINC
tests have dependencies.
Co-authored-by: NJimmy Yih <jyih@pivotal.io>
Co-authored-by: NNadeem Ghani <nghani@pivotal.io>

These are ported from the master/standby TINC tests for checking the
following:

1. gpstart -a -y does nothing when there is no standby master configured
2. gpstop -a -y skips standby master when there is one configured
Co-authored-by: NNadeem Ghani <nghani@pivotal.io>
Co-authored-by: NJimmy Yih <jyih@pivotal.io>

The gpactivatestandby utility always used $MASTER_DATA_DIRECTORY
environment variable, even when the -d flag was populated by the user.
Fix the issue by actually using the directory path supplied by the user.
Co-authored-by: NKevin Yeap <kyeap@pivotal.io>
Co-authored-by: NJimmy Yih <jyih@pivotal.io>

Quick indent fix, the tables string was appended wrongly.
Signed-off-by: NMing LI <mli@apache.org>

We find out all the sequences in the table, and verify if the
   sequence should be created. later, we get the complete dump sql
   that included the sequnce DDL and table DDL, It can create
   the sequence before creating the table

   This implementation is to avoid locking table during drop table with sequence in transaction, meantime execute command `psql -f ...` to create this sequence.
Signed-off-by: NMing LI <mli@apache.org>

Integer overflow occurs without this when copied more than 2^31 rows,
under the `COPY ON SEGMENT` mode.

Errors happen when it is casted to uint64, the type of `processed` in
`CopyStateData`, third-party Postgres driver, which takes it as an
int64, fails out of range.

* docs - resgroup memory_auditor to cat/view/gptoolkit

* cgroup mem usage output - used and limit_granted

* updates for code refactor that was just merged

* type to text

* docs: update cast information for GPDB 5/6

Update cast information. Add information about limited text casts.
See section "Type Casts"

* docs: review comments for GPDB 5/6 updated cast information.

* docs: fix typos in updated CAST info

Authored-by: NTodd Sedano <tsedano@pivotal.io>

To "rebalance" a primary-mirror pair, gprecoverseg -r performs the following steps:

1.  bring down the acting primary
2.  issue a query that triggers the failover
3.  bring up the mirror (gprecoverseg -F)

Currently these 3 steps are happening in close succession. However, there is a chance that between step 2 and step 3, the mirror promotion happens slower than we expect. The implicit assumption here is that the acting mirror has finished transitioning to the primary role before step 3 is performed.

This patch adds a retry in "sort of step 2, definitely before step 3", to ensure a good state before we can bring up the mirror.

Co-authored-by: Jesse Zhang sbjesse@gmail.com
Co-authored-by: David Kimura dkimura@pivotal.io

Since protocols do not belong to a namespace, we do not want to dump
them in table or schema filtered backups. They will only be dumped in a
full backup.
Co-authored-by: NKaren Huddleston <khuddleston@pivotal.io>
Co-authored-by: NChris Hajas <chajas@pivotal.io>

If the column statistics in `pg_statistic` has values with type
different than column type, metadata accessor should not translate the
stats and create a dummy stats instead.

This commit also reorders stats collection from the `pg_statistic` to
align with how analyze generates stats. MCV and Histogram translation is
moved to the end after NDV, nullfraction and column width extraction.
Signed-off-by: NMelanie Plageman <mplageman@pivotal.io>

Co-authored-by: NDavid Sharp <dsharp@pivotal.io>
Co-authored-by: NLarry Hamel <lhamel@pivotal.io>

Consider the following SQL, we expect logging to be turned off for table
`ext_error_logging_off`

```sql
create external table ext_error_logging_off (a int, b int)
    location ('file:///tmp/test.txt') format 'text'
    segment reject limit 100;
\d+ ext_error_logging_off
```
And then in this next case we expect error logging to be turned on for
table `ext_t2`:

```sql
create external table ext_t2 (a int, b int)
    location ('file:///tmp/test.txt') format 'text'
    log errors segment reject limit 100;
\d+ ext_t2
```

Before this patch, we are making two mistakes in handling these external
table DDL:

1. We intend to enable error logging *whenever* the user specifies
`SEGMENT REJECT` clause, completely ignoring whether he or she specifies
`LOG ERRORS` 1. Even then, we make the mistake of implicitly coercing
the OID (an unsigned 32-bit integer) to a bool (which is really just a C
`char`): that means, 255/256 of the time (99.6%) the result is `true`,
and 0.4% of the time we get a `false` instead.

The `OID` to `bool` implicit conversion could have been caught by a
`-Wconversion` GCC/Clang flag. It's most likely a leftover from commit
8f6fe2d6.

This bug manifests itself in the `dsp` regression test mysteriously
failing about once every 200 runs -- with the only diff on a `\d+` of an
external table that should have error logging turned on, but the
returned definition has it turned off.

While working on this we discovered that all of our existing external
tables have both `LOG ERRORS` and `SEGMENT REJECT`, which is why this
bug wasn't caught in the first place.

This patch fixes the issue by properly setting the catalog column
`pg_exttable.logerrors` according to the user input.

While we were at it, we also cleaned up a few dead pieces of code and
made the `dsp` test a bit friendlier to debug.
Co-authored-by: NJesse Zhang <sbjesse@gmail.com>
Co-authored-by: NDavid Kimura <dkimura@pivotal.io>