* The --batch-size=10 flag was mistakenly added to a scenario that
* requires ordering and thus a batch size of 1.

In this commit, we change the ownership of a StreamNode, so that when a
StreamNode A is added as an input to a StreamNode B, the bitmap that was
pointing to A abandons the ownership and the pointer is set to NULL. In the
above example, when GPDB frees BitmapOR, it will not attempt to free OR's
StreamNode.

In addition, if a StreamNode is an OpStream, then opstream_free function is
used to free the memory used by the StreamNode, which actually uses pfree. On
the other hand, if a StreamNode is an IndexStream, then stream_free function is
invoked, which does not pfree the StreamNode. For consistency, we add function
indexstream_free to always pfree a StreamNode of IndexStream type.
Signed-off-by: NShreedhar Hardikar <shardikar@pivotal.io>

-- add behave test for gp_default_storage_options
-- add PgHba class to represent pg_hba.conf for textual manipulation
Signed-off-by: NMarbin Tan <mtan@pivotal.io>

* Update foreign key json file name to 5.0

* This was breaking gpcheckcat which assumes the json file corresponds
* to the current greenplum version.
* Update docs for generating json file.

Authors: Chris Hajas and Jamie McAtamney

CI red with unit test failure:

```
[ RUN         ] test__open_alert_log_file__NonGucOpen
[          OK ] test__open_alert_log_file__NonGucOpen
[ RUN         ] test__open_alert_log_file__NonMaster
[          OK ] test__open_alert_log_file__NonMaster
[ RUN         ] test__open_alert_log_file__OpenAlertLog
                "gpperfmon/logs/alert_log.1.log" != "gpperfmon/logs/alert_log.12345"
                ERROR: Check of parameter filename, function fopen failed
Expected parameter declared at syslogger_test.c:69
                ERROR: syslogger_test.c:19 Failure!
[      FAILED ] test__open_alert_log_file__OpenAlertLog
[=============] 3 tests ran
[ PASSED      ] 2 tests
[ FAILED      ] 1 tests, listed below
[ FAILED      ] test__open_alert_log_file__OpenAlertLog
```

This reverts commit ffb2cf08.

There are two issues with current logfile_getname()
- First, it doesn't honor the `gp_log_format` GUC. Originally,
when the format is CSV, then `.csv` is used, and when format is TEXT,
the `.log` if used. However, currently the `.csv` is always used regardless
of the `gp_log_format` settings, hence make the content of the file
and suffix inconsistent.

- Second, it mistakenly generate logs with wrong suffix `.csv.csv`
during logfile_rotate(), due to the wrong assumption of the filename
always containing `.log` when suffix is NULL. Also, due to the calling
sequence of the logfile_rotate, an extra empty file is generated, e.g.
in this case, the file with `.csv.csv` is always empty.

Fix in this patch bring back original GPDB behavior. After the fix,
we generated correct extension, however, still an empty extra log file
generated during log rotation.

However, a separate refactoring is required to clean up all the API
changes in all the callers of logfile_getname(), since the parameter
`suffix` is no longer needed. Also, the calling of logfile_rotate()
to fix the extra empty log file issue.
Signed-off-by: NXin Zhang <xzhang@pivotal.io>

After 8.3 merge, gpdb has new polymorphic types, ANYENUM and ANYNONARRAY.

This fix adds support for ANYENUM and ANYNONARRAY in Translator.

As per postgreSQL, when a function has polymorphic arguments and results;
in the function call they must have the same actual type.
For example, a function declared as `f(ANYARRAY) returns ANYENUM`
will only accept arrays of enum types.

We already have this resolution logic implemented in
`resolve_polymorphic_argtypes()`.

Refactor the code in `PdrgpmdidResolvePolymorphicTypes()` to use
`resolve_polymorphic_argtypes()` to deduce the correct data type for
function argument and return type, based on function call.
Signed-off-by: NBhuvnesh Chaudhary <bchaudhary@pivotal.io>
Signed-off-by: NOmer Arap <oarap@pivotal.io>

We fixed all the shell checks for the scan_with_coverity script and now
the cov-build is run in gpAux. We added the following step to tar it and upload
the tarball to scan.coverity.com.

In the pipeline file we added a twice weekly trigger on Monday and
Thursday morning.
Signed-off-by: NJingyi Mei <jmei@pivotal.io>
Signed-off-by: NTom Meyer <tmeyer@pivotal.io>

The gpstringsubs.pl rule depends on gpstringsubs.pl which cause
errors like the following when building:

make[1]: Circular gpstringsubs.pl <- gpstringsubs.pl dependency dropped.

to be compatible with lower version CC.
Signed-off-by: NYuan Zhao <yuzhao@pivotal.io>

Modify gpcheckcloud makefile to put objects into its own directory.
Remove -B from gpcloud targets, to avoid recompiling gpcheckcloud everytime.
Signed-off-by: NPeifeng Qiu <pqiu@pivotal.io>

Signed-off-by: NBhunvesh Chaudhary <bchaudhary@pivotal.io>

Signed-off-by: NBhunvesh Chaudhary <bchaudhary@pivotal.io>

Signed-off-by: NBhunvesh Chaudhary <bchaudhary@pivotal.io>

Signed-off-by: NBhunvesh Chaudhary <bchaudhary@pivotal.io>

The 8.3 merge brought in a change to BufferSync() introducing the
concept of a slow checkpoint which throttles I/O so that regular
background checkpoints do not heavily affect other operations. Our
pg_basebackup call in gpinitstandby was not changed to bypass this new
functionality so here we make it use fast checkpoint to do so. The
fast arg was not utilized yet to send CHECKPOINT_IMMEDIATE to
RequestCheckpoint() so brought in the change from upstream Postgres.

Referenced upstream Postgres commit:
https://github.com/postgres/postgres/commit/387060951e40d550d37fe0457521e900d8c60feb

Signed-off-by: NHaisheng Yuan <hyuan@pivotal.io>

Signed-off-by: NXin Zhang <xzhang@pivotal.io>

The missing environment caused the `make clean` raised syntax error.

Reduces test time from 2:50h to 1:40h by batching gpfdist processes (already built into gptransfer)

* Use batch size of 10 instead of default of 2
* Eliminate testdb_2, which was not doing anything useful

Original comments is regarding to merging upstream Postgres commit
https://github.com/postgres/postgres/commit/acfce502#diff-82589fcf3640eac3ba54e51e998ec6f5.

There are two parts controled by this GUC `temp_tablespaces`:
- temporary file location
- temporary table location

If there are more than one tablespace described in this GUC, a random one will be chosen.

In GPDB, we already provide `filespace`, and using the `gpfilespace --movetempfilespace`,
user can set temp file location. However, GPDB doesn't have an easy way to create temp
tables on different tablespaces.

It is valuable to add this feature in GPDB to honor the GUC `temp_tablespaces` partially
to change the temporary table location, but need significant validations to ensure
correctness like permissions, etc.
Signed-off-by: NXin Zhang <xzhang@pivotal.io>

Removed the overlapping information logged under log_checkpoints guc
and Debug_persistent_recovery_print guc.
Signed-off-by: NXin Zhang <xzhang@pivotal.io>

[ci skip]
Signed-off-by: NXin Zhang <xzhang@pivotal.io>

Use drop protocol cascade to drop custom protocols since the custom
protocol dependency is recorded in pg_depend.
Remove a couple of gpfdist tests since they are already covered in
installcheck.

In Postgres, PREPARE TRANSACTION errors out if it accessed a temp relation.

In GPDB, two phase commit is always used, even for transactions accessing temp
relations, which is used by MPP query processing.

In this case, GPDB treats temp table as regular distributed table, but shorten
its life cycle to sessions or transactions.

GPDB also stores temp table information in shared buffers, hence MyXactAccessedTempRel
is not used, and we are disabling it with comments. The reason to keep the code around
is to help future Postgres upstream merge.
Signed-off-by: NXin Zhang <xzhang@pivotal.io>

Originally, `CheckpointStartLock` prevent any MMXLOG insert between
collecting persistent table objects information
(mmxlog_append_checkpoint_data) through XLogInsert of checkpoint record.

Since `CheckpointStartLock` is removed from upstream, to prevent error
in MMXLOG replay on standby, we have to add protection back.

For the same reason, we used PersistentObjLock in SHARED mode to
prevent any update between mmxlog_append_checkpoint_data and XLogInsert
for checkpoint record.

Here is a broken scenario before this patch:
- Create standby like `gpinitstandby -s localhost -P 12345 -F pg_system:/tmp/standby/ps,fs1:/tmp/standby/fs1`
- Create two connections c1 and c2
- c2:
  - `create tablespace ts1 filespace fs1;`: double check the new folder created under filespace fs1 (`/tmp/standby/fs1`) on standby.
  - `begin;`
  - `drop tablespace ts1;`
- set breakpoint to connection c2 on function: `PersistentTablespace_Dropped`
- c2: `commit;`: At this moment, we are ready to write a XLOG for the tablespace drop, but haven't done so. We will let checkpoint record this extra tablespace.
- Switch to connection c1:
- set breakpoint to connection c1 on function: `getTwoPhasePreparedTransactionData`: This will pause the `checkpoint` right after we collect the persistent objects information for mmxlog.
- for lldb, make sure do `pro hand -p true -s false SIGINT` to pass the signal.
- Switch back to connection c2, and continue from the breakpoint `PersistentTablespace_Dropped`, and let it finish.
- Switch back to connection c1, and let the checkpoint finish.
- Now, check the `/tmp/standby/fs1`, you will see the left behind tablespace which is already deleted from master.
Signed-off-by: NXin Zhang <xzhang@pivotal.io>

GPDB still need MIRRORED_LOCK to be held to protect resync process
including:
- fileRepResyncShmem->appendOnlyCommitCount
- FileRepResyncManager_InResyncTransition()

Once the **File Replication** removed, we can remove the MIRRORED_LOCK.

The MirrorLock held exlusively at these places:
- `FileRepResyncManager_InResyncTransition()`
- `FileRepPrimary_RunResyncManager()`: where it used the lock to access
the counter `fileRepResyncShmem->appendOnlyCommitCount`.
Signed-off-by: NXin Zhang <xzhang@pivotal.io>

Originally, the return code was ignored. Now, if unlink failed for any
reason, the error message will be logged, and also adjust the
CheckpointStats properly.

Now MirroredFlatFile_Drop() returned the same return code as unlink(),
and also the errno is preserved, so the correct error message can be
logged by the caller.
Signed-off-by: NXin Zhang <xzhang@pivotal.io>

The legacy query planner might generate a plan that returns incorrect results
for window queries that have a subquery that contains a table valued function
and the function contains a non-correlated subquery. This commit fixes this issue.

Root Cause: Before the fix, `params_in_rtable` was searching for `paramids` in
unchanged `root->parse->rtable` instead of flattened `root->glob->finaltable`. This
was causing `paramids` not found and marking initPlans unused.
Signed-off-by: NBhunvesh Chaudhary <bchaudhary@pivotal.io>

Support for BINARY mode in COPY was removed some time ago but most
of the documentation in the ref page was kept; re-add the keyword
in the syntax description to reflect that support has been enabled
again.

Binary COPY was previously disabled in Greenplum, this commit
re-enables the binary mode by incorporating the upstream code
from PostgreSQL.

Patch by Github user alldefector with additional hacking by
Daniel Gustafsson