The dispatch code for set_config() incorrectly quoted the GUC value by
surrounding it in single quotes. Not only does this not handle more
"interesting" values containing embedded single quotes, but GUCs with
the GUC_LIST_QUOTE flag, such as search_path, ended up with different
values on the segments compared to the master. For example,

    SELECT set_config('search_path', 'my_schema,public', false);

was dispatched as

    SET search_path TO 'my_schema,public';

instead of

    SET search_path TO my_schema,public;

The two are not equivalent; the latter correctly sets search_path to a
list of two elements, "my_schema" and "public", whereas the former sets
search_path to a list containing a single "my_schema,public" element.

Rather than trying to parse apart the list value, dispatch the
set_config() call to the QEs as well. This lets us pass through the
(quoted) arguments directly and guarantees that we run the same code in
both places.
Co-authored-by: NShoaib Lari <slari@pivotal.io>

...as suggested by our master version of pgindent.

When generating plan for multi-DQA, we need to build a coplan on base of
the shared scan for each distinct DQA, and then join the coplans back
together. And while building the coplan for each DQA, the arrays for
RelOptInfo and RangeTblEntry for the PlannerInfo would be rebuild and
the original arrays would have been replaced. So, for the DQA other than
the first one, we need to restore the arrays to what they are like for
the original shared scan plan before building coplan for it.
Reviewed-by: NGeorgios Kokolatos <gkokolatos@pivotal.io>
Reviewed-by: Npengzhout <ptang@pivotal.io>

So far, we use to dispatch a guc when creating gang if its flag contains
GUC_GPDB_ADDOPT. It has some drawbacks.
1. This flag left a lots footprint in guc.c, when we diff upstream code.
2. It is easy to forget marking it when new guc introduce.

To deal with these issues, and also this PR is the first step of
[PR7184](https://github.com/greenplum-db/gpdb/pull/7184).

Rename GUC_GPDB_ADDOPT flag as GUC_GPDB_NEED_SYNC, as well as, declare a new
guc flag named GUC_GPDB_NO_SYNC. For the predefine guc in guc.c and guc_gp.c
must explicitly assign one of these two flags.
For custom guc, give a default flag, GUC_GPDB_NO_SYNC, if it does not
explicit set.

Instead of the direct set flag in guc.c, we define two array, sync_guc_names_array
and unsync_guc_names_array.To recode guc names into and automatically
assign flag when the guc object is built in Postgres. Hence, we can
reduce footprint when diffing with upstream guc.c

Idea from Heikki Linnakangas <hlinnakangas@pivotal.io>
Idea from Hubert Zhang <hzhang@pivotal.io>

For a query like:
SELECT ... ORDER BY key LIMIT n;

When the query running time is longer than UNACK_QUEUE_RING_LENGTH * TIMER_SPAN
(By default is 10 seconds), and the first tuple send from OrderBy node to QD is
not acknowledged, function putIntoUnackQueueRing will calculate an
inappropriate retransmission period, which is UNACK_QUEUE_RING_LENGTH *
TIMER_SPAN.

and add missing options

ADD CONSTRAINT USING INDEX is a new feature which didn't existing for 4
or 5 branches. This is taken from upstream commit eb7ed3f3. With
partition tables, the changes are not propagated to the leaf partitions
if the operation is done on the root table, thus disallowed currently.

This introduces rel_is_interior_partition to determine if a relation is
an interior node in a partition hierarchy.
Co-authored-by: NSoumyadeep Chakraborty <sochakraborty@pivotal.io>

The movedb_failure_callback existed to ensure that if copydir fails
inside movedb, it would clean up the destination directory. We
introduced the pending deletes mechanism to ensure cleanup throughout
2PC in b0208894.

These are two conflicting mechanisms, especially evident in the
following scenario:
1. An error occurs after the ScheduleDbDirDelete on the target
tablespace directory and inside the ENSURE block.
2. movedb_failure_callback() is invoked first which calls rmtree() on the
target tablespace directory.
3. DoPendingDbDeletes() is called from within the subsequent
AbortTransaction(). Since there was a pending db delete,
dropDatabaseDirectory() is called and it eventually calls rmtree().
Since the directory was already deleted at this point this rmtree
produces the following warnings (which led to a test failure in CI):
```
WARNING:  could not open directory <target_dboid_dir>: No such file or directory
WARNING:  some useless files may be left behind in old database directory <target_dboid_dir>
```

The movedb_failure_callback mechanism is totally unnecessary once we
can ensure that the pending deletes mechanism caters to copydir
failures. This patch takes care of that by moving the
ScheduleDbDirDelete call before the copydir.

Note: To minimize upstream diff, we replace the ENSURE block with a
standalone block.
Co-authored-by: NNing Yu <nyu@pivotal.io>

Commit 3b849ec0 changes USE_SSL macro to USE_OPENSSL, and provides
compatibility switch in pg_config_manual.h, but gpfdist doesn't
include it so USE_SSL macro is missing. Include pg_config_manual.h
to get it back.

For Result pathnode, its parent would be set to NULL. So we cannot
reference its parent in cost_common_agg() when estimating cost for
aggregation. Instead, we can use the rows in the Path and use 0 as the
width.

This patch fixes github issue #8357.

commit b5871009 tends to add that but the code is not right.

Reviewed-by: Ning Yu
Reviewed-by: Asim R P
Reviewed-by: Ashwin Agrawal

The build artifacts in gp-internal-artifacts are not immutable. If
`regexp` is used when fetching the resource, then new builds (with the
same version) are not detected. This means that fixes in the packaging
may not be correctly picked up by a pipeline.
Authored-by: NBradford D. Boyle <bboyle@pivotal.io>

Commit 738d919e7315581e1548e4ee83d6a38 removed enable_partition_rules
GUC processing, but didn't follow the bouncing ball to remove code
which was made obsolete from this.

This removes the postCreate member from the CreateStmt stuct, as well
as all places using it. With the postCreate entry on the List removed
due to being dead, it follows that the entry after it is dead too so
remove all doRuleStmt handling too.

Discussion: https://github.com/greenplum-db/gpdb/pull/8285
Reported-by: Jesse Zhang

As part of melting the MADlib CI snowflakes, this commit updates the
pipeline to pull these resources from the same GCS bucket that holds the
other GPDB6 build dependencies.
Authored-by: NBradford D. Boyle <bboyle@pivotal.io>

Co-authored-by: NDavid Kimura <dkimura@pivotal.io>

Output the following in pg_xlogdump for prepare transaction records:
* Global transaction ID
* Prepare time
* Tablespace OIDs:
	* 'tablespace_oid_to_delete_on_abort'
	* 'tablespace_oid_to_delete_on_commit'

These data come from TwoPhaseFileHeader structure which is written in
xlog in GPDB.

Move TwoPhaseFileHeader to `twophase.h` in order for that structure to
be used by `xact_desc_prepare()` in `xactdesc.c`
Co-Authored-By: NDavid Kimura <dkimura@pivotal.io>

If temp_tablespaces GUC is set at session level and gang resets, the
newly created gang on segments has value different from master. This
causes the incorrect behavior where temp files will not be created in
desired location. Fix the same by having temp_tablespaces GUC marked
with GUC_GPDB_ADDOPT.

libcurl is required by gpcloud, but there was no explicit dependency
on it in autoconf which would cause compiler errors in a build with
gpcloud but without libcurl. Fix by adding a similar dependencycheck
that we have for PXF.
Reviewed-by: NLav Jain <ljain@pivotal.io>
Reviewed-by: NJimmy Yih <jyih@pivotal.io>

The test injects a fault in StartTransaction().  This function is so
generic that any concurrent test may trigger the fault, leading to
failed or misleading outcome.  The test was using old faultinjector,
which emitted status of a fault as NOTICE message.  NOTICE messages do
not appear in isolation2 answer files.  So this problem went
undetected.

The intention behind the warning was to guide the test writer to avoid
flaky behavior due to FTS intervening in the middle of a PANIC to mark
a segment as down.

This is good, however, the way the warning is emitted is not correct.
The underlying assumption in the warning implementation (see
fbd0f091) is that InjectFault() function is executed by the same
process that created the gp_inject_fault extension.  This is not valid
in the new libpq based fault injector because the fault is injected by
a transient fault handler process, which is always different from the
process that creates the extension when running regression tests.

Evidently, there are several tests injecting 'panic' type fault
already and their answer files don't have such warnings.  The
bitmap_index test modified by this commit was the only test still
using old fault injector interface to inject a 'panic' fault.  In the
old fault injector, the above mentioned assumption was applicable and
the warning was emitted.  Now that we've moved to new implementation
wholesale, the warning must be removed.

We should go back to drawing board for emitting the FTS warning but
that's a separate patch.

Remove NOTICE messages that follow a gp_inject_fault() select statement.
Replace the boolean value with text 'Success:' returned by the new
interfae.

For reference, the following sed script was used to identify 't'
following a ' gp_inject_fault ' line:

/^ gp_inject_fault $/{
   $!{ N
       :again1
       N
       s/ t$/ Success:/
       t again1
     }
   }

These tests were invoking the new fault injector interface using its
temporary name - gp_inject_fault2().  Now the new interface is the
only one present in the repository, named as gp_inject_fault().

The new interface employs a special libpq connection parameter and a
libpq message to convey fault details to destination postmaster.  This
interface has been in use for some time now.  It was getting cumbersome
to maintain both the old as well as the new interface.

To be clear, we now have only one interface to inject fault and that is
"gp_inject_fault()".

Existing tests that were using gp_inject_fault2() will be updated in a
follow up commit.

This makes the signature of the new interface match that of the old
gp_inject_fault interface, so as to make it easy to replace the old interface
with the new one.

Re-ran the script with a small tweak which generated two more hits,
soon we should be caught up with all variations we have.

gpdb, we pre-execute init plan in ExecutorStart to reduce slice number.
However, for some initplans, which require params from the same level node,
should follow upstream execution flow.

To recognize this initplan's pattern, record `extParam` when creating
`subplan` object.

Fix issue: #6953

ExecChooseHashTableSize takes KB as parameter while global_work_mem returns bytes

Currently, ORCA does not emit LockRows plannode. So when
ORCA is enabled, we do not consider the query a simple
select-for-locking case.

If a test decides to fail because not all segments are running, it will
be nice to get the details of segments that are found to be not running.

The details printed by this patch will only be shown by behave upon a test
failure.  They will not be shown when a test passes.
Reviewed-by: NGeorgios Kokolatos <gkokolatos@pivotal.io>
Reviewed-by: NJacob Champion <pchampion@pivotal.io>
Reviewed-by: NDavid Krieger <dkrieger@pivotal.io>

When there are no bats testcases present, the current coding returned an
error due to the glob not being able to expand. This was made clear by
the recent removal of the last test, and the discussion in #8270. Rather
than using a glob, use ls and if statement. This allows us to keep the
unittest framework in place even if there are currently no tests.
Co-authored-by: NAmil Khanzada <akhanzada@pivotal.io>
Co-authored-by: NDaniel Gustafsson <dgustafsson@pivotal.io>

When reading the Behave test code for stopping segments, it is not obvious that
we are waiting for the segment to stop. Therefore we are explicitly adding the
`-w` flag to show the blocking behavior.
Co-authored-by: NNikolaos Kalampalikis <nkalampalikis@pivotal.io>

* Backport pg_strong_random

- Backport pg_strong_random related things from fe0a0b59.
  (https://git.postgresql.org/gitweb/?p=postgresql.git;a=commit;h=fe0a0b5993d)
  pg_strong_random() is added for better generation of random bytes.
  Only parts of fe0a0b59 has been ported back includes the c file and
  necessary makefile changes.

  By default, the function is enabled. It can be disabled by
  --disable-strong-random when configure.
  HAVE_STRONG_RANDOM is defined if the function is available.

- Backport a few OPENSSL related compile options from 680513ab.
  (https://git.postgresql.org/gitweb/?p=postgresql.git;a=commit;h=680513ab79c)
  This is to solve a few dependencies of pg_strong_random.
  A new macro USE_OPENSSL is introduced. It will be set to true when
  --with-openssl is configured. The old USE_SSL is still being used.
  When USE_OPENSSL defined, USE_SSL will be defined as well.

This is a dependency of #8211 for endpoint token generation.
Authored-by: NChen Mulong <muchen@pivotal.io>

Wait until we've inserted the fts_probe skip fault, AND we have
observed the fault being hit. This ensures that we've completed the
in-progress fts probe before continuing on with a test.

Previously, if we don't wait and our test continues forward to inject
a panic, then then in-progress fts probe would do its job and promote
the mirror.

We lower the minimum allowed gp_fts_probe_interval to 1 second, which
allows us to configure the interval to one second during our tests and
speeds up the tests significantly.
Reported-by: NAsim R P <apraveen@pivotal.io>

pstate->raw_buf_len should never be less than zero because it's the
length of unprocessed data in externalgettup_custom(). Add an assert to
make sure this.

We should probably try and get this one right.

This patch addresses two main scenarios:
1) Allowing multiple probes both internal and external to reuse the same
results when appropriate (ie: piggybacking on previous results). Multiple
requests should share the same results if they all request before the start of
a new fts loop, and after the results of the previous probe.

2) Ensuring fresh results from an external probe. When a request occurs during
a current probe in progress, this request should get fresh results rather
"piggybacking" or using the current results.

We use similar logic as the checkpointer code to detect whether a probe is in
progress with a probe start tick and probe end tick. To request a probe, we
send a signal requesting a fts results, then wait for a new loop to start,
then wait again for that current loop to finish. This implementation uses a
busy wait loop, which includes a short sleep. In the future, we can
leverage the upstream conditaion variable implementation which enables us
to signal multiple fts notify processes.
Co-authored-by: NSoumyadeep Chakraborty <sochakraborty@pivotal.io>
Co-authored-by: NKalen Krempely <kkrempely@pivotal.io>
Co-authored-by: NDavid Krieger <dkrieger@pivotal.io>
Co-authored-by: NTaylor Vesely <tvesely@pivotal.io>
Co-Authored-by: NAlexandra Wang <lewang@pivotal.io>
Co-Authored-by: NJimmy Yih <jyih@pivotal.io>

As a continued follow-up in the CVE-2018-1058, ensure that the utilities
that connect using PyGreSQL primitives also have search_path protection.

We set an empty search_path for the connection in all cases except for
segment_reconfigurer.py, where a fully-qualified SELECT statement seemed
sufficient (there is only one query being performed on the connection,
and that seems unlikely to change).
Co-authored-by: NJamie McAtamney <jmcatamney@pivotal.io>
Co-authored-by: NJacob Champion <pchampion@pivotal.io>
Co-authored-by: NShoaib Lari <slari@pivotal.io>