Now that we do not have to use full gang for distributed transaction, that
makes in-progress distributed transaction on master might be greater than
max_prepared_xacts if max_prepared_xacts is configured with a small value.
max_prepared_xacts is used to as the inProgressXidArray length for distributed
snapshot. This might lead to distributed snapshot creation failure due to "Too
many distributed transactions for snapshot" if the system is in heavy 2pc load.
Fixing this by using GetMaxSnapshotXidCount() for the length of array
inProgressXidArray, following the setting on master.

This fixes github issue https://github.com/greenplum-db/gpdb/issues/10057

No test for this since test isolation2:prepare_limit already covered this.  (I
encountered this issue when backporting a PR that introduces the test
isolation2:prepare_limit, so need to push this at first then the backporting
PR).
Reviewed-by: NHubert Zhang <hzhang@pivotal.io>

Greenplum use unique row id path as a candidate to implement semijoin.
It is introduced long before. But GPDB6 has upgraded the kernel
version to Postgres 9.4 and introduced many new path types and
new plan nodes, thus cdbpath_dedup_fixup failed to consider them.
Some typical issues are: https://github.com/greenplum-db/gpdb/issues/9427

On Master branch, Heikki's commit 9628a332 refactored this part of code
so that it is OK on master. And for 4X and 5X, we do not have many new
kinds of plannode and pathnode, it is also OK.

It is very hard to backport commit 9628a332 to 6X, there is no concept of
a Path's target list in 9.4. And to totally remove this kind of path
is too overkilling. So the policy is to fix them one bye one if reported.

Enlarge the sleep time for a query which will be canceled later
to avoid slow execution fails the test.

Normally the test's running time should not get affected since the
sleep query will get terminate immediately.

(cherry picked from commit bbccf20c)

XLogReaderAllocate returns NULL if the xlogreader couldn't be
allocated. This NULL checks were forgotten in several places of
twophase.c and caused segmentation faults under heavy workloads.

Oracle Linux is compiled from Red Hat Enterprise Linux (RHEL) source code,
replacing Red Hat branding with Oracle's[1].

The ICW jobs for oracle7 consume the GPDB binary compiled from centos7.

[1]: Wiki https://en.wikipedia.org/wiki/Oracle_Linux

Notably we want the shared snapshot dumping information when encountering the
"snapshot collision" error, which was seen on real scenario and it is hard to
debug.

(cherry picked from commit ee2d4641)

We've seen such a case on a stable release but it is hard to debug via the
message only, so let's provide more details in the error message.

Our implementations of memory pools have a hidden dependency on _the_
global memory pool manager: typically GPOS_NEW and GPOS_DELETE will
reach for the memory pool manager singleton. This makes GPOS_DELETE on a
memory pool manager undefined behavior because we call member functions
on an object after its destructor finishes.

On the Postgres 12 merge branch, this manifests itself in a crash during
initdb. More concerning is that it only crashed when we set max
connections and shared buffers to a specific number.

* docs - update utility docs with IP/hostname information.

Add information to gpinitsystem, gpaddmirrors, and gpexpand ref. docs
--Information about using hostnames vs. IP addresses
--Information about configuring hosts that are configured with mulitple NICs

Also updated some examples in gpinitsystem

* docs - review comment updates. Add more information from dev.

* docs - change examples to show valid configurations that support failorver.
Also fix typos and minor edits.

* docs - updates based on review comments.

The failed test case is to test the command "copy lineitem to '/tmp/abort.csv'"
can be cancelled after COPY is dispatched to QEs. To verify this, it checks that
/tmp/abort.csv has fewer rows than lineitem.

The cancel logical in codes is:

QD dispatched the COPY command to QEs, then if QD get a cancel interrupt, it
sends a cancel request to QEs, however, the QD will keep receiving data from
QEs even QD already get a cancel interrupt. QD relies on QEs to receive the
cancel request and explicitly stop copying data to QD.

Obviously, QEs may already have copied out all data to QDs before they
get cancel requests, so the test case cannot guarantee /tmp/aborted.csv
has fewer rows than lineitem.

To fix this, we just verify the COPY command can be aborted with message
'ERROR:  canceling statement due to user request', the count
verification looks pointless here.

It's cherry-pick of 9480d631 from master

Pull out the implementation for binary heap into its own templated h
file.

Prior to this commit, merging two histograms was not commutative.
Meaning histogram1->Union(histogram2) could result in a row estimate of
1500 rows, but histogram2->Union(histogram1) could result in a row
estimate of 600 rows.

Now, MakeBucketMerged has been renamed to SplitAndMergeBuckets. This
function, which calculates the statistics for the merged bucket, now
consistently return the same histogram buckets regardless of the order
of input. This in turn, makes MakeUnionHistogramNormalize and
MakeUnionAllHistogramNormalize commutative.

Once we have successfully split the buckets and merged them as
necessary, we may have generated up to 3X the number of buckets that
were originally present. Thus we cap the number of buckets to be either
the max size of the two incoming buckets, or, 100 buckets

CombineBuckets will then reduce the size of the histogram by combining
consecutive buckets that have similar information. It does this by using
a combination of two ratios: freq/ndv and freq/bucket_width. These two
ratios were decided based off the following examples:

Assuming that we calculate row counts for selections like the following:
- For a predicate col = const: rows * freq / NDVs
- For a predicate col < const: rows * (sum of full or fractional frequencies)

Example 1 (rows = 100), freq/width, ndvs/width and ndvs/freq are all the same:
  ```
  Bucket 1: [0, 4)   freq .2  NDVs 2  width 4  freq/width = .05 ndv/width = .5 freq/ndv = .1
  Bucket 2: [4, 12)  freq .4  NDVs 4  width 8  freq/width = .05 ndv/width = .5 freq/ndv = .1
  Combined: [0, 12)  freq .6  NDVs 6  width 12
  ```

This should give the same estimates for various predicates, with separate or combined buckets:
```
pred          separate buckets         combined bucket   result
-------       ---------------------    ---------------   -----------
col = 3  ==>  100 * .2 / 2           = 100 * .6 / 6    = 10 rows
col = 5  ==>  100 * .4 / 4           = 100 * .6 / 6    = 10 rows
col < 6  ==>  100 * (.2 + .25 * .4)  = 100 * .5 * .6   = 30 rows
```

Example 2 (rows = 100), freq and ndvs are the same, but width is different:
```
Bucket 1: [0, 4)   freq .4  NDVs 4  width 4  freq/width = .1 ndv/width = 1 freq/ndv = .1
Bucket 2: [4, 12)  freq .4  NDVs 4  width 8  freq/width = .05 ndv/width = .5 freq/ndv = .1
Combined: [0, 12)  freq .8  NDVs 8  width 12
```

This will give different estimates with the combined bucket, but only for non-equal preds:
```
pred          separate buckets         combined bucket   results
-------       ---------------------    ---------------   --------------
col = 3  ==>  100 * .4 / 4           = 100 * .8 / 8    = 10 rows
col = 5  ==>  100 * .4 / 4           = 100 * .8 / 8    = 10 rows
col < 6  ==>  100 * (.4 + .25 * .4) != 100 * .5 * .8     50 vs. 40 rows
```

Example 3 (rows = 100), now NDVs / freq is different:
```
Bucket 1: [0, 4)   freq .2  NDVs 4  width 4  freq/width = .05 ndv/width = 1 freq/ndv = .05
Bucket 2: [4, 12)  freq .4  NDVs 4  width 8  freq/width = .05 ndv/width = .5 freq/ndv = .1
Combined: [0, 12)  freq .6  NDVs 8  width 12
```

This will give different estimates with the combined bucket, but only for equal preds:
```
pred          separate buckets         combined bucket   results
-------       ---------------------    ---------------   ---------------
col = 3  ==>  100 * .2 / 4          != 100 * .6 / 8      5 vs. 7.5 rows
col = 5  ==>  100 * .4 / 4          != 100 * .8 / 8      10 vs. 7.5 rows
col < 6  ==>  100 * (.2 + .25 * .4)  = 100 * .5 * .6   = 30 rows
```

This commit also adds an attribute to the statsconfig for MaxStatsBuckets
and changes the scaling method when creating singleton buckets.

MergeHistogramMapsforDisjPreds

This commit refactors MakeStatsFilter to use
MakeHistHashMapConjOrDisjFilter instead of individually calling
MakeHistHashMapConj and MakeHistHashMapDisj.

This commit also modifies MergeHistogramMapsForDisjPreds to avoid copy
and creating unnecessary histogram buckets.

* docs - add information for SSL with standby master

--SSL file should not be in $MASTER_DATA_DIRECTORY

Also
--Add not about not using NULL ciphers
--Correct default directory for SSL files to $MASTER_DATA_DIRECTORY

* docs - review comment updates

Commit 0b2c7325 into 6X Stable, first
added in PR #10451, did not require the git submodule update on Windows
because this has previously been added with Pygresql - this caused a
build failure for windows clients because PyYAML source was not found.

Adding the git submodule pull should resolve this
Authored-by: NTyler Ramer <tramer@vmware.com>

Local fork at gpMgmt/bin/ext/yaml was removed by
03960e45333c3a7d8fe677b9015ce2a7c33c502b. Unpack
it from gpMgmt/bin/pythonSrc/ext just like pygresql.

Use yaml.safe_load rather than yaml.load as yaml.load is deprecated
Co-authored-by: NTyler Ramer <tramer@vmware.com>
Co-authored-by: NJamie McAtamney <jmcatamney@vmware.com>

Yaml was imported but unused in several locations.
gpMgmt/test/behave/mgmt_utils/steps/mgmt_utils.py had numerous unused
or duplicated imports.
Co-authored-by: NTyler Ramer <tramer@vmware.com>
Co-authored-by: NJamie McAtamney <jmcatamney@vmware.com>

It seems this yaml class is dead code. Removing it for this reason.
Co-authored-by: NTyler Ramer <tramer@vmware.com>
Co-authored-by: NJamie McAtamney <jmcatamney@vmware.com>

The version of PyYAML vendored in gpMgmt/bin/ext is old, unmaintained,
and does not support python3. Actually, it does not even contain a
`__version__` attribute, so it is not possible to know the version.

We need to unvendor yaml and address CVEs that have been found in the
library since the version vendored in source.

Also update yaml.load to use yaml.safe_load instead.
Co-authored-by: NTyler Ramer <tramer@vmware.com>
Co-authored-by: NJamie McAtamney <jmcatamney@vmware.com>

Currently GPDB tries to pull up EXPR sublinks to inner joins. For query

select * from foo where foo.a >
    (select avg(bar.a) from bar where foo.b = bar.b);

GPDB would transform it to:

select * from foo inner join
    (select bar.b, avg(bar.a) as avg from bar group by bar.b) sub
on foo.b = sub.b and foo.a > sub.avg;

To do that, GPDB needs to recurse through the quals in sub-select and
extract quals of form 'outervar = innervar' and then build new
SortGroupClause items and TargetEntry items based on these quals for
sub-select.

But for quals of form 'function(outervar, innervar1) = innvervar2', GPDB
handles them incorrectly and will cause wrong results issues as
described in issue #9615.

This patch fixes this issue by treating these kinds of quals as not
compatible correlated and thus the sub-select would not be converted to
join.
Reviewed-by: NHeikki Linnakangas <hlinnakangas@pivotal.io>
Reviewed-by: NAsim R P <apraveen@pivotal.io>
(cherry picked from commit dcdc6c0b)

Remove the set `gp_fts_probe_retries to 1` which may cause FTS probe failed.
This was first added to reduce the test time, but set a lower retry
value may cause the test failed to probe FTS update segment
configuration. Since reduce the `gp_fts_replication_attempt_count` also
save the test time, so skip alter ``gp_fts_probe_retries`.

Also find an assertion may not match when mark mirror down happens before
walsender exit, which will free the replication status before walsender
exit and try to record disconnect time and failure count. Which lead the
segment crash and starts recover.

When alter table add a column to AOCS table, the storage setting (compresstype,
compresslevel and blocksize) of the new column can be specified in the ENCODING
clause; it inherits the setting from the table if ENCODING is not specified; it
will use the value from GUC 'gp_default_storage_options' when the table dosen't
have the compression configuration.

Currently, replicated tables are not allowed to inherit a parent
table. But ALTER TABLE .. INHERIT can pass around the restriction.

On the other hand, a replicated table is allowed to be inherited
by a hash distributed table. It makes things much complicated.
When the parent table is declared as a replicated table inherited by
a hash distributed table, its data on the parent is replicated
but the data on the child is hash distributed. When running
`select * from parent;`, the generated plan is:
```
gpadmin=# explain select * from parent;
                                 QUERY PLAN
-----------------------------------------------------------------------------
 Gather Motion 3:1  (slice1; segments: 3)  (cost=0.00..4.42 rows=14 width=6)
   ->  Append  (cost=0.00..4.14 rows=5 width=6)
         ->  Result  (cost=0.00..1.20 rows=4 width=7)
               One-Time Filter: (gp_execution_segment() = 1)
               ->  Seq Scan on parent  (cost=0.00..1.10 rows=4 width=7)
         ->  Seq Scan on child  (cost=0.00..3.04 rows=2 width=4)
 Optimizer: Postgres query optimizer
(7 rows)
```
It's not particularly useful for the parent table to be replicated.
So, we disallow the replicated table to be inherited.
Reported-by: NHeikki Linnakangas <hlinnakangas@pivotal.io>
Reviewed-by: NHubert Zhang <hzhang@pivotal.io>
(cherry picked from commit dc4b839e)

When there is a big lag between primary and mirror replay, gp_replica_check
will fail if the checkpoint is not replayed in about 60 seconds. Extend the
timeout to 600 seconds to reduce the chance of flaky.

[Lockfile](https://pypi.org/project/lockfile/) has not been maintained
since around 2015. Further, the functionality it provided seems poor - a
review of the code indicated that it used the presence of the PID file
itself as the lock - in Unix, using a file's existence followed by a
creation is not atomic, so a lock could be prone to race conditions.

The lockfile package also did not clean up after itself - a process
which was destroyed unexpectedly would not clear the created locks, so
some faulty logic was added to mainUtils.py, which checked to see if a
process with the same PID as the lockfile's creator was running. This
is obviously failure prone, as a new process might be assigned the same
PID as the old lockfile's owner, without actually being the same process.

(Of note, the SIG_DFL argument to os.kill() is not a signal at all, but
rather of type signal.handler. It appears that the python cast this
handler to the int 0, which, according to man 2 kill, leads to no signal
being sent, but existance and permission checks are still performed. So
it is a happy accident that this code worked at all)

This commit removes lockfile from the codebase entirely.

It also adds a "PIDLockFile" class which provides an atomic-guarenteed
lock via the mkdir and rmdir commands on Unix - thus, it is not safely
portable to Windows, but this should not be an issue as only Unix-based
utilities use the "simple_main()" function.

PIDLockFile provides API compatible classes to replace most of the
functionality from lockfile.PidLockFile, but does remove any timeout
logic as it was not used in any meaningful sense - a hard-coded timeout
of 1 second was used, but an immediate result of if the lock is held is
sufficient.

PIDLockFile also includes appropriate __enter__, __exit__, and __del__
attributes, so that, should we extend this class in the future, with
syntax is functional, and __del__ calls release, so a process reaped
unexpectedly should still clean its own locks as part of the garbage
collection process.
Authored-by: NTyler Ramer <tramer@pivotal.io>

Do not remove PYLIB_SRC_EXT during make clean/distclean

Commit 8190ed40 removed lockfile from
mainUtils, but did not remove a reference to its source directory in the
make clean/distclean target. As a result, because LOCKFILE_DIR is no
longer defined, the make clean/distclean target removes the
PYLIB_SRC_EXT directory.

Previously, the PdrgpcrsAddEquivClass function would modify the input
colref set. This does not appear intentional, as this same reference may
be accessed in other places. This caused Orca to fall back to planner in
some cases during translation with "Attribute number 0 not found in
project list".
Co-authored-by: Nmubo.fy <mubo.fy@alibaba-inc.com>
Co-authored-by: NChris Hajas <chajas@pivotal.io>
Co-authored-by: NHans Zeller <hzeller@vmware.com>
(cherry picked from commit 84c027afdff13f1d91447a5a88809f6e85399a1b)

The entry in aocsseg table might be compacted and waiting for drop, so we
should use 'state' to filter the unused entry.

column 'vpinfo' in pg_aoseg.pg_aocsseg_xxx record the 'eof' of each attribute
in the AOCS table. Add a new check 'aoseg_table' in gpcheckcat, it checks the
number of attributes in 'vpinfo' is the same as the number of attributes in
'pg_attribute'. This check is performed in parallel and independently on each
segment, and it checks aoseg table and pg_attribute in different transaction,
so it should be run 'offline' to avoid false alarm.

When ExecReScanBitmapHeapScan get executed, bitmap state (tbmiterator
and tbmres) gets freed in freeBitmapState. So the tbmres is NULL, and we
need to reinit bitmap state to start scan from the beginning and reset AO/AOCS
bitmap pages' flags(baos_gotpage, baos_lossy, baos_cindex and baos_ntuples).

Especially when ExecReScan happens on the bitmap append only scan and
not all the matched tuples in bitmap are consumed, for example, Bitmap
Heap Scan as inner plan of the Nest Loop Semi Join. If tbmres not get init,
and not read all tuples in last bitmap, BitmapAppendOnlyNext will assume the
current bitmap page still has data to return. but bitmap state already freed.

From the code, for Nest Loop Semi Join, when a match find, a new outer slot is
requested, and then `ExecReScanBitmapHeapScan` get called, `node->tbmres` and
`node->tbmiterator` set to NULL. `node->baos_gotpage` still keeps true.
When execute `BitmapAppendOnlyNext`, it skip create new `node->tbmres`.
And jump to access `tbmres->recheck`.
Reviewed-by: NJinbao Chen <jinchen@pivotal.io>
Reviewed-by: NAsim R P <pasim@vmware.com>
(cherry picked from commit cb5d18d1)

Make some utilities search path safe, so it'll not calling any external
functions that has the same name with our built-in functions.

This fix does not guarantee to fix CVE-2018-1058.

Backport from 070d6221.

Co-authored-by: Jamie McAtamney jmcatamney@pivotal.io
Co-authored-by: Jacob Champion pchampion@pivotal.io
Co-authored-by: Shoaib Lari slari@pivotal.io

The path constructed in OpenAOSegmentFile() didn't take into account
"t_" semantic of filename. Ideally, the correct filename is passed to
function, so no need to construct the same.

Would be better if can move MakeAOSegmentFileName() inside
OpenAOSegmentFile(), as all callers call it except
truncate_ao_perFile(), which doesn't fit that model.

(cherry picked from commit 327abdb5)

* docs - gphdfs2pxf migration pxf supports avro compression

* missing plural

We've moved the repo that holds trigger commits to a private repo since
there wasn't anything interesting there.

- generate-greenplum-path.sh use second parameter `yes` to generate
PYTHONHOME
- ONLY gpAux need to run this script with second parameter `yes`, since
in GPDB6 we always vendor python.
- developer build on their box won't be impacted by this change, since
regular build doesn't have second parameter, hence PYTHONHOME will be
skipped.

We added test-generate-greeplum-path.bash to verify the correctness of
the script.

[#173567726]
Co-authored-by: NBradford D. Boyle <bradfordb@vmware.com>
Co-authored-by: NTingFang Bao <bbob@vmware.com>
Co-authored-by: NXin Zhang <zhxin@vmware.com>
Co-authored-by: NBradford D. Boyle <bradfordb@vmware.com>
Co-authored-by: NTingFang Bao <bbob@vmware.com>

Force to update checkpoint timeline id in control data after promotion and before pg_rewind. (#10402)

After promotion there is a checkpoint but not an immediate one, so the
checkpoint timeline id in pg_control data might not be updated in time when
running pg_rewind which depends on that to decide the necessity of incremental
recovery.  Let's run checkpoint after promotion in tests so that later
pg_rewind will not bypass the incremental recovery.

This should be able to fix some long-existing annoying flaky pg_rewind tests.
Reviewed-by: N(Jerome)Junfeng Yang <jeyang@pivotal.io>
(cherry picked from commit e072dff7)

This reverts commit b0afc59b.

--Also, add some links to other topics.