* fix gpload fial when capital letters in column name in merge mode

add double quotations for column names when create staging table
omit distributio
Co-authored-by: NXiaoxiaoHe <hxiaoxiao@vmware.com>

fix match column condition to resovle primary key conflict when using the gpload
merge mode to import data to the Multi-level partition table
fix fail when special char and capital letters in column names
Co-authored-by: NXiaoxiaoHe <hxiaoxiao@vmware.com>

MASTER_DATA_DIECTORY must be exported to before calling
gpconfig else it will fail to set the GUC

Allow setting DCA_VERSION_FILE to enable testing

Allow overriding the value of DCA_VERSION_FILE so that during testing it
can be manipulated. Also add a test for the same to ensure that DCA
configuration GUCs are set properly on the environment.

Commit 445fc7cc hardened some parts of analyzedb. However, it missed a
couple of cases.

1) When the statement to get the modcount from the pg_aoseg table failed
due to a dropped table, the transaction was also terminated. This caused
further modcount queries to fail and while those tables were analyzed,
it would error and not properly record the mod count. Therefore, we now
restart the transaction when it errors.

2) If the table is dropped and then recreated while analyzedb is running
(or some other mechanism that results in the table being successfully
analyzed, but the pg_aoseg table did not exist during the initial
check), the logic to update the modcount may fail. Now, we skip the
update for the table if this occurs. In this case, the modcount would
not be recorded and the next analyzedb run will consider the table
modified (or dirty) and re-analyze it, which is the desired behavior.

Add gpreload behave test

* gpreload test where a table (public.t1) and
  view (gpreload_schema.t1) have the same name.

Add gpreload CI test jobs.
Co-authored-by: NEd Espino <eespino@vmware.com>

* Coverity: Resource leak

Fix some resource leak.

(cherry picked from commit e27ec070)

* Coverity: Identical code for different branches

Clean identical code in heap.c and analyze.c

(cherry picked from commit f076205c)

* Coverity: Variable unused

Remove unused varaible.
For tuplesort.h, we doesn't support mksort based cluster,
so we should just set is_mk_tuplesortstate to false

(cherry picked from commit e5c86775)

* Coverity: Logically dead code

Remove dead code.
insertDesc is alwasy NULL in ao_vacuum_rel_compact()

(cherry picked from commit 38552bf7)

* Coverity: Sizeof not portable

sizeof(HeapTuple *) should be sizeof(HeapTuple)

(cherry picked from commit cf23db49)

* Coverity: Behave test misformat

conn = dbconn.connect() should be aligned with if statement, or it
will never be executed.

(cherry picked from commit 126ba1c6)

* Coverity: Check return value of strcmp

return value of strcmp is not checked in some branches.

(cherry picked from commit fd498cf9)

This reverts commit 7118e8ac.

The initialization file (passed as gpinitsystem -I <file>) can have two
formats: legacy (5-field) and new (6-field, that has the HOST_ADDRESS).

This commit fixes a bug in which an internal sorting routine that matched
a primary with its corresponding mirror assumed that <file> was always
in the new format.  The fix is to convert any input <file> to the new
format via re-writing the QD_ARRAY, PRIMARY_ARRAY and MIRROR_ARRAY to
have 6 fields.  We also always use '~' as the separator instead of ':'
for consistency.

The bug fixed is that a 5-field <file> was being sorted numerically,
causing either the hostname (on a multi-host cluster) or the port (on
a single-host cluster) to be used to sort instead or the content.
This could result in the primary and its corresponding mirror being
created on different contents, which fortunately hit an internal error
check.

Unit tests and a behave test have been added as well.  The behave test
uses a demo cluster to validate a legacy gpinitsystem initialization
file format (e.g. one that has 5 fields) successfully creates a
Greenplum database.
Co-authored-by: NDavid Krieger <dkrieger@vmware.com>

During testing, I encountered an incremental gprecoverseg hang issue.
Incremental gprecoverseg is based on pg_rewind.  pg_rewind launches a single
mode postgres process and quits after crash recovery if the postgres instance
was not cleanly shut down - this is used to ensure that the postgres is in a
consistent state before doing incremental recovery. I found that the single
mode postgres hangs with the below stack.

\#1  0x00000000008cf2d6 in PGSemaphoreLock (sema=0x7f238274a4b0, interruptOK=1 '\001') at pg_sema.c:422
\#2  0x00000000009614ed in ProcSleep (locallock=0x2c783c0, lockMethodTable=0xddb140 <default_lockmethod>) at proc.c:1347
\#3  0x000000000095a0c1 in WaitOnLock (locallock=0x2c783c0, owner=0x2cbf950) at lock.c:1853
\#4  0x0000000000958e3a in LockAcquireExtended (locktag=0x7ffde826aa60, lockmode=3, sessionLock=0 '\000', dontWait=0 '\000', reportMemoryError=1 '\001', locallockp=0x0) at lock.c:1155
\#5  0x0000000000957e64 in LockAcquire (locktag=0x7ffde826aa60, lockmode=3, sessionLock=0 '\000', dontWait=0 '\000') at lock.c:700
\#6  0x000000000095728c in LockSharedObject (classid=1262, objid=1, objsubid=0, lockmode=3) at lmgr.c:939
\#7  0x0000000000b0152b in InitPostgres (in_dbname=0x2c769f0 "template1", dboid=0, username=0x2c59340 "gpadmin", out_dbname=0x0) at postinit.c:1019
\#8  0x000000000097b970 in PostgresMain (argc=5, argv=0x2c51990, dbname=0x2c769f0 "template1", username=0x2c59340 "gpadmin") at postgres.c:4820
\#9  0x00000000007dc432 in main (argc=5, argv=0x2c51990) at main.c:241

It tries to hold the lock for template1 on pg_database with lockmode 3 but
it conflicts with the lock with lockmode 5 which was held by a recovered dtx
transaction in startup RecoverPreparedTransactions(). Typically the dtx
transaction comes from "create database" (by default the template database is
template1).

Fixing this by using the postgres database for single mode postgres execution.
The postgres database is commonly used in many background worker backends like
dtx recovery, gdd and ftsprobe. With this change, we do not need to worry
about "create database" with template postgres, etc since they won't succeed,
thus avoid the lock conflict.

We may be able to fix this in InitPostgres() by bypassing the locking code in
single mode but the current fix seems to be safer.  Note InitPostgres()
locks/unlocks some other catalog tables also but almost all of them are using
lock mode 1 (except mode 3 pg_resqueuecapability per debugging output).  It
seems that it is not usual in real scenario to have a dtx transaction that
locks catalog with mode 8 which conflicts with mode 1.  If we encounter this
later we need to think out a better (might not be trivial) solution for this.
For now let's fix the issue we encountered at first.

Note in this patch the code fixes in buildMirrorSegments.py and twophase.c are
not related to this patch. They do not seem to be strict bugs but we'd better
fix them to avoid potential issues in the future.
Reviewed-by: NAshwin Agrawal <aashwin@vmware.com>
Reviewed-by: NAsim R P <pasim@vmware.com>
(cherry picked from commit 288908f3)

Use yaml.safe_load rather than yaml.load as yaml.load is deprecated
Co-authored-by: NTyler Ramer <tramer@vmware.com>
Co-authored-by: NJamie McAtamney <jmcatamney@vmware.com>

Yaml was imported but unused in several locations.
gpMgmt/test/behave/mgmt_utils/steps/mgmt_utils.py had numerous unused
or duplicated imports.
Co-authored-by: NTyler Ramer <tramer@vmware.com>
Co-authored-by: NJamie McAtamney <jmcatamney@vmware.com>

It seems this yaml class is dead code. Removing it for this reason.
Co-authored-by: NTyler Ramer <tramer@vmware.com>
Co-authored-by: NJamie McAtamney <jmcatamney@vmware.com>

The version of PyYAML vendored in gpMgmt/bin/ext is old, unmaintained,
and does not support python3. Actually, it does not even contain a
`__version__` attribute, so it is not possible to know the version.

We need to unvendor yaml and address CVEs that have been found in the
library since the version vendored in source.

Also update yaml.load to use yaml.safe_load instead.
Co-authored-by: NTyler Ramer <tramer@vmware.com>
Co-authored-by: NJamie McAtamney <jmcatamney@vmware.com>

[Lockfile](https://pypi.org/project/lockfile/) has not been maintained
since around 2015. Further, the functionality it provided seems poor - a
review of the code indicated that it used the presence of the PID file
itself as the lock - in Unix, using a file's existence followed by a
creation is not atomic, so a lock could be prone to race conditions.

The lockfile package also did not clean up after itself - a process
which was destroyed unexpectedly would not clear the created locks, so
some faulty logic was added to mainUtils.py, which checked to see if a
process with the same PID as the lockfile's creator was running. This
is obviously failure prone, as a new process might be assigned the same
PID as the old lockfile's owner, without actually being the same process.

(Of note, the SIG_DFL argument to os.kill() is not a signal at all, but
rather of type signal.handler. It appears that the python cast this
handler to the int 0, which, according to man 2 kill, leads to no signal
being sent, but existance and permission checks are still performed. So
it is a happy accident that this code worked at all)

This commit removes lockfile from the codebase entirely.

It also adds a "PIDLockFile" class which provides an atomic-guarenteed
lock via the mkdir and rmdir commands on Unix - thus, it is not safely
portable to Windows, but this should not be an issue as only Unix-based
utilities use the "simple_main()" function.

PIDLockFile provides API compatible classes to replace most of the
functionality from lockfile.PidLockFile, but does remove any timeout
logic as it was not used in any meaningful sense - a hard-coded timeout
of 1 second was used, but an immediate result of if the lock is held is
sufficient.

PIDLockFile also includes appropriate __enter__, __exit__, and __del__
attributes, so that, should we extend this class in the future, with
syntax is functional, and __del__ calls release, so a process reaped
unexpectedly should still clean its own locks as part of the garbage
collection process.
Authored-by: NTyler Ramer <tramer@pivotal.io>

Do not remove PYLIB_SRC_EXT during make clean/distclean

Commit 8190ed40 removed lockfile from
mainUtils, but did not remove a reference to its source directory in the
make clean/distclean target. As a result, because LOCKFILE_DIR is no
longer defined, the make clean/distclean target removes the
PYLIB_SRC_EXT directory.

The entry in aocsseg table might be compacted and waiting for drop, so we
should use 'state' to filter the unused entry.

column 'vpinfo' in pg_aoseg.pg_aocsseg_xxx record the 'eof' of each attribute
in the AOCS table. Add a new check 'aoseg_table' in gpcheckcat, it checks the
number of attributes in 'vpinfo' is the same as the number of attributes in
'pg_attribute'. This check is performed in parallel and independently on each
segment, and it checks aoseg table and pg_attribute in different transaction,
so it should be run 'offline' to avoid false alarm.

Make some utilities search path safe, so it'll not calling any external
functions that has the same name with our built-in functions.

This fix does not guarantee to fix CVE-2018-1058.

Backport from 070d6221.

Co-authored-by: Jamie McAtamney jmcatamney@pivotal.io
Co-authored-by: Jacob Champion pchampion@pivotal.io
Co-authored-by: Shoaib Lari slari@pivotal.io

- generate-greenplum-path.sh use second parameter `yes` to generate
PYTHONHOME
- ONLY gpAux need to run this script with second parameter `yes`, since
in GPDB6 we always vendor python.
- developer build on their box won't be impacted by this change, since
regular build doesn't have second parameter, hence PYTHONHOME will be
skipped.

We added test-generate-greeplum-path.bash to verify the correctness of
the script.

[#173567726]
Co-authored-by: NBradford D. Boyle <bradfordb@vmware.com>
Co-authored-by: NTingFang Bao <bbob@vmware.com>
Co-authored-by: NXin Zhang <zhxin@vmware.com>
Co-authored-by: NBradford D. Boyle <bradfordb@vmware.com>
Co-authored-by: NTingFang Bao <bbob@vmware.com>

When cleaning up the master-only files on the new segments we used to do
the job one by one, when there are tens or hundreds of segments it can
be very slow.

Now we cleanup in parallel.

(cherry picked from commit 857763ae)
Reviewed-by: NZhenghua Lyu <zlv@pivotal.io>

Mirror registration passes through several steps at the moment:
1. CREATE_QE_ARRAY (QE_MIRROR_ARRAY is ordered by content)
2. ARRAY_REORDER (QE_MIRROR_ARRAY is ordered by port)
3. CREATE_ARRAY_SORTED_ON_CONTENT_ID (form QE_MIRROR_ARRAY_SORTED_ON_CONTENT_ID
   on a base of QE_MIRROR_ARRAY)
4. REGISTER_MIRRORS (walk through QE_MIRROR_ARRAY, register mirrors with
   pg_catalog.gp_add_segment_mirror on master's gp_segment_configuration
   and update QE_MIRROR_ARRAY with returned dbids)
5. CREATE_SEGMENT (walk through QE_MIRROR_ARRAY_SORTED_ON_CONTENT_ID with old
   dbids and create mirrors on segment hosts with pg_basebackup)
The problem is in a step 4 - we update the wrong array (QE_MIRROR_ARRAY instead
of QE_MIRROR_ARRAY_SORTED_ON_CONTENT_ID). Because of that we get inconsistency
between mirror dbids on gp_segment_configuration and internal.auto.conf files.
This can cause inoperable cluster state in some situations when we promote a
failed primary from a mirror with wrong dbids (FTS can't solve this issue).

Also fixed column indexes in array used for segment arrray ordering.
It was not done after commit https://github.com/greenplum-db/gpdb/commit/03c7d557720c5a78af1e2574ac385d10a0797f5e
which prepend array with new hostname column.
Co-authored-by: NVasiliy Ivanov <ivi@arenadata.io>
(cherry picked from commit 8d008792)

Setting `$PYTHONHOME` for releases where python is not vendored causes
the system python to fail with an error message about not being able to
find the site module.

This commit updates the logic in generate-greenplum_path.sh to check if
we are vendoring python and if we are, it includes setting `$PYTHONHOME`
in greenplum_path.sh. If we are vendoring python, then at build-time
`$PYTHONHOME` is set to point at our vendored python.

[#173046174]
Authored-by: NBradford D. Boyle <bradfordb@vmware.com>

[#173046174]
Authored-by: NBradford D. Boyle <bradfordb@vmware.com>

Authored-by: NTingfang Bao <baotingfang@gmail.com>

When upgrading from GPDB5 to GPDB6, gpupgrade will need to be able to call
binaries from both major versions. Relying on LD_LIBRARY_PATH is not an option
because this can cause binaries to load libraries from the wrong version.
Instead, we need the libraries to have RPATH/RUNPATH set correctly. Since the
built binaries may be relocated we need to use a relative path.

This commit disables the rpath configure option (which would result in an
absolute path) and use LDFLAGS to use `$ORIGIN`.

For most ELF files a RUNPATH of `$ORIGIN/../lib` is correct. For pygresql
python module and the quicklz_compressor extension, the RUNPATH needs to be
adjusted accordingly. The LDFLAGS for those artifacts can be modified with
different environment variables PYGRESQL_LDFLAGS and QUICKLZ_LDFLAGS.

We always use `--enable-new-dtags` to set RUNPATH. On CentOS 6, with new dtags,
both DT_RPATH and DT_RUNPATH are set and DT_RPATH will be ignored.

[#171588878]
Co-authored-by: NBradford D. Boyle <bboyle@pivotal.io>
Co-authored-by: NXin Zhang <xzhang@pivotal.io>

Following the [Greenplum Server RPM Packaging Specification][0], we need
to update greenplum_path.sh file, and ensure many environment variables
set correct.

There are a few basic requirments for Greenplum Path Layer:

* greenplum-path.sh shall be installed to `${installation
  prefix}/greenplum-db-[package-version]/greenplum_path.sh`
* ${GPHOME} is set by given parameter, by default it should point to
  `%{installation prefix}/greenplum-db-devel`
* `${LD_LIBRARY_PATH}` shall be safely set to avoid a trailing colon
  (which will cause the linker to search the current directory when
  resolving shared objects)
* `${PYTHONHOME}` shall be set to `${GPHOME}/ext/python`
* `${PYTHONPATH}` shall be set to `${GPHOME}/lib/python`
* `${PATH}` shall be set to `${GPHOME}/bin:${PYTHONHOME}/bin:${PATH}`
* If the file `${GPHOME}/etc/openssl.cnf` exists then `${OPENSSL_CONF}`
  shall be set to `${GPHOME}/etc/openssl.cnf`
* The greenplum_path.sh file shall pass [ShellCheck][1]

[0]: https://github.com/greenplum-db/greenplum-database-release/blob/master/Greenplum-Server-RPM-Packaging-Specification.md#detailed-package-behavior
[1]: https://github.com/koalaman/shellcheckCo-authored-by: NTingfang Bao <bbao@pivotal.io>
Co-authored-by: NXin Zhang <zhxin@vmware.com>
Co-authored-by: NNing Wu <ningw@vmware.com>
Co-authored-by: NShaoqi Bai <bshaoqi@vmware.com>

This reverts commit 87fef901.

When introducing a new mirror, we need two steps:
1. start mirror segment
2. update gp_segment_configuration catalog

Previously gp_add_segment_mirror will be called to update
the catalog, but dbid is chosen by get_availableDbId() which
cannot ensure to be the same dbid in internal.auto.conf.
Reported by issue9837
Reviewed-by: NPaul Guo <pguo@pivotal.io>
Reviewed-by: NBhuvnesh Chaudhary <bhuvnesh2703@gmail.com>

cherry-pick from commit: f7965d and 1ee999

while gpload is loading data if the configure file contains "error_table" and doesn't contain "preload", an error of no attribute "staging_table" or "fast_path" occurs.

gpload in the latest windows client package requires VS redistributable
package. Output more meaningful message if pg.py fails to load.

Previously, gpinitsystem did not allow the user to specify a hostname
and address for each segment in the input file used with -I; it only
accepted one value per segment and used it for both hostname and
address.

This commit changes the behavior so that the user can specify both
hostname and address. If the user specifies only the address (such as by
using an old config file), it will preserve the old behavior and set
both hostname and address to that value. It also adds a few tests around
input file parsing so SET_VAR is more resilient to further refactors.

The specific changes involved are the following:

1) Change SET_VAR to be able to parse either the old format (address
only) or new format (host and address) of the segment array
representation.
2) Move SET_VAR from gpinitsystem to gp_bash_functions.sh and remove the
redundant copy in gpcreateseg.sh.
3) Remove a hardcoded "~0" in QD_PRIMARY_ARRAY in gpinitsystem,
representing a replication port value, that was left over from 5X.
4) Improve the check for the number of fields in the segment array
representation.

Also, Remove use of ignore warning flag and use [[ ]] the check for IGNORE_WARNING

Previously, gpintsystem was incorrectly filling the hostname field of each
segment in gp_segment_configuration with the segment's address. This commit
changes it to correctly resolve hostnames and update the catalog accordingly.

This reverts commit 12ef7352, Revert "gpinitsystem: update catalog with
correct hostname".
   Commit message from 12ef7352:
   The commit requires some additional tweaks to the input file logic for
   backwards compatibility purposes, so we're reverting this until the full
   fix is ready.

This reverts commit e4add7cb.  The commit
requires some additional tweaks to the input file logic for backwards
compatibility purposes, so we're reverting this until the full fix is ready.

Previously, gpintsystem was incorrectly filling the hostname field of each
segment in gp_segment_configuration with the segment's address. This commit
changes it to correctly resolve hostnames and update the catalog accordingly.
Co-authored-by: NKalen Krempely <kkrempely@pivotal.io>

gpinitsystem did not quote the username while performing ALTER USER. When the username
is a numeric value the postgres parser gets upset - unless the username is quoted.

See here for more details:

https://www.postgresql.org/docs/9.4/sql-syntax-lexical.html#SQL-SYNTAX-IDENTIFIERS

- SQL identifiers and key words must begin with a letter (a-z, but also letters with
  diacritical marks and non-Latin letters) or an underscore (_).

- Also, there is a second kind of identifier: the delimited identifier or quoted
  identifier. It is formed by enclosing an arbitrary sequence of characters in
  double-quotes (")

- use variable interpolation provided by psql to properly quote user-provided values.

- use RETVAL to perform testing due to Commit d7b7a40aCo-authored-by: NJacob Champion <pchampion@pivotal.io>
(cherry picked from commit f188ecb5)

Previously, analyzedb would error out and fail if a table was dropped
during analyzedb. Now, we silently skip dropped tables when determining
the tables to analyze.

Previously, running analyzedb with an input file (`analyzedb -f
<config_file`) containing a root partition would fail as we did not
properly populate the list of leaf partitions. The logic in analyzedb
assumes that we enumerate leaf partitions from the root partition that
the user had input (either from the command line or from an input file).
While we did this properly when the table was passed in from the command
line, we looked for the table name rather than the schema-qualifed table
for input files.

This would cause partitioned heap tables to fail when writing the
report/status files at the end, and would cause analyzedb to not track
DML changes in partitioned AO tables. Now, we properly check for the
schema-qualified table name.

(cherry picked from commit d1611944)

Add max_retries flag for gpload. It indicates the max times on connecting to GPDB timed out.
max_retries default value is 0 which means no retry.
If max_retries is -1 or other negative value, it means retry forever.

Test has been done manually.

( cherry pick from master commit: b891b85b)

This fixes multiple occurrences of duplicated words in sentences, like
"the the" and "is is" etc.

Backported from master 1d44a0c5Reviewed-by: NMel Kiyama <mkiyama@pivotal.io>
Reviewed-by: NHeikki Linnakangas <hlinnakangas@pivotal.io>

1. When two gppkg packages have the same dependencies, gppkg utility
   will refuse to install the second gppkg package and throw an error.
   This patch fixes this issue and the second gppkg package can install
   successfully.

2. Fix install/uninstall issue if the master and standby master use the
   same node address.

PS: This patch is backported from the master branch