- 07 9月, 2020 1 次提交
-
-
由 xiaoxiao 提交于
* fix gpload fial when capital letters in column name in merge mode add double quotations for column names when create staging table omit distributio Co-authored-by: NXiaoxiaoHe <hxiaoxiao@vmware.com>
-
- 03 9月, 2020 1 次提交
-
-
由 xiaoxiao 提交于
fix match column condition to resovle primary key conflict when using the gpload merge mode to import data to the Multi-level partition table fix fail when special char and capital letters in column names Co-authored-by: NXiaoxiaoHe <hxiaoxiao@vmware.com>
-
- 02 9月, 2020 1 次提交
-
-
由 Bhuvnesh Chaudhary 提交于
MASTER_DATA_DIECTORY must be exported to before calling gpconfig else it will fail to set the GUC Allow setting DCA_VERSION_FILE to enable testing Allow overriding the value of DCA_VERSION_FILE so that during testing it can be manipulated. Also add a test for the same to ensure that DCA configuration GUCs are set properly on the environment.
-
- 25 8月, 2020 1 次提交
-
-
由 Chris Hajas 提交于
Commit 445fc7cc hardened some parts of analyzedb. However, it missed a couple of cases. 1) When the statement to get the modcount from the pg_aoseg table failed due to a dropped table, the transaction was also terminated. This caused further modcount queries to fail and while those tables were analyzed, it would error and not properly record the mod count. Therefore, we now restart the transaction when it errors. 2) If the table is dropped and then recreated while analyzedb is running (or some other mechanism that results in the table being successfully analyzed, but the pg_aoseg table did not exist during the initial check), the logic to update the modcount may fail. Now, we skip the update for the table if this occurs. In this case, the modcount would not be recorded and the next analyzedb run will consider the table modified (or dirty) and re-analyze it, which is the desired behavior.
-
- 18 8月, 2020 1 次提交
-
-
由 Abhijit Subramanya 提交于
Add gpreload behave test * gpreload test where a table (public.t1) and view (gpreload_schema.t1) have the same name. Add gpreload CI test jobs. Co-authored-by: NEd Espino <eespino@vmware.com>
-
- 07 8月, 2020 1 次提交
-
-
由 Hubert Zhang 提交于
* Coverity: Resource leak Fix some resource leak. (cherry picked from commit e27ec070) * Coverity: Identical code for different branches Clean identical code in heap.c and analyze.c (cherry picked from commit f076205c) * Coverity: Variable unused Remove unused varaible. For tuplesort.h, we doesn't support mksort based cluster, so we should just set is_mk_tuplesortstate to false (cherry picked from commit e5c86775) * Coverity: Logically dead code Remove dead code. insertDesc is alwasy NULL in ao_vacuum_rel_compact() (cherry picked from commit 38552bf7) * Coverity: Sizeof not portable sizeof(HeapTuple *) should be sizeof(HeapTuple) (cherry picked from commit cf23db49) * Coverity: Behave test misformat conn = dbconn.connect() should be aligned with if statement, or it will never be executed. (cherry picked from commit 126ba1c6) * Coverity: Check return value of strcmp return value of strcmp is not checked in some branches. (cherry picked from commit fd498cf9)
-
- 03 8月, 2020 1 次提交
-
- 01 8月, 2020 1 次提交
-
-
由 bhuvnesh chaudhary 提交于
The initialization file (passed as gpinitsystem -I <file>) can have two formats: legacy (5-field) and new (6-field, that has the HOST_ADDRESS). This commit fixes a bug in which an internal sorting routine that matched a primary with its corresponding mirror assumed that <file> was always in the new format. The fix is to convert any input <file> to the new format via re-writing the QD_ARRAY, PRIMARY_ARRAY and MIRROR_ARRAY to have 6 fields. We also always use '~' as the separator instead of ':' for consistency. The bug fixed is that a 5-field <file> was being sorted numerically, causing either the hostname (on a multi-host cluster) or the port (on a single-host cluster) to be used to sort instead or the content. This could result in the primary and its corresponding mirror being created on different contents, which fortunately hit an internal error check. Unit tests and a behave test have been added as well. The behave test uses a demo cluster to validate a legacy gpinitsystem initialization file format (e.g. one that has 5 fields) successfully creates a Greenplum database. Co-authored-by: NDavid Krieger <dkrieger@vmware.com>
-
- 22 7月, 2020 1 次提交
-
-
由 Paul Guo 提交于
During testing, I encountered an incremental gprecoverseg hang issue. Incremental gprecoverseg is based on pg_rewind. pg_rewind launches a single mode postgres process and quits after crash recovery if the postgres instance was not cleanly shut down - this is used to ensure that the postgres is in a consistent state before doing incremental recovery. I found that the single mode postgres hangs with the below stack. \#1 0x00000000008cf2d6 in PGSemaphoreLock (sema=0x7f238274a4b0, interruptOK=1 '\001') at pg_sema.c:422 \#2 0x00000000009614ed in ProcSleep (locallock=0x2c783c0, lockMethodTable=0xddb140 <default_lockmethod>) at proc.c:1347 \#3 0x000000000095a0c1 in WaitOnLock (locallock=0x2c783c0, owner=0x2cbf950) at lock.c:1853 \#4 0x0000000000958e3a in LockAcquireExtended (locktag=0x7ffde826aa60, lockmode=3, sessionLock=0 '\000', dontWait=0 '\000', reportMemoryError=1 '\001', locallockp=0x0) at lock.c:1155 \#5 0x0000000000957e64 in LockAcquire (locktag=0x7ffde826aa60, lockmode=3, sessionLock=0 '\000', dontWait=0 '\000') at lock.c:700 \#6 0x000000000095728c in LockSharedObject (classid=1262, objid=1, objsubid=0, lockmode=3) at lmgr.c:939 \#7 0x0000000000b0152b in InitPostgres (in_dbname=0x2c769f0 "template1", dboid=0, username=0x2c59340 "gpadmin", out_dbname=0x0) at postinit.c:1019 \#8 0x000000000097b970 in PostgresMain (argc=5, argv=0x2c51990, dbname=0x2c769f0 "template1", username=0x2c59340 "gpadmin") at postgres.c:4820 \#9 0x00000000007dc432 in main (argc=5, argv=0x2c51990) at main.c:241 It tries to hold the lock for template1 on pg_database with lockmode 3 but it conflicts with the lock with lockmode 5 which was held by a recovered dtx transaction in startup RecoverPreparedTransactions(). Typically the dtx transaction comes from "create database" (by default the template database is template1). Fixing this by using the postgres database for single mode postgres execution. The postgres database is commonly used in many background worker backends like dtx recovery, gdd and ftsprobe. With this change, we do not need to worry about "create database" with template postgres, etc since they won't succeed, thus avoid the lock conflict. We may be able to fix this in InitPostgres() by bypassing the locking code in single mode but the current fix seems to be safer. Note InitPostgres() locks/unlocks some other catalog tables also but almost all of them are using lock mode 1 (except mode 3 pg_resqueuecapability per debugging output). It seems that it is not usual in real scenario to have a dtx transaction that locks catalog with mode 8 which conflicts with mode 1. If we encounter this later we need to think out a better (might not be trivial) solution for this. For now let's fix the issue we encountered at first. Note in this patch the code fixes in buildMirrorSegments.py and twophase.c are not related to this patch. They do not seem to be strict bugs but we'd better fix them to avoid potential issues in the future. Reviewed-by: NAshwin Agrawal <aashwin@vmware.com> Reviewed-by: NAsim R P <pasim@vmware.com> (cherry picked from commit 288908f3)
-
- 15 7月, 2020 4 次提交
-
-
由 Tyler Ramer 提交于
Use yaml.safe_load rather than yaml.load as yaml.load is deprecated Co-authored-by: NTyler Ramer <tramer@vmware.com> Co-authored-by: NJamie McAtamney <jmcatamney@vmware.com>
-
由 Tyler Ramer 提交于
Yaml was imported but unused in several locations. gpMgmt/test/behave/mgmt_utils/steps/mgmt_utils.py had numerous unused or duplicated imports. Co-authored-by: NTyler Ramer <tramer@vmware.com> Co-authored-by: NJamie McAtamney <jmcatamney@vmware.com>
-
由 Tyler Ramer 提交于
It seems this yaml class is dead code. Removing it for this reason. Co-authored-by: NTyler Ramer <tramer@vmware.com> Co-authored-by: NJamie McAtamney <jmcatamney@vmware.com>
-
由 Tyler Ramer 提交于
The version of PyYAML vendored in gpMgmt/bin/ext is old, unmaintained, and does not support python3. Actually, it does not even contain a `__version__` attribute, so it is not possible to know the version. We need to unvendor yaml and address CVEs that have been found in the library since the version vendored in source. Also update yaml.load to use yaml.safe_load instead. Co-authored-by: NTyler Ramer <tramer@vmware.com> Co-authored-by: NJamie McAtamney <jmcatamney@vmware.com>
-
- 09 7月, 2020 1 次提交
-
-
由 Tyler Ramer 提交于
[Lockfile](https://pypi.org/project/lockfile/) has not been maintained since around 2015. Further, the functionality it provided seems poor - a review of the code indicated that it used the presence of the PID file itself as the lock - in Unix, using a file's existence followed by a creation is not atomic, so a lock could be prone to race conditions. The lockfile package also did not clean up after itself - a process which was destroyed unexpectedly would not clear the created locks, so some faulty logic was added to mainUtils.py, which checked to see if a process with the same PID as the lockfile's creator was running. This is obviously failure prone, as a new process might be assigned the same PID as the old lockfile's owner, without actually being the same process. (Of note, the SIG_DFL argument to os.kill() is not a signal at all, but rather of type signal.handler. It appears that the python cast this handler to the int 0, which, according to man 2 kill, leads to no signal being sent, but existance and permission checks are still performed. So it is a happy accident that this code worked at all) This commit removes lockfile from the codebase entirely. It also adds a "PIDLockFile" class which provides an atomic-guarenteed lock via the mkdir and rmdir commands on Unix - thus, it is not safely portable to Windows, but this should not be an issue as only Unix-based utilities use the "simple_main()" function. PIDLockFile provides API compatible classes to replace most of the functionality from lockfile.PidLockFile, but does remove any timeout logic as it was not used in any meaningful sense - a hard-coded timeout of 1 second was used, but an immediate result of if the lock is held is sufficient. PIDLockFile also includes appropriate __enter__, __exit__, and __del__ attributes, so that, should we extend this class in the future, with syntax is functional, and __del__ calls release, so a process reaped unexpectedly should still clean its own locks as part of the garbage collection process. Authored-by: NTyler Ramer <tramer@pivotal.io> Do not remove PYLIB_SRC_EXT during make clean/distclean Commit 8190ed40 removed lockfile from mainUtils, but did not remove a reference to its source directory in the make clean/distclean target. As a result, because LOCKFILE_DIR is no longer defined, the make clean/distclean target removes the PYLIB_SRC_EXT directory.
-
- 08 7月, 2020 3 次提交
-
-
由 xiong-gang 提交于
The entry in aocsseg table might be compacted and waiting for drop, so we should use 'state' to filter the unused entry.
-
由 xiong-gang 提交于
column 'vpinfo' in pg_aoseg.pg_aocsseg_xxx record the 'eof' of each attribute in the AOCS table. Add a new check 'aoseg_table' in gpcheckcat, it checks the number of attributes in 'vpinfo' is the same as the number of attributes in 'pg_attribute'. This check is performed in parallel and independently on each segment, and it checks aoseg table and pg_attribute in different transaction, so it should be run 'offline' to avoid false alarm.
-
由 (Jerome)Junfeng Yang 提交于
Make some utilities search path safe, so it'll not calling any external functions that has the same name with our built-in functions. This fix does not guarantee to fix CVE-2018-1058. Backport from 070d6221. Co-authored-by: Jamie McAtamney jmcatamney@pivotal.io Co-authored-by: Jacob Champion pchampion@pivotal.io Co-authored-by: Shoaib Lari slari@pivotal.io
-
- 06 7月, 2020 1 次提交
-
-
由 Xin Zhang 提交于
- generate-greenplum-path.sh use second parameter `yes` to generate PYTHONHOME - ONLY gpAux need to run this script with second parameter `yes`, since in GPDB6 we always vendor python. - developer build on their box won't be impacted by this change, since regular build doesn't have second parameter, hence PYTHONHOME will be skipped. We added test-generate-greeplum-path.bash to verify the correctness of the script. [#173567726] Co-authored-by: NBradford D. Boyle <bradfordb@vmware.com> Co-authored-by: NTingFang Bao <bbob@vmware.com> Co-authored-by: NXin Zhang <zhxin@vmware.com> Co-authored-by: NBradford D. Boyle <bradfordb@vmware.com> Co-authored-by: NTingFang Bao <bbob@vmware.com>
-
- 30 6月, 2020 1 次提交
-
-
由 Hubert Zhang 提交于
For non partition table, we will skip gpexpand external tables, but for partition table, when one of its child parition is a external table, we error out when gpexpand. This is not a correct behavior. Since data of foreign table is located outside gpdb, skip these tables when gpexpand is enough. Co-Authored-by: Ning Yu nyu@pivotal.io (cherry picked from commit b9691eba)
-
- 29 6月, 2020 1 次提交
-
-
由 Ning Yu 提交于
When cleaning up the master-only files on the new segments we used to do the job one by one, when there are tens or hundreds of segments it can be very slow. Now we cleanup in parallel. (cherry picked from commit 857763ae) Reviewed-by: NZhenghua Lyu <zlv@pivotal.io>
-
- 26 6月, 2020 1 次提交
-
-
由 Denis Smirnov 提交于
Mirror registration passes through several steps at the moment: 1. CREATE_QE_ARRAY (QE_MIRROR_ARRAY is ordered by content) 2. ARRAY_REORDER (QE_MIRROR_ARRAY is ordered by port) 3. CREATE_ARRAY_SORTED_ON_CONTENT_ID (form QE_MIRROR_ARRAY_SORTED_ON_CONTENT_ID on a base of QE_MIRROR_ARRAY) 4. REGISTER_MIRRORS (walk through QE_MIRROR_ARRAY, register mirrors with pg_catalog.gp_add_segment_mirror on master's gp_segment_configuration and update QE_MIRROR_ARRAY with returned dbids) 5. CREATE_SEGMENT (walk through QE_MIRROR_ARRAY_SORTED_ON_CONTENT_ID with old dbids and create mirrors on segment hosts with pg_basebackup) The problem is in a step 4 - we update the wrong array (QE_MIRROR_ARRAY instead of QE_MIRROR_ARRAY_SORTED_ON_CONTENT_ID). Because of that we get inconsistency between mirror dbids on gp_segment_configuration and internal.auto.conf files. This can cause inoperable cluster state in some situations when we promote a failed primary from a mirror with wrong dbids (FTS can't solve this issue). Also fixed column indexes in array used for segment arrray ordering. It was not done after commit https://github.com/greenplum-db/gpdb/commit/03c7d557720c5a78af1e2574ac385d10a0797f5e which prepend array with new hostname column. Co-authored-by: NVasiliy Ivanov <ivi@arenadata.io> (cherry picked from commit 8d008792)
-
- 25 6月, 2020 5 次提交
-
-
由 Bradford D. Boyle 提交于
Setting `$PYTHONHOME` for releases where python is not vendored causes the system python to fail with an error message about not being able to find the site module. This commit updates the logic in generate-greenplum_path.sh to check if we are vendoring python and if we are, it includes setting `$PYTHONHOME` in greenplum_path.sh. If we are vendoring python, then at build-time `$PYTHONHOME` is set to point at our vendored python. [#173046174] Authored-by: NBradford D. Boyle <bradfordb@vmware.com>
-
由 Bradford D. Boyle 提交于
[#173046174] Authored-by: NBradford D. Boyle <bradfordb@vmware.com>
-
由 Tingfang Bao 提交于
Authored-by: NTingfang Bao <baotingfang@gmail.com>
-
由 Bradford D. Boyle 提交于
When upgrading from GPDB5 to GPDB6, gpupgrade will need to be able to call binaries from both major versions. Relying on LD_LIBRARY_PATH is not an option because this can cause binaries to load libraries from the wrong version. Instead, we need the libraries to have RPATH/RUNPATH set correctly. Since the built binaries may be relocated we need to use a relative path. This commit disables the rpath configure option (which would result in an absolute path) and use LDFLAGS to use `$ORIGIN`. For most ELF files a RUNPATH of `$ORIGIN/../lib` is correct. For pygresql python module and the quicklz_compressor extension, the RUNPATH needs to be adjusted accordingly. The LDFLAGS for those artifacts can be modified with different environment variables PYGRESQL_LDFLAGS and QUICKLZ_LDFLAGS. We always use `--enable-new-dtags` to set RUNPATH. On CentOS 6, with new dtags, both DT_RPATH and DT_RUNPATH are set and DT_RPATH will be ignored. [#171588878] Co-authored-by: NBradford D. Boyle <bboyle@pivotal.io> Co-authored-by: NXin Zhang <xzhang@pivotal.io>
-
由 Tingfang Bao 提交于
Following the [Greenplum Server RPM Packaging Specification][0], we need to update greenplum_path.sh file, and ensure many environment variables set correct. There are a few basic requirments for Greenplum Path Layer: * greenplum-path.sh shall be installed to `${installation prefix}/greenplum-db-[package-version]/greenplum_path.sh` * ${GPHOME} is set by given parameter, by default it should point to `%{installation prefix}/greenplum-db-devel` * `${LD_LIBRARY_PATH}` shall be safely set to avoid a trailing colon (which will cause the linker to search the current directory when resolving shared objects) * `${PYTHONHOME}` shall be set to `${GPHOME}/ext/python` * `${PYTHONPATH}` shall be set to `${GPHOME}/lib/python` * `${PATH}` shall be set to `${GPHOME}/bin:${PYTHONHOME}/bin:${PATH}` * If the file `${GPHOME}/etc/openssl.cnf` exists then `${OPENSSL_CONF}` shall be set to `${GPHOME}/etc/openssl.cnf` * The greenplum_path.sh file shall pass [ShellCheck][1] [0]: https://github.com/greenplum-db/greenplum-database-release/blob/master/Greenplum-Server-RPM-Packaging-Specification.md#detailed-package-behavior [1]: https://github.com/koalaman/shellcheckCo-authored-by: NTingfang Bao <bbao@pivotal.io> Co-authored-by: NXin Zhang <zhxin@vmware.com> Co-authored-by: NNing Wu <ningw@vmware.com> Co-authored-by: NShaoqi Bai <bshaoqi@vmware.com>
-
- 10 6月, 2020 1 次提交
-
- 08 6月, 2020 1 次提交
-
-
由 Hubert Zhang 提交于
When introducing a new mirror, we need two steps: 1. start mirror segment 2. update gp_segment_configuration catalog Previously gp_add_segment_mirror will be called to update the catalog, but dbid is chosen by get_availableDbId() which cannot ensure to be the same dbid in internal.auto.conf. Reported by issue9837 Reviewed-by: NPaul Guo <pguo@pivotal.io> Reviewed-by: NBhuvnesh Chaudhary <bhuvnesh2703@gmail.com> cherry-pick from commit: f7965d and 1ee999
-
- 04 6月, 2020 1 次提交
-
-
由 Wen Lin 提交于
-
- 21 5月, 2020 1 次提交
-
-
由 Wen Lin 提交于
while gpload is loading data if the configure file contains "error_table" and doesn't contain "preload", an error of no attribute "staging_table" or "fast_path" occurs.
-
- 13 5月, 2020 2 次提交
-
-
由 Ning Yu 提交于
We use "pkill postgres" to cleanup leaked segments in the behave tests, if the postgress processes already exited the pkill command would fail with code 1, "No processes matched or none of them could be signalled". Fixed by ignoring the return code of pkill. (cherry picked from commit a92e0a33)
-
- 12 5月, 2020 1 次提交
-
-
由 Peifeng Qiu 提交于
gpload in the latest windows client package requires VS redistributable package. Output more meaningful message if pg.py fails to load.
-
- 07 5月, 2020 2 次提交
-
-
由 Bhuvnesh Chaudhary 提交于
Previously, gpinitsystem did not allow the user to specify a hostname and address for each segment in the input file used with -I; it only accepted one value per segment and used it for both hostname and address. This commit changes the behavior so that the user can specify both hostname and address. If the user specifies only the address (such as by using an old config file), it will preserve the old behavior and set both hostname and address to that value. It also adds a few tests around input file parsing so SET_VAR is more resilient to further refactors. The specific changes involved are the following: 1) Change SET_VAR to be able to parse either the old format (address only) or new format (host and address) of the segment array representation. 2) Move SET_VAR from gpinitsystem to gp_bash_functions.sh and remove the redundant copy in gpcreateseg.sh. 3) Remove a hardcoded "~0" in QD_PRIMARY_ARRAY in gpinitsystem, representing a replication port value, that was left over from 5X. 4) Improve the check for the number of fields in the segment array representation. Also, Remove use of ignore warning flag and use [[ ]] the check for IGNORE_WARNING
-
由 Bhuvnesh Chaudhary 提交于
Previously, gpintsystem was incorrectly filling the hostname field of each segment in gp_segment_configuration with the segment's address. This commit changes it to correctly resolve hostnames and update the catalog accordingly. This reverts commit 12ef7352, Revert "gpinitsystem: update catalog with correct hostname". Commit message from 12ef7352: The commit requires some additional tweaks to the input file logic for backwards compatibility purposes, so we're reverting this until the full fix is ready.
-
- 29 4月, 2020 1 次提交
-
-
由 Ning Yu 提交于
The gpinitsystem will fail if the gpinitsystem logs contain errors or warnings from previous tests.
-
- 31 3月, 2020 1 次提交
-
-
由 Jamie McAtamney 提交于
This reverts commit e4add7cb. The commit requires some additional tweaks to the input file logic for backwards compatibility purposes, so we're reverting this until the full fix is ready.
-
- 18 3月, 2020 1 次提交
-
-
由 Jamie McAtamney 提交于
Previously, gpintsystem was incorrectly filling the hostname field of each segment in gp_segment_configuration with the segment's address. This commit changes it to correctly resolve hostnames and update the catalog accordingly. Co-authored-by: NKalen Krempely <kkrempely@pivotal.io>
-
- 14 3月, 2020 2 次提交
-
-
由 Adam Berlin 提交于
gpinitsystem did not quote the username while performing ALTER USER. When the username is a numeric value the postgres parser gets upset - unless the username is quoted. See here for more details: https://www.postgresql.org/docs/9.4/sql-syntax-lexical.html#SQL-SYNTAX-IDENTIFIERS - SQL identifiers and key words must begin with a letter (a-z, but also letters with diacritical marks and non-Latin letters) or an underscore (_). - Also, there is a second kind of identifier: the delimited identifier or quoted identifier. It is formed by enclosing an arbitrary sequence of characters in double-quotes (") - use variable interpolation provided by psql to properly quote user-provided values. - use RETVAL to perform testing due to Commit d7b7a40aCo-authored-by: NJacob Champion <pchampion@pivotal.io> (cherry picked from commit f188ecb5)
-
由 Ashuka Xue 提交于
Previously, analyzedb would error out and fail if a table was dropped during analyzedb. Now, we silently skip dropped tables when determining the tables to analyze.
-