- 18 6月, 2020 3 次提交
-
-
由 Jinbao Chen 提交于
We merge the brin from Postgres9.5, but greenplum did not enable brin on ao/aocs table. The reason brin cannot be used directly on the ao / aocs table is that the storage structure of ao / aocs is different from the heap table. Heap has only one physical file, and all block numbers are continuous. The revmap in brin is a array that spans multiple blocks, but it does not make sense in ao/aocs table. Ao/aocs has 128 segment files, and the block numbers in these segments are distributed over the entire value range. If we use an array to record the information of each block, this array will be too large. So we introduced an upper structure to solve this problem. The upper level is a array which records the block number of the revmap block. The revmap blocks are not continuous. When we need an new revmap block, just extend a new one and record the block number in the upper level array. Reviewed-by: NAsim R P <pasim@vmware.com> Reviewed-by: NAshwin Agrawal <aagrawal@pivotal.io> Reviewed-by: Nxiong-gang <gxiong@pivotal.io> Reviewed-by: NAdam Lee <adam8157@gmail.com>
-
由 Mel Kiyama 提交于
* docs - update postGIS 2.5.4 docs Updates for Greenplum PostGIS 2.5.4 v2 --Add list of PostGIS extensions --Add support for PostGIS TIGER geocoder, address standardizer and address rules files. --Update install/uninstall instructions to use CREATE EXTENSION command --Remove postgis_manager.sh script --Remove PostGIS Raster limitation. * docs updated PostGIS 2.5.4 docs based on review comments. * docs - removed postgis_raster extension. * docs - review comment updates -Added section for installing the PostGIS package -Updated section on removing PostGIS package -Fix typos. * docs - updated platform requirements for PostGIS 2.5.4 v2 -also removed "beta" from GreenplumR
-
由 Tyler Ramer 提交于
PyGreSQL may now be installed via pip or via Ubuntu apt. Update the travis pipeline as well, using submodules to pull the necessary python dependencies. Thus, they are removed from PIP as well. Authored-by: NTyler Ramer <tramer@pivotal.io>
-
- 17 6月, 2020 9 次提交
-
-
由 David Yozie 提交于
-
由 Hao Wu 提交于
This patch fixes the issue: https://github.com/greenplum-db/gpdb/issues/10224 Replicated table is not allowed to be a partition table. So an existing partition table must not be altered its distribution policy to REPLICATED. Reported-by: NHeikki Linnakangas <hlinnakangas@pivotal.io> Reviewed-by: NZhenghua Lyu <zlv@pivotal.io>
-
由 Zhenghua Lyu 提交于
Greenplum tests the visibility of heap tuples firstly using distributed snapshot. Distributed snapshot is generated on QD and then dispatched to QEs. Some utility statement needs to work under the latest snapshot when executing, so that they invoke the function `GetLatestSnapshot` in QEs. But remember we cannot get the latest distributed snapshot. Subtle cases are: Alter Table or Alter Domain statements on QD get snapshot in Portal Run and then try to hold locks on the target table in ProcessUtilitySlow. Here is the key point: 1. try to hold lock ==> it might be blocked by other transactions 2. then it will be waked up to continue 3. when it can continue, the world has changed because other transactions then blocks it has been over Previously, on QD we do not getsnapshot before we dispatch utility statement to QEs which leads to the distributed snapshot does not reflect the "world change". This will lead to some bugs. For example, if the first transaction is to rewrite the whole heap, and then the second Alter Table or Alter Domain statements continues with the distributed snapshot that txn1 does not commit yet, it will see no tuples in the new heap! This commit fixes the issue by getting a local snapshot when invoking `GetLatestSnapshot` when in QEs. See Github issue: https://github.com/greenplum-db/gpdb/issues/10216Co-authored-by: NHubert Zhang <hzhang@pivotal.io>
-
由 Tyler Ramer 提交于
We encounted a bug in escaping dbname and connection options in pygresql 5.1.2, which we submitted a patch for here: https://github.com/PyGreSQL/PyGreSQL/pull/40 This has been merged, but it will take time to be added to a tagged release. For this reason, we have downloaded the source using this commit, https://github.com/PyGreSQL/PyGreSQL/commit/b1e040e989b5b1b75f42c1103562bfe8f09f93c3 to install. Co-authored-by: NTyler Ramer <tramer@pivotal.io> Co-authored-by: NJamie McAtamney <jmcatamney@pivotal.io>
-
由 Tyler Ramer 提交于
The test addressed in this commit was added in commit f3df8b18, fails for the entirely unrelated reason that, due to a modification of sql_isolation_testcase.py, the line numbers are different. I find this test very fragile for this reason, and for the fact that we're relying on an execution failure in isolation2 python code to test the database code. This means that any refactoring of isolation2 will cause this test to fail - which should not be. I looked into adding an ignore to the exact lines, but isolation2 wants there to be a matched ignore in the input sql file - which makes the test useless, because we're looking for some exact exception from isolation2 from a valid sql input. Isolation2 doesn't give us the framework to ignore just some messages on the output side. Using a isolation2 init modification still just ignores the actual problem, but in a different file. This fix should just be considered a tempory work to get the pipeline green while a better solution is determined later. Authored-by: NTyler Ramer <tramer@pivotal.io>
-
由 Tyler Ramer 提交于
The update to pygresql pg connection allows the output of sql isolation2 testing to be more similar to psql. Thus, we are reverting some of the changes made in commits 20b3aa3a to instead be more inline with the usual psql output. Notably, trailing zeroes on floats are trimmed. Co-authored-by: NTyler Ramer <tramer@pivotal.io> Co-authored-by: NJamie McAtamney <jmcatamney@pivotal.io>
-
由 Tyler Ramer 提交于
Due to refactor of dbconn and newer versions of pygresql, using `with dbconn.connect() as conn` no longer attempts to close a connection, even if it did prior. Instead, this syntax uses the connection itself as context and, as noted in execSQL, overrides the autocommit functionality of execSQL. Therefore, close the connection manually to ensure that execSQL is auto-commited, and the connection is closed. Co-authored-by: NTyler Ramer <tramer@pivotal.io> Co-authored-by: NJamie McAtamney <jmcatamney@pivotal.io>
-
由 Tyler Ramer 提交于
One reason pygresql was previously modified was that it did not handle closing a connection very gracefully. In the process of updating pygresql, we've wrapped the connection it provides with a ClosingConnection function, which should handle gracefully closing the connection when the "with dbconn.connect as conn" syntax is used. This did, however, illustrate issues where a cursor might have been created as the result of a dbconn.execSQL() call, which seems to hold the connection open if not specifically closed. It is therefore necessary to remove the ability to get a cursor from dbconn.execSQL(). To highlight this difference, and to ensure that future calls to this library is easy to use, I've cleaned up and clarified the dbconn execution code, to include the following features. - dbconn.execSQL() closes the cursor as part of the function. It returns no rows - functions dbconn.query() is added, which behaves like dbconn.execSQL() except that it now returns a cursor - function dbconn.execQueryforSingleton() is renamed dconn.querySingleton() - function dbconn.execQueryforSingletonRow() is renamed dconn.queryRow() Authored-by: NTyler Ramer <tramer@pivotal.io>
-
由 Tyler Ramer 提交于
This commit updates pygresql from 4.0.0 to 5.1.2, which requires numerous changes to take advantages of the major result syntax change that pygresql5 implemented. Of note, cursors or query objects automatically cast returned values as appropriate python types - list of ints, for example, instead of a string like "{1,2}". This is the bulk of the changes. Updating to pygresql 5.1.2 provides numerous benfits, including the following: - CVE-2018-1058 was addressed in pygresql 5.1.1 - We can save notices in the pgdb module, rather than relying on importing the pg module, thanks to the new "set_notices()" - pygresql 5 supports python3 - Thanks to a change in the cursor, using a "with" syntax guarentees a "commit" on the close of the with block. This commit is a starting point for additional changes, including refactoring the dbconn module. Additionally, since isolation2 uses pygresql, some pl/python scripts were updated, and isolation2 SQL output is further decoupled from pygresql. The output of a psql command should be similar enough to isolation2's pg output that minimal or no modification is needed to ensure gpdiff can recognize the output. Co-Authored-by: NTyler Ramer <tramer@pivotal.io> Co-authored-by: NJamie McAtamney <jmcatamney@pivotal.io>
-
- 16 6月, 2020 5 次提交
-
-
由 Jesse Zhang 提交于
We had a bug in a few of the combine functions where if the combine function returned a NULL, it didn't set fcinfo->isnull = true. This led to a segfault when we would spill in the final hashagg of a two-stage agg inside the serial function. So, properly mark NULL outputs from the combine functions. Co-authored-by: NDenis Smirnov <sd@arenadata.io> Co-authored-by: NSoumyadeep Chakraborty <sochakraborty@pivotal.io>
-
由 Jesse Zhang 提交于
Earlier, we always deducted FREEABLE_BATCHFILE_METADATA inside closeSpillFile() regardless of whether the spill file was already suspended. This deduction, is already performed inside suspendSpillFiles(). This double accounting leads to hashtable->mem_for_metadata becoming negative and we get: FailedAssertion("!(hashtable->mem_for_metadata > 0)", File: "execHHashagg.c", Line: 2141) Co-authored-by: NSoumyadeep Chakraborty <sochakraborty@pivotal.io>
-
由 Jesse Zhang 提交于
This commit fixes the following assertion failure message reported in: (#9902) https://github.com/greenplum-db/gpdb/issues/9902 FailedAssertion("!(hashtable->nbuckets > spill_set->num_spill_files)", File: "execHHashagg.c", Line: 1355) hashtable->nbuckets can actually end up being equal to spill_set->num_spill_files, which causes the failure. This is because: hashtable->nbuckets is set with HashAggTableSizes->nbuckets, which can end up being equal to: gp_hashagg_default_nbatches. Refer: nbuckets = Max(nbuckets, gp_hashagg_default_nbatches); Also, spill_set->num_spill_files is set with HashAggTableSizes->nbatches, which is further set to gp_hashagg_default_nbatches. Thus, these two entities can be equal. Co-authored-by: NSoumyadeep Chakraborty <sochakraborty@pivotal.io>
-
由 (Jerome)Junfeng Yang 提交于
Increase the retry count to prevent test failed. Most of the time, the failure is because slow processing.
-
由 Chris Hajas 提交于
We need to ignore the output when enabling/disabling an Orca xform, as if the server is not compiled with Orca there will be a diff (and we don't really care about this output). Additionally, clean up unnecessaary/excessive setting of GUCs Some of these gucs were on by default or only intended for a specific test. Explicitly setting them caused them to appear at the end of `explain verbose` plans, making the expected output more difficult to match with if the server was built with/without Orca.
-
- 15 6月, 2020 4 次提交
-
-
由 Paul Guo 提交于
Some test cases have been failing due to too few retries. Let's increase them and also create some common UDF for use. Reviewed-by: NHubert Zhang <hzhang@pivotal.io> Reviewed-by: NAshwin Agrawal <aagrawal@pivotal.io>
-
由 Paul Guo 提交于
Fix flakiness of "select 1" output after master reset due to injected panic fault before_read_command (#10275) Several tests inject panic in before_read_command to trigger master reset. Previous we run "select 1" after the fault inject query to verify, but the output is not deterministic sometimes. i.e. sometimes we do not see the line PANIC: fault triggered, fault name:'before_read_command' fault type:'panic' This was actually observed in test crash_recovery_redundant_dtx per commit message and test comment. It ignores the output of "select 1", but probably we still want the output to verify the fault is encountered. It's still mysterious why sometimes the PANIC message is missing. I spent some time on digging but reckon that I can not root cause in short time. One guess is that the PANIC message was although sent to the frontend in errfinish() but the kernel buffer-ed data was dropped after abort() due to ereport(PANIC); Another guess is something wrong related to libpq protocol (not saying it's a libpq bug). In any case, it does not deserve much time to work on the tests only, so simply mask the PANIC message to make the test result deterministic and also not affect the test purpose. Reviewed-by: NHubert Zhang <hzhang@pivotal.io>
-
由 xiong-gang 提交于
When move a query to a resource group whose memory_limit is 0, the available memory is the current available global shared memory.
-
由 xiong-gang 提交于
When the error happens after ProcArrayEndTransaction, it will recurse back to AbortTransaction, we need to make sure it will not generate extra WAL record and not fail the assertions.
-
- 13 6月, 2020 2 次提交
-
-
由 Junfeng(Jerome) Yang 提交于
-
由 Hubert Zhang 提交于
The flaky case happens when select an external table with option "fill missing fields". By gdb the qe, this value is not false on QE sometimes. When ProcessCopyOptions, we use intVal(defel->arg) to parse the boolean value, which is not correct. Using defGetBoolean to replace it. Also fix a pxf_fdw test case, which should set fill_missing_fields to true explicitly.
-
- 12 6月, 2020 2 次提交
-
-
由 (Jerome)Junfeng Yang 提交于
Remove pg_exttable.h since the catalog is no longer exist anymore. Move function declaration in pg_exttable.h into external.h. Extract related code into external.c which maintains all codes that can not be moved into an external table fdw extension. Also, move the external table orca interface into external.c as a workaround. Maybe provide orca fdw routine in the future. Extract the external table's execution logic into external table fdw extension. Create the gp_exttable_fdw extension during gpinitsystem to allow creating system external tables.
-
由 David Yozie 提交于
-
- 11 6月, 2020 5 次提交
-
-
由 Hubert Zhang 提交于
This reverts commit 026e4595. This commit break pxf test case. We need to handle it firstly.
-
由 Hubert Zhang 提交于
The test case restarts all primaries and expects the old session would fail for the next query since gangs are cached. But the restart may last more than 18s which is the max idle time QEs could exist. In this case, the new query in the old session will just fetch a new gang without expected errors. Just set gp_vmem_idle_resource_timeout to 0 to fix this flaky test. Reviewed-by: NPaul Guo <pguo@pivotal.io>
-
由 Hubert Zhang 提交于
The flaky case happens when select an external table with option "fill missing fields". By gdb the qe, this value is not false on QE sometimes. When ProcessCopyOptions, we use intVal(defel->arg) to parse the boolean value, which is not correct. Using defGetBoolean to replace it.
-
由 J·Y 提交于
-
由 Lena Hunter 提交于
* clarifying pg_upgrade note * graph edits * graph analytics updates * menu edits and code spacing * graph further edits * insert links for modules
-
- 10 6月, 2020 4 次提交
-
-
由 David Yozie 提交于
-
由 Huiliang.liu 提交于
* Add GUC write_to_gpfdist_timeout write_to_gpfdist_timeout controls timeout value (in seconds) for writing data to gpfdist server. Default value is 300, valid scope is [1, 7200] Set CURLOPT_TIMEOUT as write_to_gpfdist_timeout For any error, retry with double interval time, returns SQL ERROR if write_to_gpfdist_timeout is reached Add regression test for GUC writable_external_table_timeout
-
由 Chris Hajas 提交于
The python changes required new images, so we now need a separate pipeline for slack commands from 6X. We also no longer need libsigar.
-
由 Lena Hunter 提交于
* clarifying pg_upgrade note * gpinitsystem -O clarification * further reviews
-
- 09 6月, 2020 2 次提交
-
-
由 Paul Guo 提交于
After gprecoverseg, need to wait until the cluster is synchronized before running subsequent tests. Reviewed-by: NHubert Zhang <hzhang@pivotal.io> Reviewed-by: NReviewed-by: Ashwin Agrawal <aagrawal@pivotal.io>
-
由 David Yozie 提交于
* Update statement about mirroring recommendations & support * Updates based on k8s feedback
-
- 08 6月, 2020 3 次提交
-
-
由 Heikki Linnakangas 提交于
Was left unused by commit e4b499aa that removed the pg_exttable catalog table.
-
由 Zhenghua Lyu 提交于
Previous commit 62579728 fixes a lateral panic issue but does not handle all the bad cases because it only check if the query tree contains limit clause. Bad cases for example: if the subquery is like `q1 union all (q2 limit 1)` then the whole query tree does not contain limit clause. Another bad case is the lateral subquery may contain groupby. like: select * from t1_lateral_limit t1 cross join lateral (select (c).x+t2.a, sum(t2.a+t2.b) from t2_lateral_limit t2 group by (c).x+t2.a)x; When planning the lateraled subquery we do not know where is the param in the subquery's query tree. Thus it is a bit complicated to precisely and efficiently resolve this issue. This commit adopts a simple method to fix panic issue: it justs check the subquery's query tree to see if there is any group-by or limit clause, if so, force gather each relation and materialize them. This is not the best plan we might get. But let's make it correct first and I think in future we should seriously consider how to fully and efficiently support lateral.
-
由 Paul Guo 提交于
Use guc gp_role only now and replace the functionality of guc gp_session_role with it also. Previously we have both gucs. The difference of the two gucs are (copied from code comment): * gp_session_role * * - does not affect the operation of the backend, and * - does not change during the lifetime of PostgreSQL session. * * gp_role * * - determines the operating role of the backend, and * - may be changed by a superuser via the SET command. This is not friendly for coding. For example, You could find Gp_role and Gp_session_role are set as GP_ROLE_DISPATCH on Postmaster & many aux processes on all nodes (even QE nodes) in a cluster, so you can see that to differ from QD postmaster and QE postmaster, current gpdb uses an additional -E option in postmaster arguments. These makes developers confusing when writing role branch related code given we have three related variables. Also some related code is even buggy now (e.g. 'set gp_role' even FATAL quits). With this patch we just have gp_role now. Some changes which might be interesting in the patch are: 1. For postmaster, we should specify '-c gp_role=' (e.g. via pg_ctl argument) to determine the role else we assume the utility role. 2. For stand-alone backend, utility role is enforced (no need to specify by users). 3. Could still connect QE/QD nodes using utility mode with PGOPTIONS, etc as before. 4. Remove the '-E' gpdb hacking and align the '-E' usage with upstream. 5. Move pm_launch_walreceiver out of the fts related shmem given the later is not used on QE. Reviewed-by: NBhuvnesh Chaudhary <bchaudhary@pivotal.io> Reviewed-by: NGang Xiong <gxiong@pivotal.io> Reviewed-by: NHao Wu <gfphoenix78@gmail.com> Reviewed-by: NYandong Yao <yyao@pivotal.io>
-
- 06 6月, 2020 1 次提交
-
-
由 Lisa Owen 提交于
* docs - add info about moving a query to a different resource group * need to be superuser * remove upgrade/downgrade info for master
-