The fix is to perform the same steps as a TRUNCATE command - set new relfiles
and drop existing ones for parent AO table as well as all its auxiliary tables.

This fixes issue #913.  Thank you, Tao-Ma for reporting the issue and proposing
a fix as PR #960.  This commit implements Tao-Ma's idea but implementation
differs from the original proposal.

This is useful if a test wants to use gp_toolkit.__gp_{aoseg|aocsseg}*
functions.

Analyze collects a sample from the table, in case the
sample contains columns with huge length, it may
result in memory usage to go high cancelling the query.

This commit masks wide values `i.e pg_column_size(col) > WIDTH_THRESHOLD
(1024)` in variable length columns to avoid high
memory usage while collecting sample. Column values exceeding
WIDTH_THRESHOLD will be marked as NULL and will be ignored from the collected
samples tuples while computing stats on the relation.

In case of expression/predicate indexes on the relation, the wide columns will be
treated as NULL and will not be filtered out. Is it rare to have such
indexes on very wide columns, so the effects on stats (nullfrac etc) will be minimal.
Signed-off-by: NOmer Arap <oarap@pivotal.io>

The breakin/out functions should be marked as STRICT,
because the underlying C functions, textin/textout,
don't expect a NULL to be passed to them.
Signed-off-by: NEkta Khanna <ekhanna@pivotal.io>

A lot of tests assumed OID == relfilenode. We updated the tests to not assume
that anymore.

These tests assumed OID == relfilenode. We updated the tests to not assume it
anymore.

This is needed to prevent relations possibly overwriting each
other. The O_EXCL is present in postgres's mdcreate() but for some
reason we don't have it here. This adds it back.
Signed-off-by: NXin Zhang <xzhang@pivotal.io>

The master allocates an OID and provides it to segments during
dispatch. The segments then check if they can use this OID as its
relfilenode. If a segment cannot use the preassigned OID as the
relation's relfilenode, it will generate a new relfilenode value via
nextOid counter. This can result in a race condition between the
generation of the new OID and the segment file being created on disk
after being added to persistent tables. To combat this race condition,
we have a small OID cache... but we have found in testing that it was
not enough to prevent the issue.

To fully solve the issue, we decouple OID and relfilenode on both QD and QE
segments by introducing a nextRelfilenode counter which is similar to
the nextOid counter. The QD segment will generate the OIDs and its own
relfilenodes. The QE segments only use the preassigned OIDs from the QD
dispatch and generate a relfilenode value from their own nextRelfilenode
counter.

Current sequence generation is always done on QD sequence server, and
assumes the OID is always same as relfilenode when handling sequence client
requests from QE segments. It is hard to change this assumption so we have a
special OID/relfilenode sync for sequence relations for GP_ROLE_DISPATCH and
GP_ROLE_UTILITY.

Reference gpdb-dev thread:
https://groups.google.com/a/greenplum.org/forum/#!topic/gpdb-dev/lv6Sb4I6iSISigned-off-by: NXin Zhang <xzhang@pivotal.io>

[ci skip]

Initialize TransactionXmin to avoid situations where scanning pg_authid or other
tables mostly in BuildFlatFiles() via SnapshotNow may try to chase down
pg_subtrans for older "sub-committed" transaction. File corresponding to which
may not and is not supposed to exist. Setting TransactionXmin will avoid calling
SubTransGetParent() in TransactionIdDidCommit() for older XIDs. Also, along the
way initialize RecentGlobalXmin as Heap access method needs it set.

Repro for record of one such case:
```
CREATE ROLE foo;

BEGIN;
SAVEPOINT sp;
DROP ROLE foo;
RELEASE SAVEPOINT sp; -- this is key which marks in clog as sub-committed.

kill or gpstop -air
< N transactions to atleast cross pg_subtrans single file limit roughly
CLOG_XACTS_PER_BYTE * BLCKSZ* SLRU_PAGES_PER_SEGMENT >

restart -- errors recovery with missing pg_subtrans
```

The extension for executable binaries is defined in X, replace the
old (and now defunct) references to EXE_EXT. Also remove a commented
out dead gpfdist rule in gpMgmt from before the move to core.

This reverts commit ab4398dd.
[#142986717]

Similar to ea818f0e, we remove the
sensitivity to segment count in test `dml_oids_delete`. Without this,
this test was passing for the wrong reason:

0. The table `dml_heap_r` was set up with 3 tuples, whose values in the
distribution column `a` are 1, 2, NULL respectively. On a 2-segment
system, the 1-tuple and 2-tuple are on distinct segments, and because of
a quirk of our local OID counter synchronization, they will get the same
oids.

0. The table `tempoid` will be distributed randomly under ORCA, with
tuples copied from `dml_heap_r`

0. The intent of the final assertion is checking that the OIDs are not
changed by the DELETE. Also hidden in the assumption is that the tuples
stay on the same segment as the source table.

0. However, the compounding effect of that "same oid" with a randomly
distributed `tempoid` will lead to a passing test when we have two
segments!

This commit fixes that. So this test will pass for the right reason, and
also on any segment count.

When I was trying to understand how does Orca generate plan for CTE using
shared input scan, I found that the share input scan is generated during CTE
producer & consumer DXL node to PlannedStmt translation stage, instead of Expr
to DXL stage inside Orca. It turns out CDXLPhysicalSharedScan is not used
anywhere, so remove all the related dead code.

It's an error in standard C - at least in older standards - to typedef
the same type more than once, even if the definition is the same. Newer
versions of gcc don't complain about it, but you can see the warnings
with -pedantic (among a ton of other warnings, search for "redefinition").

To fix, remove the duplicate typedefs. The ones in src/backend/gpopt and
src/include/gpopt were actually OK, because a duplicate typedef is OK in
C++, and those files are compiled with a C++ compiler. But many of the
typedefs in those files were not used for anything, so I nevertheless
removed duplicate ones there too, that caught my eye.

In gpmon.h, we were redefining apr_*_t types when postgres.h had been
included. But as far as I can tell, that was always - all the files that
included gpmon, included postgres.h directly or indirectly before that.
Search & replace the references to apr_*_t types in that file with the
postgres equivalents, to make it more clear what they actually are.

CdbCellBuf was only used in hash aggregates, and it only used a fraction
of the functionality. In essence, it was using it as a very simple memory
allocator, where each allocation was fixed size, and the only way to free
was to reset the whole cellbuf. But the same code was using a different,
but similar, mpool_* mechanism for allocating other things stored in
the hash buckets. We might as well use mpool_alloc for the HashAggEntry
struct as well, and get rid of all the cellbuf code.

CdbPtrBuf was completely unused.

The comment about backporting to a 10 year old version has passed
it's due date so remove. Also actually use the referenced variable
to make the code less confusing to readers (the compiler will be
smart enough about stack allocations anyways). Also reflows and
generally tidies up the comment a little.

appendStringInfo() is a variadic function treating the passed
string as a format specifier. This is wasteful processing when
just adding a constant string which can be done faster with a
call to appendStringInfoString() where no format processing is
performed.

This leaves lots of appendStringInfo() calls in the processing
but they are from upstream and will be addressed when we merge
with future versions of postgres. The calls in this patch are
the GPDB specific ones.

With the last remaining testsuites moved over to ICW, there is no
longer anything left running in BugBuster. Remove the remaining
files and BugBuster makefile integration in one big swing of the
git rm axe. The only thing left in use was a data file which was
referenced from ICW, move this to regress/data instead.

This moves the memory_quota tests more or less unchanged to ICW.
Changes include removing ignore sections and minor formatting as
well as a rename to bb_memory_quota.

This combines the various mpph tests in BugBuster into a single
new ICW suite, bb_mpph. Most of the existing queries were moved
over with a few pruned that were too uninteresting, or covered
elsewhere.

The BugBuster tests combined are: load_mpph, mpph_query,
mpph_aopart, hashagg and opperf.

QD used to send a transient types table to QEs, then QE would remap the
tuples with this table before sending them to QD. However in complex
queries QD can't discover all the transient types so tuples can't be
correctly remapped on QEs. One example is like below:

    SELECT q FROM (SELECT MAX(f1) FROM int4_tbl
                   GROUP BY f1 ORDER BY f1) q;
    ERROR:  record type has not been registered

To fix this issue we changed the underlying logic: instead of sending
the possibly incomplete transient types table from QD to QEs, we now
send the tables from motion senders to motion receivers and do the remap
on receivers. Receivers maintain a remap table for each motion so tuples
from different senders can be remapped accordingly. In such way, queries
contain multi-slices can also handle transient record type correctly
between two QEs.

The remap logic is derived from the executor/tqueue.c in upstream
postgres. There is support for composite/record types and arrays as well
as range types, however as range types are not yet supported in GPDB so
the logic is put under a conditional compilation macro, in theory it
shall be automatically enabled when range types are supported in GPDB.

One side effect for this approach is that on receivers a performance
down is introduced as the remap requires recursive checks on each tuple
of record types. However optimization is made to make this side effect
minimum on non-record types.

Old logic that building transient types table on QD and sending them to
QEs are retired.
Signed-off-by: NGang Xiong <gxiong@pivotal.io>
Signed-off-by: NNing Yu <nyu@pivotal.io>

Commit d9148a54 enabled the record array as well as the comparison of
record types, the comparison functions/operators' OIDs used in upstream
postgres are already used by others in GPDB, and many test cases are
assuming comparison of record types should fail. As we don't actually
need this comparison feature at the moment in GPDB we simply remove
these functions for now.
Signed-off-by: NNing Yu <nyu@pivotal.io>

Implement comparison of generic records (composite types), and invent a pseudo-type record[] to represent arrays of possibly-anonymous composite types. Since composite datums carry their own type identification, no extra knowledge is needed at the array level.

The main reason for doing this right now is that it is necessary to support
the general case of detection of cycles in recursive queries: if you need to
compare more than one column to detect a cycle, you need to compare a ROW()
to an array built from ROW()s, at least if you want to do it as the spec
suggests.  Add some documentation and regression tests concerning the cycle
detection issue.

The old mechanism was to scan the complete plan, searching for a pattern
with a Join, where the outer side included an Append node. The inner
side was duplicated into an InitPlan, with the pg_partition_oid aggregate
to collect the Oids of all the partitions that can match. That was
inefficient and broken: if the duplicated plan was volatile, you might
choose wrong partitions. And scanning the inner side twice can obviously
be slow, if there are a lot of tuples.

Rewrite the way such plans are generated. Instead of using an InitPlan,
inject a PartitionSelector node into the inner side of the join.

Fixes github issues #2100 and #2116.

This commit fix issue #1621. Current external table implementation
only recognize LF as line end. If table is created with CR as line
end then no data can be selected because whole data is never
splitted into lines.
Signed-off-by: NPeifeng Qiu <pqiu@pivotal.io>

* Adding last activity time in the SessionState.

* Adding last activity time in the session_state_memory_entries_f and updating view session_level_memory_consumption.

* Adding unit tests.

* Adding SessionState initialization test.

* Changing last_idle_time to idle_start as per PR suggestion.

* Rewrite Kerberos test suite

* Remove obsolete Kerberos test stuff from pipeline and TINC

We now have a rewritten Kerberos test script in installcheck-world.

* Update ICW kerberos test to run on concourse container

This adds a whole new test script in src/test/regress, implemented in plain bash. It sets up temporary a KDC as part of the script, and doesn't therefore rely on a pre-existing Kerberos server, like the old MU_kerberos-smoke test job did. It does require MIT Kerberos server-side utilities to be installed, instead, but no server needs to be running, and no superuser privileges are required.

This supersedes the MU_kerberos-smoke behave tests. The new rewritten bash script tests the same things:
  1. You cannot connect to the server before running 'kinit' (to verify that the server doesn't just let anyone in, which could happen if the pg_hba.conf is misconfigured for the test, for example)
  2. You can connect, after running 'kinit'
  3. You can no longer connect, if the user account is expired

The new test script is hooked up to the top-level installcheck-world target.

There were also some Kerberos-related tests in TINC. Remove all that, too. They didn't seem interesting in the first place, looks like they were just copies of a few random other tests, intended to be run as a smoke test, after a connection had been authenticated with Kerberos, but there was nothing in there to actually set up the Kerberos environment in TINC.

Other minor patches added:

* Remove absolute path when calling kerberos utilities
-- assume they are on path, so that they can be accessed from various installs
-- add clarification message if sample kerberos utility is not found with 'which'

* Specify empty load library for kerberos tools

* Move kerberos test to its own script file
-- this allows a failure to be recorded without exiting Make, and
therefore the server can be turned off always

* Add trap for stopping kerberos server in all cases
* Use localhost for kerberos connection
Signed-off-by: NMarbin Tan <mtan@pivotal.io>
Signed-off-by: NChumki Roy <croy@pivotal.io>
Signed-off-by: NLarry Hamel <lhamel@pivotal.io>

The loop to print each constraint's name was broken: it printed the name of
the first constraint multiple times. A test case, as matter of principle.

In the passing, change the set of tests around this error to all use the
same partitioned table, rather than drop and recreate it for each command.
And reduce the number of partitions from 10 to 5. Shaves some milliseconds
from the time to run the test.

This comes from 4.3_STABLE repo
Signed-off-by: NTom Meyer <tmeyer@pivotal.io>

GPDB supports range and list partitions. Range partitions are represented as a set of rules. Each rule defines the boundaries of a part. E.g., a rule might say that a part contains all values between (0, 5], where left bound is 0 exclusive, but the right bound is 5, inclusive. List partitions are defined by a list of values that the part will contain. 

ORCA uses the above rule definition to generate expressions that determine which partitions need to be scanned. These expressions are of the following types:

1. Equality predicate as in PartitionSelectorState->levelEqExpressions: If we have a simple equality on partitioning key (e.g., part_key = 1).

2. General predicate as in PartitionSelectorState->levelExpressions: If we need more complex composition, including non-equality such as part_key > 1.

Note:  We also have residual predicate, which the optimizer currently doesn't use. We are planning to remove this dead code soon.

Prior to  this PR, ORCA was treating both range and list partitions as range partitions. This meant that each list part will be converted to a set of list values and each of these values will become a single point range partition.

E.g., consider the DDL:

```sql
CREATE TABLE DATE_PARTS (id int, year int, month int, day int, region text)
DISTRIBUTED BY (id)
PARTITION BY RANGE (year)
    SUBPARTITION BY LIST (month)
       SUBPARTITION TEMPLATE (
        SUBPARTITION Q1 VALUES (1, 2, 3), 
        SUBPARTITION Q2 VALUES (4 ,5 ,6),
        SUBPARTITION Q3 VALUES (7, 8, 9),
        SUBPARTITION Q4 VALUES (10, 11, 12),
        DEFAULT SUBPARTITION other_months )
( START (2002) END (2012) EVERY (1), 
  DEFAULT PARTITION outlying_years );
```

Here we partition the months as list partition using quarters. So, each of the list part will contain three months. Now consider a query on this table:

```sql
select * from DATE_PARTS where month between 1 and 3;
```

Prior to this ORCA generated plan would consider each value of the Q1 as a separate range part with just one point range. I.e., we will have 3 virtual parts to evaluate for just one Q1: [1], [2], [3]. This approach is inefficient. The problem is further exacerbated when we have multi-level partitioning. Consider the list part of the above example. We have only 4 rules for 4 different quarters, but we will have 12 different virtual rule (aka constraints). For each such constraint, we will then evaluate the entire subtree of partitions.

After this PR, we no longer decompose rules into constraints for list parts and then derive single point virtual range partitions based on those constraints. Rather, the new ORCA changes will use ScalarArrayOp to express selectivity on a list of values. So, the expression for the above SQL will look like 1 <= ANY {month_part} AND 3 >= ANY {month_part}, where month_part will be substituted at runtime with different list of values for each of quarterly partitions. We will end up evaluating that expressions 4 times with the following list of values:

Q1: 1 <= ANY {1,2,3} AND 3 >= ANY {1,2,3}
Q2: 1 <= ANY {4,5,6} AND 3 >= ANY {4,5,6}
...

Compare this to the previous approach, where we will end up evaluating 12 different expressions, each time for a single point value:

First constraint of Q1: 1 <= 1 AND 3 >= 1
Second constraint of Q1: 1 <= 2 AND 3 >= 2
Third constraint of Q1: 1 <= 3 AND 3 >= 3
First constraint of Q2: 1 <= 4 AND 3 >= 4
...

The ScalarArrayOp depends on a new type of expression PartListRuleExpr that can convert a list rule to an array of values. ORCA specific changes can be found here: https://github.com/greenplum-db/gporca/pull/149

auto-vacuum limit would be reached first and then warn limit, followed by other
limits. So, there is no reason to rollback after bumping the xid to auto-vacuum
limit. Can land in all kinds of wierd issues by doing the same. Practically,
these tests need to be fully re-written maybe by modifying the GUCs and then
actually generating XID to reach the same instead of similating by bumping the
counter, but will attempt that in another commit.

In some of persistent table functions, GetTopTransactionId() was called inside
critical section. With laxy xid allocation if at DDL time transactionId stop
limit has reached, this causes simple ERROR upgraded to PANIC. Hence modify code
to call GetTopTransactionId() before entering critical section to avoid PANIC
but just have ERROR as before.

Commit fb86c90d "Simplify management of
distributed transactions." cleanedup lot of code for LocalDistribXactData and
introduced LocalDistribXactData in PROC for debugging purpose. But it's only
correctly maintained for QE's, QD never populated LocalDistribXactData in
MyProc. Instead TMGXACT also had LocalDistribXactData which was just set
initially for QD but never updated later and confused more than serving the
purpose. Hence removing LocalDistribXactData from TMGXACT, as it already has
other fields which provide required information. Also, cleaned-up QD related
states as even in PROC only QE uses LocalDistribXactData.

As part of 8.3 merge, upstream commit 295e6398
"Implement lazy XID allocation" was merged. But transactionIds were still
allocated in StartTransaction as code changes required to make it work for GPDB
with distrbuted transaction was pending, thereby feature remained as
disabled. Some progress was made by commit
a54d84a3 "Avoid assigning an XID to
DTX_CONTEXT_QE_AUTO_COMMIT_IMPLICIT queries." Now this commit addresses the
pending work needed for handling deferred xid allocation correctly with
distributed transactions and fully enables the feature.

Important highlights of changes:

1] Modify xlog write and xlog replay record for DISTRIBUTED_COMMIT. Even if
transacion is read-only for master and no xid is allocated to it, it can still
be distributed transaction and hence needs to persist itself in such a case. So,
write xlog record even if no local xid is assigned but transaction is
prepared. Similarly during xlog replay of the XLOG_XACT_DISTRIBUTED_COMMIT type,
perform distributed commit recovery ignoring local commit. Which also means for
this case don't commit to distrbuted log, as its only used to perform reverse
map of localxid to distributed xid.

2] Remove localXID from gxact, as its no more needed to be maintained and used.

3] Refactor code for QE Reader StartTransaction. There used to be wait-loop with
sleep checking to see if SharedLocalSnapshotSlot has distributed XID same as
that of READER to assign reader some xid as that of writer, for SET type
commands till READER actually performs GetSnapShotData(). Since now a) writer is
not going to have valid xid till it performs some write, writers transactionId
turns out InvalidTransaction always here and b) read operations like SET doesn't
need xid, any more hence need for this wait is gone.

4] Thow error if using distributed transaction without distributed xid. Earlier
AssignTransactionId() was called for this case in StartTransaction() but such
scenario doesn't exist hence convert it to ERROR.

5] QD earlier during snapshot creation in createDtxSnapshot() was able to assign
localXid in inProgressEntryArray corresponding to distribXid, as localXid was
known by that time. That's no more the case and localXid mostly will get
assigned after snapshot is taken. Hence now even for QD similar to QE's snapshot
creation time localXid is not populated but later found in
DistributedSnapshotWithLocalMapping_CommittedTest(). There is chance to optimize
and try to match earlier behavior somewhat by populating gxact in
AssignTransactionId() once locakXid is known but currently seems not so much
worth it as QE's anyways have to perform the lookups.