After merging with e006a24a, Anti Semi Join will
be denoted by `JOIN_ANTI` instead of `JOIN_LASJ`

Ref [#142355175]
Signed-off-by: NDhanashree Kashid <dkashid@pivotal.io>

Since `InClauseInfo` and `OuterJoinInfo` are now combined into
`SpecialJoinInfo` after merging with e006a24a; this commit remove them
from the relevant places.

Access `join_info_list` instead of `in_info_list` and `oj_info_list`

Previously, `CdbRelDedupInfo` contained list of `InClauseInfo` s. While
making join decisions and overall join processing, we traversed this list
and invoked cdb specific functions: `cdb_make_rel_dedup_info()`, `cdbpath_dedup_fixup()`

Since `InClauseInfo` is no longer available,  `CdbRelDedupInfo` will contain list of
`SpecialJoinInfo` s. All the cdb specific routines which were previously called for
`InClauseInfo` list will now be called if `CdbRelDedupInfo` has valid `SpecialJoinInfo`
list and if join type in `SpecialJoinInfo` is `JOIN_SEMI`. A new helper routine `hasSemiJoin()`
has been added which traverses `SpecialJoinInfo` list to check if it contains `JOIN_SEMI`.

Ref [#142355175]
Signed-off-by: NDhanashree Kashid <dkashid@pivotal.io>

After merging with e006a24a, the jointype JOIN_IN has been renamed to
JOIN_SEMI.
This commit makes appropriate changes in ORCA for the same.

Ref [#142355175]
Signed-off-by: NDhanashree Kashid <dkashid@pivotal.io>

Add pullup decisions specific to CDB from `convert_IN_to_join`

Ref [#142355175]
Signed-off-by: NDhanashree Kashid <dkashid@pivotal.io>

After merging with e006a24a, this commit adds CDB specific restrictions
as follows:

0. Add pullup decisions specific to CDB from
`convert_EXISTS_to_join` and `convert_NOT_EXISTS_to_join`

0. Before this cherry-pick, we used to generate extra quals
for the NOT EXISTS query. This was done by calling `cdbpullup_expr()`
in `convert_NOT_EXISTS_to_join()`.
However, for the exact same query with EXISTS, we never generated these
extra quals.
```
create table foo(t text, n numeric, i int, v varchar(10)) distributed by (t);
explain select * from foo t0 where not exists (select 1 from foo t1 where t0.i=t1.i + 1);
                                           QUERY PLAN
-------------------------------------------------------------------------------------------------
 Gather Motion 3:1  (slice2; segments: 3)  (cost=1.08..2.12 rows=4 width=19)
   ->  Hash Left Anti Semi Join  (cost=1.08..2.12 rows=2 width=19)
         Hash Cond: t0.i = (t1.i + 1)
         ->  Seq Scan on foo t0  (cost=0.00..1.00 rows=1 width=19)
         ->  Hash  (cost=1.04..1.04 rows=1 width=4)
               ->  Broadcast Motion 3:3  (slice1; segments: 3)  (cost=0.00..1.04 rows=1 width=4)
                     ->  Seq Scan on foo t1  (cost=0.00..1.00 rows=1 width=4)
                           Filter: (i + 1) IS NOT NULL  -> extra filter
 Settings:  optimizer=off
 Optimizer status: legacy query optimizer
(10 rows)

explain select * from foo t0 where exists (select 1 from foo t1 where t0.i=t1.i + 1);
                                           QUERY PLAN
-------------------------------------------------------------------------------------------------
 Gather Motion 3:1  (slice2; segments: 3)  (cost=1.08..2.12 rows=4 width=19)
   ->  Hash EXISTS Join  (cost=1.08..2.12 rows=2 width=19)
         Hash Cond: t0.i = (t1.i + 1)
         ->  Seq Scan on foo t0  (cost=0.00..1.00 rows=1 width=19)
         ->  Hash  (cost=1.04..1.04 rows=1 width=4)
               ->  Broadcast Motion 3:3  (slice1; segments: 3)  (cost=0.00..1.04 rows=1 width=4)
                     ->  Seq Scan on foo t1  (cost=0.00..1.00 rows=1 width=4)
 Settings:  optimizer=off
 Optimizer status: legacy query optimizer
(9 rows)

```
Currently with this commit, the combined pull-up code for EXISTS and NOT EXISTS
does not generate extra filters. This will be a future TODO.

0. Use `is_simple_subquery` in `simplify_EXISTS_query` to check if
subquery can be pulled up or not.

Ref [#142355175]
Signed-off-by: NDhanashree Kashid <dkashid@pivotal.io>

With the new flow, we don't need the following functions:

 - pull_up_IN_clauses
 - convert_EXISTS_to_join
 - convert_NOT_EXISTS_to_antijoin
 - not_null_inner_vars
 - safe_to_convert_NOT_EXISTS
 - convert_sublink_to_join

Ref [#142355175]
Signed-off-by: NEkta Khanna <ekhanna@pivotal.io>

Original Flow:
cdb_flatten_sublinks
	+--> pull_up_IN_clauses
		+--> convert_sublink_to_join

New Flow:
cdb_flatten_sublinks
	+--> pull_up_sublinks

This commit contains relevant changes for the above flow.

Previously, `try_join_unique` was part of `InClauseInfo`. It was getting
set in `convert_IN_to_join()` and used in `cdb_make_rel_dedup_info()`.
Now, since `InClauseInfo` is not present and we construct
`FlattenedSublink` instead in `convert_ANY_sublink_to_join()`. And later
in the flow, we construct `SpecialJoinInfo` from `FlattenedSublink` in
`deconstruct_sublink_quals_to_rel()`. Hence, adding `try_join_unique` as
part of both `FlattenedSublink` and `SpecialJoinInfo`.

Ref [#142355175]
Signed-off-by: NEkta Khanna <ekhanna@pivotal.io>

commit e006a24a
Author: Tom Lane <tgl@sss.pgh.pa.us>
Date:   Thu Aug 14 18:48:00 2008 +0000

    Implement SEMI and ANTI joins in the planner and executor.  (Semijoins replace
    the old JOIN_IN code, but antijoins are new functionality.)  Teach the planner
    to convert appropriate EXISTS and NOT EXISTS subqueries into semi and anti
    joins respectively.  Also, LEFT JOINs with suitable upper-level IS NULL
    filters are recognized as being anti joins.  Unify the InClauseInfo and
    OuterJoinInfo infrastructure into "SpecialJoinInfo".  With that change,
    it becomes possible to associate a SpecialJoinInfo with every join attempt,
    which permits some cleanup of join selectivity estimation.  That needs to be
    taken much further than this patch does, but the next step is to change the
    API for oprjoin selectivity functions, which seems like material for a
    separate patch.  So for the moment the output size estimates for semi and
    especially anti joins are quite bogus.

Ref [#142355175]
Signed-off-by: NEkta Khanna <ekhanna@pivotal.io>

* admin guide working w/dbs section - pull some topics up a level

* promote ddl, crud topics; move querying topic up

Behave by default just grabs the latest parse_type package as part of
the setup requirement. However, the newest parse_type package (0.4.2)
uses a new convention on how it handles packaging which is not in the
older python versions before 2.7.13. Since we're in python 2.7.12, we
break.

Force requirement to use an older version as a hack to bypass this
issue.
Signed-off-by: NMarbin Tan <mtan@pivotal.io>

This reverts commit f35b369c.

Several GUCs are simply enumerated strings that are parsed into integer
types behind the scenes. As of 8.4, the GUC system recognizes a new
type, enum, which will do this for us. Move as many as we can to the new
system.

As part of this,
- gp_idf_deduplicate was changed from a char* string to an int, and new
  IDF_DEDUPLICATE_* macros were added for each option
- password_hash_algorithm was changed to an int
- for codegen_optimization_level, "none" is the default now when codegen
  is not enabled during compilation (instead of the empty string).

A couple of GUCs that *could* be represented as enums
(optimizer_minidump, gp_workfile_compress_algorithm) have been
purposefully kept with the prior system because they require the GUC
variable to be something other than an integer anyway.
Signed-off-by: NJacob Champion <pchampion@pivotal.io>

We had a very simplistic implementation in parse-analysis already, which
converted the FILTER WHERE clause into a CASE-WHEN expression. That did
not work for non-strict aggregates, and didn't deparse back into a FILTER
expression nicely, to name a few problems with it. Replace it with the
PostgreSQL implementation.

TODO:

* ORCA support. It now falls back to the Postgres planner.
* I disabled the three-stage DQA plan types if there are any FILTERs

Signed-off-by: NAshwin Agrawal <aagrawal@pivotal.io>

Adding timestamp to better trace the command.

Adding ClusterConfiguration to improve the reuse, and remove
ColdStartMaster class.
Signed-off-by: NXin Zhang <xzhang@pivotal.io>

Under SEGWALREP, the mode for gp_segment_configuration is 'n' for
not-in-sync and 's' for in-sync.

When new primary or mirror is added to the cluster, the initial state of
the mode should be 'n' instead of 's' (in the original filerep mode.)

If mirror exists and synchronized with the primary, then the mode will
be updated to 's' to indicate the primary-mirror pair are in-sync.
Signed-off-by: NAshwin Agrawal <aagrawal@pivotal.io>

Originally, gp_add_segment() is only used to add a primary segment. We
created a new gp_add_segment_primary() to keep the original
functionality, which automatically generate dbid and contentid.

We generalized the gp_add_segment() to be able to directly update
gp_segment_configuration and pg_filespace_entry for adding any type of
segments with full specification of segment and filespace mappings. In
this case, the new gp_add_segment() doesn't generate dbid and contentid
automatically, and rely on input parameters.

Originally there is separate code path in gp_add_segment_mirror() to
figure out primary dbid, such logic is actually common even with
gp_add_segment(), which can add both primary and mirror.

In that case, we refactor the primary dbid detection logic in common
function add_segment(), and refactor the gp_add_segment_mirror() to use
the add_segment() instead of add_segment_config_entry().

We update gpinitsystem to use the function gp_add_segment() instead of
update the gp_segment_configuration and pg_filespace_entry tables
directly via SQL.
Signed-off-by: NXin Zhang <xzhang@pivotal.io>

Signed-off-by: NAshwin Agrawal <aagrawal@pivotal.io>

- Move Conan instructions above manual orca setup instructions
  so that people don't do manual work first (like we did).
- Inline recommendation to parallelize make with -j8.
- Fix formatting of code blocks in README.md

See issues:
 - https://github.com/greenplum-db/gpdb/issues/3178
 - https://github.com/greenplum-db/gpdb/pull/3332

This commit will display the contents of the Optimizer Mem Account when
the optimizer GUC is on and explain_memory_verbosity is set to
'summary'.
Signed-off-by: NSambitesh Dash <sdash@pivotal.io>

When we merged with PostgreSQL 8.3, and got support for NULLS FIRST, we
missed or stubbed out some code, that is required to honor NULLS FIRST in an
aggregate's ORDER BY, e.g. "select array_agg(a order by b desc nulls last)"

Signed-off-by: NTaylor Vesely <tvesely@pivotal.io>

It wasn't very useful. ORCA and Postgres both just stack WindowAgg nodes
on top of each other, and no-one's been unhappy about that, so we might as
well do that, too. This reduces the difference between GPDB and the upstream
implementation, and will hopefully make it smoother to switch.

Rename the Window Plan node type to WindowAgg, to match upstream, now
that it is fairly close to the upstream version.

To match upstream.

This test case could throw either "ROWS parameter cannot be negative", or
"Division By Zero", depending on which gets evaluated first. Remove the
division by zero error, to make it more predictable.

This case is to test cancel/terminate queries concurrently while they are 
running or waiting in resource group.

A Motion node often needs to "merge" the incoming streams, to preserve the
overall sort order. Instead of carrying sort order information throughout
the later stages of planning, in the Flow struct, pass it as argument
directly to make_motion() and other functions, where a Motion node is
created. This simplifies things.

To make that work, we can no longer rely on apply_motion() to add the final
Motion on top of the plan, when the (sub-)query contains an ORDER BY. That's
because we no longer have that information available at apply_motion(). Add
the Motion node in grouping_planner() instead, where we still have that
information, as a path key.

When I started to work on this, this also fixed a bug, where the sortColIdx
of plan flow node may refer to wrong resno. A test case for that is
included. However, that case was since fixed by other coincidental changes
to partition elimination, so now this is just refactoring.

Concourse doesn't support AIX natively, we need to clone the repo
with the correspond commit on remote machine, compile the packages,
and download them back to concourse container as output.

Testing client and loader for platform without gpdb server is
another challenge. We setup GPDB server on concourse container just
like most installcheck tests, and use SSH tunnel to forward ports
from and to the remote host. This way both CL tools and GPDB server
feel they are on the same machine, and the test can run normally.

Replace popen() with popen_with_stderr() which is used in external web
table also to collect the stderr output of program.

Since popen_with_stderr() forks a `sh` process, it's almost always
sucessful, this commit catches errors happen in fwrite().

Also passes variables as the same as what external web table does.
Signed-off-by: NXiaoran Wang <xiwang@pivotal.io>

Previous code us python package psutil to get the mount
information of the system which will read the content
of /etc/mtab. In some environments, /etc/mtab does not
contain the mount point information of cgroups. In this
commit, we scan /proc/self/mounts to find out cgroup
mount point.

There are cases where during execution a Memory Intensive Operator (MI)
may not use all the memory that is allocated to it. This means that this
extra memory (quota - allocated) can be relinquished for other MI nodes
to use during execution of a statement. For example

->  Hash Join
         ->  HashAggregate
         ->  Hash
In the above query fragment the HashJoin operator has a MI operator for
both its inner and outer subtree. If there ever is the case that the
Hash node used much less memory than was given as its quota it will now
call MemoryAccounting_DeclareDone() and the difference between its
quota and allocated amount will be added to the allocated amount of the
RelinquishedPool. Doing this will enable HashAggregate to request memory
from this RelinquishedPool if it exhausts its quota to prevent spilling.

This PR adds two new API's to the MemoryAccounting Framework

MemoryAccounting_DeclareDone(): Add the difference between a memory
account's quota and its allocated amount to the long living
RelinquishedPool

MemoryAccounting_RequestQuotaIncrease(): Retrieve all relinquished
memory by incrementing an operator's operatorMemKb and setting the
RelinquishedPool to 0

Note: This PR introduces the facility for Hash to relinquish memory to
the RelinquishedPool memory account and for the Agg operator
(specifically HashAgg) to request an increase to its quota before it
builds its hash table. This commit does not generally apply this
paradigm to all MI operators
Signed-off-by: NSambitesh Dash <sdash@pivotal.io>
Signed-off-by: NMelanie Plageman <mplageman@pivotal.io>

Before this cherry-pick the below query would have errored out

WITH outermost(x) AS (
  SELECT 1
  UNION (WITH innermost as (SELECT 2)
         SELECT * FROM innermost
         UNION SELECT 3)
)
SELECT * FROM outermost;
Signed-off-by: NMelanie Plageman <mplageman@pivotal.io>

To update 5.json, we ran:

cat src/include/catalog/*.h | perl src/backend/catalog/process_foreign_keys.pl > gpMgmt/bin/gppylib/data/5.json
Signed-off-by: NJacob Champion <pchampion@pivotal.io>