Given a database in which the "public" schema does not have its default
OID 2200 (e.g. it was dropped and recreated), pg_upgrade would ignore
the previous OID and use the built-in "public" schema at restoration.
This is because creation of the "public" schema is skipped in
dump / restore.

Add a binary_upgrade field to RestoreOptions, to teach pg_dump /
pg_restore not to skip the public schema during binary upgrade. This
adds a new "--binary-upgrade" option to pg_restore.
Co-authored-by: NJacob Champion <pchampion@pivotal.io>
Co-authored-by: NJesse Zhang <sbjesse@gmail.com>

Use gpstart '-B' ('--parallel') option value to limit the maximum size
of a thread pool used by gpsegstart to create segments.

Put a limit on the number of workers in a pool created by gpstart.

When there are many segments on one server node, the unlimited number
of workers may open unlimited number of pipes.
This may lead to a failure of 'select()' call
in 'gpMgmt/bin/gppylib/gpsubprocess.py' and inability
to start the cluster at all.

This commit prevents the described problem from happening.

Closes https://github.com/greenplum-db/gpdb/issues/6176

Discussion: https://github.com/greenplum-db/gpdb/pull/6410
Reviewed-by: Heikki Linnakangas

GPDB assigns 100KB for each non memory intensive operator, and report an
error if there is not enough memory reserved for the operators.  In low
memory resource groups it is easy to trigger this error with large
queries.

Improved by always assign at least 100KB memory to operators, both
memory intensive or non-intensive, the memory can be allocated from
shared memory, and will raise OOM error if there is not enough shared
memory.

* Disable Current Of for replicated table

Let's list the plans using CURRENT OF:

    explain delete from t1 WHERE CURRENT OF c1;
                            QUERY PLAN
    -----------------------------------------------------------
     Delete on t1  (cost=0.00..100.01 rows=1 width=10)
       ->  Tid Scan on t1  (cost=0.00..100.01 rows=1 width=10)
             Output: ctid, gp_segment_id
             TID Cond: CURRENT OF c1
             Filter: CURRENT OF c1

    explain UPDATE r1 SET b = 3 WHERE CURRENT OF c1;
                            QUERY PLAN
    -----------------------------------------------------------
     Update on r1  (cost=0.00..100.01 rows=1 width=18)
       ->  Tid Scan on r1  (cost=0.00..100.01 rows=1 width=18)
             Output: ctid, gp_segment_id
             TID Cond: CURRENT OF c1
             Filter: CURRENT OF c1

(ctid + gp_segment_id) can identify a unique row for hash distributed
table or randomly distributed table, but for replicated table, it can
only identify a row in one replica, so in this case other replicas
are not updated or deleted properly. There is no proper way to let
CURRENT OF work fine for replicated table, so just disable it for now.

Updatable view works fine now because all the operations are designed
to executed locally, rows of each replica are never flowed to other
segments and each replica have the same data flow. For example:
     create view v_r2 as select * from r2 where c2 = 3;

     explain update v_r2 set c2 = 4 from p1 where p1.c2=v_r2.c2;
                            QUERY PLAN
    -----------------------------------------------------------
     Update on r2
       ->  Nested Loop
             ->  Seq Scan on r2
                   Filter: (c2 = 3)
             ->  Materialize
                   ->  Broadcast Motion 3:3
                         ->  Seq Scan on p1
                               Filter: (c2 = 3)
As the plan shows, p1 is broadcasted, so all replicas of replicated
table operate locally

* docs - pxf CLI refactor, add ref pages for pxf, pxf cluster

* display pxf cluster first in ref landing page

* remedy -> solution

* remove java 1.7

* remove commands that source greenplum_path.sh

* edits requested by david

* fix link

* update OSS book for new ref pages, add jdbc, edits

* docs - ANALYZE - clarify using analyze on partitioned tables.

* docs - ANALYZE - review comment updates

The 'M' type QD->QE dispatch message includes a serialized version of the
current resource group information. That included a 'cpuset' field, which
has 1 kB of space reserved for it, and the serialization routine included
all the padding zeros in it. (Copying or memsetting that into a local
variable in SerializeResGroupInfo() wasn't completely free either).
Rewrite the serialized format to be less silly.
Reviewed-by: NJacob Champion <pchampion@pivotal.io>

numeric_round() and numeric_trunc() initialized the NumericVar to
represent the argument with init_var_from_num(), and then called
round_var() on it. That's not cool, init_var_from_num() creates a
read-only NumericVar, whose 'digits' points directly to the argument
datum, and round_var() scribbles on it. There is a comment in
init_var_from_num() warning explicitly to not do exactly that, but we
failed to heed the warning.

Fixes github issue https://github.com/greenplum-db/gpdb/issues/6383. I
also added a test case for this, although it's a bit hard to see this bug
to reappear in exactly the same form. It's not far-fetched that we might
re-introduce it in a slightly different form, though, so I also added some
extra Asserts in round_var() and trunc_var() for it.
Reviewed-by: NEkta Khanna <ekhanna@pivotal.io>

Commit e00ca2c4 moved gpfdist into
src/bin and also folded it into the main autoconf program for GPDB.
The local config/ files were however mistakenly left behind. Remove
the files as they are no longer used.

Co-authored-by: NChris Hajas <chajas@pivotal.io>
Co-authored-by: NSambitesh Dash <sdash@pivotal.io>

This test suite was mostly testing unsupported append-only table
functionality which has since long been implemented, but since the
tests were wrapped in ignore blocks it wasn't even testing that.

All the tests in this suite are covered by other suite which are
better maintained so remove to shave a few cycles off ICW time
(no wallclock time improvement of ICW is expected since it was
running in a parallel group).

Discussion: https://groups.google.com/a/greenplum.org/forum/#!topic/gpdb-dev/mxld3WyhPZ8Reviewed-by: NHeikki Linnakangas <hlinnakangas@pivotal.io>
Reviewed-by: NAshwin Agrawal <aagrawal@pivotal.io>

As the comments in code explain, the simple union all pullup optimization
is problematic in GPDB. I don't think we have plans to "fix" it any time
soon.

The RHEL 5 platform is no longer supported, so remove references to it
in the various enterprise build scripts.
Authored-by: NTaylor Vesely <tvesely@pivotal.io>

* docs - updates for jsonb type

* docs - jsonb docs - review comment updates.

* docs - jsonb docs - remove duplicate text.

Commit f8afec06 extended the zstd
check to also look for zstd_errors.h, and updated the comment. The
comment update was however only committed to configure and not the
source file configure.in. This synchronizes the files.

There was a FIXME about that. Add a test case for it, too, since we
apparently have no test coverage for gp_enable_motion_mk_sort=off.
Reviewed-by: NBaiShaoqi <sbai@pivotal.io>

Previously GPDB would refuse to pull up ANY sublink if the subquery
returns a set-returning function (SRF) in the targetlist, while
PostgreSQL will do that kind of pull-up.

This restriction was added in GPDB years ago to workaround MPP-7085.
That issue has been fixed in other place, so we can remove the
restriction now.

When all the cpu cores are allocated the default cpuset group should
fallback to core 0.  However this fallback logic was only added on
CREATE / ALTER RESOURCE GROUP, but missing in startup logic, an empty cpu
core list "" is set to cgroup and cause a runtime error:

    can't write data to file '/sys/fs/cgroup/cpuset/gpdb/1/cpuset.cpus':
      No space left on device (resgroup-ops-linux.c:916)

Fixed by converting "" to "-1" in startup logic.

GDB on centos/rhel6 doesn't recognize dwarf 4 symbols, as it was
originally implemented in GDB 7.4. Centos 6 ships with GDB 7.2, so it
can't read the newer dwarf formats.

From the GCC manual:

-gstrict-dwarf

	Disallow using extensions of later DWARF standard version than
	selected with -gdwarf-version. On most targets using
	non-conflicting DWARF extensions from later standard versions is
	allowed.

-gdwarf-[version]

	Produce debugging information in DWARF format (if that is supported).
	The value of version may be either 2, 3, 4 or 5; the default version for
	most targets is 4. DWARF Version 5 is only experimental.
Co-authored-by: NTaylor Vesely <tvesely@pivotal.io>
Co-authored-by: NBen Christel <bchristel@pivotal.io>

This removes gpexpand_1 and gpexpand_2 concourse jobs in favor of the
gpexpand behave tests.
Co-authored-by: NDavid Krieger <dkrieger@pivotal.io>
Co-authored-by: NKalen Krempely <kkrempely@pivotal.io>

Update documentation to match the default setting when adding the option
in gpMgmt/bin/gppylib/programs/clsRecoverSegment.py:

```
    addTo.add_option("-B", None, type="int", default=16,
```

[ci skip]
Co-authored-by: NDavid Krieger <dkrieger@pivotal.io>
Co-authored-by: NKalen Kempely <kkrempely@pivotal.io>
Co-authored-by: NJim Doty <jdoty@pivotal.io>

We can use the more straightforward cdbhashrandomseg() instead.
Reviewed-by: NDaniel Gustafsson <dgustafsson@pivotal.io>

- Some tests were expanding into /tmp which ran out of space, so
now expand into /data/gpdata.
- Consolidate test to verify redistribution after expand.
- Actually use dump of ICW database in the relevant test.
Co-authored-by: NDavid Krieger <dkrieger@pivotal.io>
Co-authored-by: NJim Doty <jdoty@pivotal.io>
Co-authored-by: NKalen Krempely <kkrempely@pivotal.io>

Because expand copies the entire master directory, we need to zero out the
distributed_xlog since the segment is not yet part of the cluster, to
allow it to only see local transactions.  Fixed the log level to WARN
when expanding a cluster with unique indexes.
Co-authored-by: NDavid Krieger <dkrieger@pivotal.io>
Co-authored-by: NJim Doty <jdoty@pivotal.io>
Co-authored-by: NKalen Krempely <kkrempely@pivotal.io>

This was previously not implemented, because the gp_distribution_policy
catalog was not populated in QE segments. Starting with commit 7efe3204,
it is, so we can easily support this now.
Reviewed-by: NXiaoran Wang <xiwang@pivotal.io>
Reviewed-by: NAdam Lee <ali@pivotal.io>

The gang management now has a smaller granularity and a better management
of memory context, meanwhile, gangs are not pre-assigned anymore for main
plan and init plans. An issue was reported for old implementation of gang
management, although it cannot been reproduced now, it's still necessary
to add a test case to avoid regression in the feature.

The test strategy is preparing a query contains both main and init plan,
main plan create a unnamed portal and will need multiple slices , init
plan also need to allocated multiple slices in a named portal (eg: a
cursor), the expect result is gang management works fine in such mixed
combination.

Calling ereport(ERROR..) will break out of any codepath, never to
return, so returning NULL after invocation is dead code. Resources
are also automatically clenaed on erroring out, so remove the pfree
call (which is also dead code) rather than moving it before the
ereport.
Reviewed-by: NJacob Champion <pchampion@pivotal.io>
Reviewed-by: NVenkatesh Raghavan <vraghavan@pivotal.io>
Reviewed-by: NJimmy Yih <jyih@pivotal.io>

For consistency: this is how we represent column indexes e.g. in Sort,
Unique, MergeAppend and many other plan types.
Reviewed-by: NMelanie Plageman <mplageman@pivotal.io>

Create the CdbHash object in the initialization phase, and reuse it for
all the tuples. I'm not sure how much performance difference this makes,
but seems cleaner anyway.
Reviewed-by: NMelanie Plageman <mplageman@pivotal.io>

ORCA generated plans, where the "hash filter" in the Result node was set
to an empty set of columns. That meant "discard all the rows, on all
segments, except one segment". This is used at least with set-returning
functions, where we don't care where the function is executed, but it only
needs to be executed once. (The planner creates a one-to-many Redistribute
Motion plan in that scenario, which makes a lot more sense to me, but
doing the same in ORCA would require more invasive surgery than what I'm
capable of.)

Instead of executing the subplan, and throwing away the result one row at
a time, use a Result plan with a One-Off Filter. That's more efficient.
Also, it allows removing the Result.hashFilter boolean flag, because there
the weird case of a hashFilter with zero columns is gone. You can check
"hashList != NIL" directly now.

The old method would always choose the same segment, which seems bad for
load distribution. The way it was chosen seemed totally accidental too:
we initialized the cdbhash object to the initial constant value, and
then reduced that into the target segment number, using the jump
consistent hash algorithm. We computed that for every row, but the result
was always the same. On a three-node cluster, the target was always
segment 1. Now, we pick a segment at random when generating the plan.
Reviewed-by: NMelanie Plageman <mplageman@pivotal.io>

This was moved to encnames.c in upstream commit 420ea688, but for some
reason, we failed to remove it in the 8.4 merge from the old location.
It's unused.