We already had a hack for the EXECUTE ON ALL SEGMENTS case, by setting
prodataaccess='s'. This exposes the functionality to users via DDL, and adds
support for the EXECUTE ON MASTER case.

There was discussion on gpdb-dev about also supporting ON MASTER AND ALL
SEGMENTS, but that is not implemented yet. There is no handy "locus" in the
planner to represent that. There was also discussion about making a
gp_segment_id column implicitly available for functions, but that is also
not implemented yet.

The old behavior was that a function that if a function was marked as
IMMUTABLE, it could be executed anywhere. Otherwise it was always executed
on the master. For backwards-compatibility, this keeps that behavior for
EXECUTE ON ANY (the default), so even if a function is marked as EXECUTE ON
ANY, it will always be executed on the master unless it's IMMUTABLE.

There is no support for these new options in ORCA. Using any ON MASTER or
ON ALL SEGMENTS functions in a query cause ORCA to fall back. This is the
same as with the prodataaccess='s' hack that this replaces, but now that it
is more user-visible, it would be nice to teach ORCA about it.

The new options are only supported for set-returning functions, because for
a regular function marked as EXECUTE ON ALL SEGMENTS, it's not clear how
the results should be combined. ON MASTER would probably be doable, but
there's no need for that right now, so punt.

Another restriction is that a function with ON ALL SEGMENTS or ON MASTER can
only be used in the FROM clause, or in the target list of a simple SELECT
with no FROM clause. So "SELECT func()" is accepted, but "SELECT func() FROM
foo" is not. "SELECT * FROM func(), foo" works, however. EXECUTE ON ANY
functions, which is the default, work the same as before.

If there are aggregation queries with aliases same as the table actual
columns and they are propagated further from subqueries and grouping is
applied on the column alias it may result in inconsistent targetlists
for aggregation plan causing crash.

	CREATE TABLE t1 (a int) DISTRIBUTED RANDOMLY;
	SELECT substr(a, 2) as a
	FROM
		(SELECT ('-'||a)::varchar as a
			FROM (SELECT a FROM t1) t2
		) t3
	GROUP BY a;

In this commit, we add more detailed memory metrics to the 'memory_usage'
column of gp_resgroup_status include current/available memory usage in
a group, current/available memory usage for a slot, current/available
memory usage for the shared part.

Previously, waiters waiting on a dropped resource group need to be
reassigned to a new group, to achieve it, ResGroupSlotAcquire is
modified to be complicated and not easy to understand, this commit
refines it.

Author: Gang Xiong <gxiong@pivotal.io>

Allow CREATE RESOURCE GROUP and ALTER RESOURCE GROUP to set concurrency
to 0, so there will eventually be no running queries after some time, so
the resource group can be dropped. On drop all pending queries will be
moved to the new resource group assigned to the role; but if the role is
also dropped the pending queries will all be canceled. Another thing is
we do not allow setting concurrency of admin group to zero, superuser is
under admin group and only superuser can alter resource group, so once
concurrency of admin group is set to zero, there will be no chance to set
it again.
Signed-off-by: NNing Yu <nyu@pivotal.io>

Because we don't know the data location of the result of SELECT query,
ON SEGMENT is forbidden.

This commit does two changes:
1. Remove the restriction that sum of memory_spill_ratio and memory_shared_quota
must be no larger than 100.
2. Change the range of memory_spill_ratio to be [0, 100].

Function FaultInjectorIdentifierStringToEnum(faultName) pass a const
string to a non-const parameter, which cause a build warnig. But on the
second thought, we have supported injecting fault by fault name without
corresponding fault identifier, so it's better to use faultname instead
of fault enum identifier in the ereport.

Adds a clusterstart command to gpsegwalrep.py allow a user to start a
cluster with WALRep configured. This is a developer utility that assumes
all cluster replicas are present on localhost, and thus is not intended
for production use.

* docs - memory_spill_ratio guc and related content

* operator -> transaction

* Remove dependency on system curl; Fix bug with OSX

* Add ifdef for CURLOPT_RESOLVE

* Incorporate feedback

* brew curl not needed anymore

GPOS raises exception with different severity level, but
they were being logged to GPDB logs at LOG severity level.
This disabled users to not turn off logging for GPOS exceptions, unless
GPDB log setting was changed higher than LOG severity level.

This is the initial commit which introduces the functionality. If an
exception is created without the GPDB severity level, it will default to
LOG severity level in GPDB.
Signed-off-by: NJemish Patel <jpatel@pivotal.io>

Signed-off-by: NAbhijit Subramanya <asubramanya@pivotal.io>

Signed-off-by: NXin Zhang <xzhang@pivotal.io>

New API: void set_gp_replication_config(const char *name, const char *value)

This function is inspired by the upstream ALTER SYSTEM command
AlterSystemSetConfigFile() from commit
7dfab04a.

Once we merged the upstream changes, we can remove this function and
directly use the AlterSystemSetConfigFile().
Signed-off-by: NAbhijit Subramanya <asubramanya@pivotal.io>

We use this file to store the GUC value `sychronous_standby_names` in
order to control the blocking behavior between primary and mirrors as
used in upstream. When this GUC is on, the primary is blocked and
waiting for the commits propagated to mirrors regardless of mirror
status. When this GUC is off, primary just archive and won't wait for
mirrors.

gp_replication.conf is now read unconditionally by the GUC parsing logic
and needs to be set up by initdb. Refactor set_null_conf() to take a
filename so that we don't copy-paste more code.
Signed-off-by: NJacob Champion <pchampion@pivotal.io>

add_assignment previously didn't handle the case where an existing
config assignment was not found, and segfaulted. Fix that by inserting
brand-new assignments at the end of the config list.

The pipeline was failing as the ORCA tag for 2.45.0 was not pushed.
It is currently available.

These shouldn't happen, but Coverity warned about these. GCC would also
complain, but I've been compiling with -Wno-maybe-uninitialized lately,
because of noise.

Actually, this isn't quite enough; ORCA also needs to mark GPOS_RAISE
with the "noreturn" attribute, so that the compiler gets the hint.
Opened https://github.com/greenplum-db/gporca/pull/234 about that.

Using fault name instead of enum as the key of fault hash table

GPDB fault injector uses fault enum as the key of fault hash table.
If someone wants to inject fault into gpdb extensions(a separate repo),
she has to hard code the extension related fault enums into gpdb core
code, this is not a good practice.
So we simply use fault name as the hash key to remove the need of hard
code the fault enum. Note that fault injector API doesn't change.

This reverts commit 378426fe.

> --2017-09-17 12:12:00--  https://github.com/greenplum-db/gporca/releases/download/v2.45.0/bin_orca_centos5_release.tar.gz
> Resolving github.com... 192.30.255.113, 192.30.255.112
> Connecting to github.com|192.30.255.113|:443... connected.
> HTTP request sent, awaiting response... 404 Not Found
> 2017-09-17 12:12:00 ERROR 404: Not Found.

In GPDB, we have so far used a WindowFrame struct to represent the start
and end window bound, in a ROWS/RANGE BETWEEN clause, while PostgreSQL
uses the combination of  a frameOptions bitmask and start and end
expressions. Refactor to replace the WindowFrame with the upstream
representation.

This removes pg_window.winframemakerfunc column. It was only used for
LEAD/LAG, and only in the Postgres planner. Hardcode the same special
handling for LEAD/LAG in planwindow.c instead, based on winkind.

This is one step in refactoring the planner and executor further, to
replace the GPDB implementation of window functions with the upstream
one.

I was about to add this as part of the PostgreSQL 8.4 merge, as a check
when upgrading from 8.3 to 8.4, because the hash algorithm was changed
in 8.4. However, turns out that pg_dump doesn't support hash partitioned
tables at all, so pg_upgrade won't work on a database that contains any
hash partitioned tables, even on a same-version upgrade. Hence, let's
add this check unconditionally on all server versions.

There are comments talking about the hash function change, because of that
devleopment history. I think that's useful documentation, just in case
we ever start to support hash partitions in pg_dump, so I left it there.

Upstream uses >= here. It was changed in GPDB, to use > instead of >=. but
I don't see how that's more correct or better. I tracked that change in
the old pre-open-sourcing repository to this commit:

commit f3e98a1ef5fc5915662077b137c563371ea1c0a4
Date: Mon Apr 6 15:04:33 2009 -0800

   Fixed guc check for ReservedBackends.

   [git-p4: depot-paths = "//cdb2/main/": change = 33269]

So, there was no explanation there either, what the alleged problem was.

Should call setQryDistributionPolicy() after applyColumnNames(), otherwise
the column names specified in the CREATE TABLE cannot be used in the
DISTRIBUTED BY clause. Add test case.

Fixes github issue #3285.

Historically this function was used to special case a few operators that
were not considered to be MemoryIntensive. However, now it always
returns true. This commit removes the function and also moves the case
for T_FunctionScan in IsMemoryIntensiveOperator into the group that
always returns true, as this is its current behavior

Signed-off-by: NXin Zhang <xzhang@pivotal.io>

While attempting to decorrelate the subquery, we were
incorrectly pulling up the join before calculating the window function
of the results of the join. In cases where we have subqueries with
window function and the subquery has outer references we should not
attempt decorrelating it.

There are further optimization which can be done in case of existential
queries but this PR fixes the plan.
Signed-off-by: NJemish Patel <jpatel@pivotal.io>

* docs: COPY command add PROGRAM clause

* docs: copy - edits from review comments.

Again attempting a blind fix..

I can't easily run these tests myself, so hoping I get this fixed blindly..

The bzip2 library is only used by the gfile/fstream code, used for external
tables and gpfdist. The usage of bzip2 was in #ifndef WIN32 blocks, so it
was only built on non-Windows systems.

Instead of tying it to the platform, use a proper autoconf check and
HAVE_LIBBZ2 flags. This makes it possible to build gpfdist with bzip2
support on Windows, as well as building without bzip2 on non-Windows
systems. That makes it easier to test the otherwise Windows-only codepaths
on other platforms. --with-libbz2 is still the default, but you can now use
--without-libbz2 if you wish.

I'm sure that some regression tests will fail if you actually build the
server without libbz2, but I'm not going to address that right now. We have
similar problems with other features that are in principle optional, but
cause some regression tests to fail.

Also use "#ifdef HAVE_LIBZ" rather than "#ifndef WIN32" to enable/disable
zlib support in gpfdist. Building the server still fails if you use
--without-zlib, but at least you can build the client programs without
zlib, also on non-Windows systems.

Remove obsolete copy of bzlib.h from the repository while we're at it.

If a the sample of a column consists entirely of "too wide" values, which
are left out of the sample when it's passed to the compute_stats function,
we pass an empty sample to it. The default compute_stats gets confused by
that, and computes the null fraction as 0 / 0 = NaN, so we end up storing
NaN as stanullfrac.

If all the values in the sample are wide values, then they're surely not
NULLs, so the right thing to do is to store stanullfrac = 0. That is a
bit non-linear with the normal compute_stats function, which effectively
treats too wide values as not existing at all, which artificially inflates
the null fraction. Another non-linear thing is that we store stawidth=1024
in this special case, but the normal computation again ignores the wide
values in computing stawidth. If we wanted to do something about that, we
should adjust the normal computation to take those wide values better into
account, but that's a different story, and now we at least won't store NaN
in stanullfrac any longer.

Fixes github issue #3259.

Commit b4f125bd changed ALTER TYPE SET DEFAULT ENCODING to no longer
accept SQL type aliases. A consequence of that is that "char" no longer
meand "character varying", but actual "char" datatype. Change the tests
to use the PostgreSQL name for that datatype, "bpchar".