* Add log_temp_files guc

* Add subnav log_temp_files to ditamap

* Add mention of spill files and pg_log

The only user-visible effect is that the "non integer constant in %s" error
message now gets an error location.

This is from upstream commit 517ae403. We had already backported most of
that commit, so let's take it to completion.

Because Greenplum uses persistent tables (PT), if `gp_disable_tuple_hints` is
false, then reading PT tables during startup will cause hint bit
change, and attempt to mark page dirty, which will try to create a
Full Page Image in XLOG.

Because we are in init processing, write to XLOG is disallowed. We must
protect against the XLOG write.
Signed-off-by: NXin Zhang <xzhang@pivotal.io>
Signed-off-by: NAshwin Agrawal <aagrawal@pivotal.io>

* resource groups - sql ref pages - create new, updates to others

* david s requested edits

* more param description updates

* cannot submit RG DDL cmds in explicit transaction

* can now alter memory limit and shared; syntax corrections

* create RG - clarify MEMORY_SHARED_QUOTA and MEMORY_SPILL_RATIO descripts

* small edits requested by simon

* clarify default group usage

* remove reference to rg name NONE

* docs - add some resource group GUCs to reference guide

* incoporate david s requested edits

* remove resource_scheduler guc, edits from simon

Commit f51f2f57 backported upstream commit caddcb8f
but missed one destroyPQExpBuffer() call in the merge. Add to avoid
a tiny leak in the early exit path and to align us with upstream.

If src is null, or if strdup() fails, then srcdup will be null and
so will srcdup_start as it's a pointer to srcdup.  Regardless of
which, we can limit us to test srcdup to catch both error cases.
Also, avoid doublefree by not freeing srcdup_start.

getaddrinfo() returns a set of addrinfo structs, which needs to be
freed again with freeaddrinfo. Ensure to return resources before we
leave socket setup.

The loop failed to close the opened file descriptors causing a
small leak. Clean up as we go along.

We want to set toFile to point to the memory allocated by Safe_strdup,
not change toFile.

* docs - resource group system views to sys catalog ref guide

* incorporate requested edits from david

* clarify cpu/memory real-time usage description

This is potentially a tiny bit faster, if the coercion can be performed
just once at parse/plan time, rather than on every row.

This fixes some of the bogus error checks and inconsistencies in handling
the ROWS expressions. For example, before, if you passed a string constant
as the ROWS expression, you got an error, but if you passed a more
complicated expression, that returned a string, the string was cast to an
integer at runtime. And those casts evaded the plan-time checks for
negative values.

Also, move the checks for negative ROWS/RANGE value from the parser to the
beginning of execution, even in the cases where the value is a constant, or
a stable expression that only needs to be executed once. We were missing
the checks in ORCA, so this fixes the behavior with ORCA for such queries.

Consistently use 64-bit integers for the variables and fields that hold
row counts, in the window functions code. These are used for the ROWS
PRECEDING / FOLLOWING clauses. I don't expect anyone to deal with more than
2 billion rows in a single query's window function, but better safe than
sorry. Also, upstream uses int64 for these, and bigint is also used for
OFFSET/LIMIT, so let's be consistent.

There were two problems with the existing NULL check. Firstly, it would
not catch expressions that returned NULL at runtime, only NULL constants.
You got a non-friendly "Unexpected internal error" error instead. Secondly,
it rejected expressions that contained NULL constants anywhere in the
expression, even if the expression as whole returned a non-NULL.

To fix both issues, add a NULL-check with a better error message to the
executor, and remove the parse-time check altogether. I copied the error
message texts from the upstream, so that you get the same error, even
though the implementation is quite different.

Add regression tests for those cases too, we apparently didn't have any.

Fixes github issue #2999.

The way ORCA was tied into the planner, running a planner_hook
was not supported in the intended way. This commit moves ORCA
into standard_planner() instead of planner() and leaves the hook
for extensions to make use of, with or without ORCA. Since the
intention with the optimizer GUC is to replace the planner in
postgres, while keeping the planning proess, this allows for
planner extensions to co-operate with that.

In order to reduce the Greenplum footprint in upstream postgres
source files for future merges, the ORCA functions are moved to
their own file.

Also adds a memaccounting class for planner hooks since they
otherwise ran in the planner scope, as well as a test for using
planner_hooks.

Both versions of the function, the one in readfast.c and readfuncs.c, were
identical. No need to define a special fast version then.

The alteration of memory_spill_ratio will take effect immediately.

In addition, remove 'SET statement_mem' from resource group
test cases, as this GUC will take no effect in resource group.

We make a copy of resource group capabilities in the shared memory, and
apply the change at the end of the ALTER transaction. In this way, we can
access shared memory content instead of catalog at the beginning of a
transaction as the visibility is the same.
Signed-off-by: NNing Yu <nyu@pivotal.io>

In some resource group test cases, the result will depend on the
number of segments, and current test cases are based on three
segments. This commit is to fix this issue.

In addition, in isolation2, gpdiff needs at least 3 dashes to identify 
the result of 'select' statement. Change the elements to be above 
100 in test case 'resgroup_memory_materialize_spill' to meet this 
requirement.

Hopefully reduces merge conflicts a bit.

Perhaps it's slightly useful, but prefer to stay in sync with upstream.

Hopefully reduce merge conflicts in the future.

gp_delete_persistent_*_node_entry() are currently only used for testing but with
asserts enabled fail as do not acquire PersistentObjLock. Since they are
modifying the persistent tables should be protected by `PersistentObjLock`.

When `gp_persistent_reset_all()` is called, gp_relation_node index is truncated
and while writing meta-page checksum calculation was missed. Ideally, following
`gp_persistent_build_all()` fixes the same and correctly builds the index and
calculates the checksum file. Better to get proper message like "ERROR: Did not
find gp_relation_node entry for relation name....." on operations after
reset_all than page verification failed.

We also refactor the code out of `CreateCheckPoint()` and make it into
its own function `GetXLogCleanUpTo()`.

Unit test added to verify the new logic works for both master and
segments.

This is hidden under #ifdef USE_SEGWALREP.
Signed-off-by: NAsim R P <apraveen@pivotal.io>

This test checks PersistentObjLock is not acquired while holding io_in_progress
lock on a buffer. Tests will pass if PersistentObjLock is not acquired.
Signed-off-by: NAsim R P <apraveen@pivotal.io>

Signed-off-by: NTaylor Vesely <tvesely@pivotal.io>

Adds two new LW locks to protect filespace and tablespace hash tables
to disambiguate them from PersistentObjLock. Previously, PersistentObjLock was
overloaded to both protect these hash tables along with persistent heap tables.
If two backends try to flush the same dirty buffer, a deadlock could
potentially arise in which backend 1 holds PersistentObjLock and requests
io_in_progress lock of the buffer to be evicted.  Backend 2 holds
io_in_progress lock on the same buffer and attempts to obtain file path.
Because the file path is in hash tables protected by PersistentObjLock, backend
2 requests PersistentObjLock and blocks due to backend 1.
Signed-off-by: NAsim R P <apraveen@pivotal.io>

Comments were copied from the add node code path and did not match their
intent. This commit repairs the language of these comments to match their
behavior.
Signed-off-by: NTaylor Vesely <tvesely@pivotal.io>

To match upstream more closely, hopefully avoiding some silly merge
conflicts in the future.

extent -> extend. And reformat.

Remove premature optimization in create_indexscan_plan, where we stripped
RestrictInfo decorations while we iterated through the list. That was
premature, because while it works, order_qual_clauses() can make use of the
information in the RestrictInfo, if it's still present. This reverts that
piece of code to the way it is in the upstream. I'm not sure if this makes
any observable difference, but let's be tidy in any case.

Also, reorder the extract_actual_clauses and order_qual_clauses calls
in create_ctescan_plan, so that order_qual_clauses is called first. For
the same reason.

I repurposed the existing psql_gpdb_du test for this. It was very small,
I think we can put tests for other \d commands in the same file. In the
passing fix a typo in a comment there, and move the expected output from
output/ to expected/, because it doesn't need any string replacements.

The \dx+ command requires pg_describe_object() function, which we don't
have in GPDB yet. Until then, treat \dx+ the same as \dx.

Note that this code is written such that \dx+ can still be used, when
connected to a PostgreSQL 9.1 or later server. Not that we particularly care
about that case, but might as well.

Per github issue #2574.