As part of creating a new relfile node, for example during reindex, the
persistent TID and the serial number in the relation descriptor used to be
updated with new values immediately.

This commit prevents such updates leaving the description unchanged.  Instead,
the relation descriptor will be udpated upon next cache invalidation.

The new ICG test is added to its own file and not in gp_create_table.sql
because reindex of pg_class index seems to affect planner, causing other tests
to fail with "insufficient memory reserved for statement" error.

We have observed that the persistent TID and serial number associated with a
relfilenode in a relation descriptor may be updated to a new value before
relfilenode is updated.  If an XLog record is written with such a relation
descriptor, then the XLog record fails to apply during recovery.

This commit adds a check to validate sanity of persistent information in a
relation descriptor by fetching persistent information using relfilenode from
gp_relation_node.

The validation is controlled by a new GUC, gp_validate_pt_info_relcache.

use asynchronous way to create gang and dispatch command by default.

Currently, when the stats are not existing for a relation, planner
relies on default values instead of running pg_relation_size to save
time. However, this causes planner to produce bad plans. Earlier, the
code had the feature to collect stats for planner if the table did not
had any stats, but it was removed.

This fix re-introduces the feature with the GUC to control whether to use
default values or run pg_relation_size to get actual stats if stats are
not present. By default, the GUC is turned off.

[#129570829]

This failure was introduced when we set the new GUC
(codegen_optimization_level) during init.
Fixes issue from this commit :
f160fa5a.

Used when compiling generated code. EXPLAIN codegen also
runs optimize with this optimize level, making it easier to see
the features the compiler optimizes.

Looking at old git history, this was added back in 2009. The related
ticket on adding it said:

    Add GUC that make every buffer page from the buffer pool flushed on
    eviction.

    Note that this will NOT necessarily flush all buffer pages when the
    postmaster is shutdown. I think this is acceptable for our purposes.

    (Our purpose is to make sure that overwrites of the buffer pages are not
    lost and instead are always written to disk so we can catch errors)

I'm not sure what errors that was meant to catch, or how, but I don't
think we have any regression tests or anything that uses it anymore.
Let's remove it, to make merging with upstream easier.

The previous commits have removed all usage of CaQL. It's no longer
needed.

Currenty, there is no easy way to evict entry from appendonly writer hash
table, most of current tests reduce the value of max_appendonly_tables which
needs database restart to take effect and then need to actually create that
many concurrent AO table inserts to evict the hash table entry. Using this
GUC can easily tests the eviction behavior.

This fixes the naming in c85f858e

A new feature of ORCA is to more efficiently handle array constraints.
It includes a new preprocessing stage, and a new way of internally
representing array constraints. This feature can be enabled by use of
this GUC.

A new feature of ORCA is to more efficiently handle array constraints.
It includes a new preprocessing stage, and a new way of internally
representing array constraints. This feature can be enabled by use of
this GUC.

This GUC is disabled by default.

Currently, when users run an `UNION ALL` query, such as
```sql
SELECT a FROM foo UNION ALL SELECT b FROM bar
```,
GPDB will parallelize the execution across all the segments, but the
table scans on `foo` and `bar` are not executed parallel. With this GUC
enabled, we add redistribution motion node under APPEND(UNION) operator,
which makes all the children of APPEND(UNION) operator execute in
parallel in every segment.

This closes #836
Signed-off-by: NMarc Spehlmann <cramja@gmail.com>

reorder header file

Currently gp_connections_per_thread=0 means using main thread to
do dispatch instead of creating new threads. It's not working if
segments report error before sending back any data.
For example:
create table foo(id int);
insert into foo values(0);
set gp_connections_per_thread=0;
select 100/id from foo;

We disable it for now and will reenable it when we have the new
implementation of non-threaded dispatcher.

This seems like a completely unnecessary change. Reduce the diff vs.
upstream.

This GUC will be used to control the MEMO size as well as optimization
time for large IN list or large array comparison expressions.

Only the Array with less number of elements than the GUC will be
expanded and participate in constraint derivation.

Trade-off of using this GUC is loss of potential benefits from the
constraint derivation (e.g. conflict detection, partition elimination)
with shorter optimization time and less memory utilization.

The checks for invalid TIDs are very cheap, a few CPU instructions. It's
better to catch bugs involving invalid TID early, so let's always check
for them.

The LOGs in storeAttrDefault() that were also tied to this GUC seemed
oddly specific. They were probably added long time ago to hunt for some
particular bug, and don't seem generally useful, so I just removed them.

Previously, previous free TID validation was done under the GUC
persistent_integrity_checks. This commit extracts the previous free TID
validation into another GUC validate_previous_free_tid and is enabled by
default. If the validation detects a corruption in the free TID list, we
will now switch to a new free TID list and leave the corrupted one detached
for cleanup during persistent table rebuild or during crash recovery.

Authors: Jimmy Yih and Abhijit Subramanya

Validation 1: look up new tuple's heap tid in unique index before insert.

The tuple being inserted is already inserted in heap.  Before its entry is
added to a unique index, we want to see if the index already has an entry with
this heap tid.  This should catch duplicate entries created in index but not in
the heap relation.  The validation is enabled by GUC "gp_indexcheck_insert".

Validation 2: index should point to all visible tuples after vacuum.

For each entry in index after it was vacuumed, fetch the heap tuple
and validate that it is visible.  For specific tables, validate that
the key is the same.  The validation is controlled by GUC
"gp_indexcheck_vacuum".

Closes #673.

This commit  generates code for code path: ExecVariableList > slot_getattr > _slot_getsomeattrs > slot_deform_tuple. This code path is executed during scan in simple select queries that do not have a where clause (e.g., select bar from foo;).

For each attribute A in the target list, regular implementation of ExecvariableList retrieves the slot that A comes from and calls slot_getattr. slot_get_attr() will eventually call slot_deform_tuple (through _slot_getsomeattrs), which fetches all yet unread attributes of the slot until the given attribute.

This commit generates the code for the case that all the attributes in target list use the same slot (created during a scan i.e, ecxt_scantuple). Moreover, instead of looping over the target list one at a time, it uses slot_getattr only once, with the largest attribute index from the target list.

If during code generation time, the completion is not possible (e.g., attributes use different slots), then function returns false and codegen manager will be responsible to manage the clean up.

This implementation does not support:
* Null attributes
* Variable length attributes
* Fixed length attributes passed by reference (e.g., uuid)
If at execution time, we see any of the above types of attributes, we fall back to the regular function.

Moreover, this commit:
* renames existing "codegen" guc, which is used for initiating llvm libraries, to "init_codegen" (Note: we set this guc in gpconfig),
* adds guc "codegen", which enables code generation and compilation at query execution time,
* enhances the existing code generation utilities by adding a function that creates all IR instructions for falling back to regular functions, and
* removes the existing code that generates the code of a naive slot_deform_tuple, which simply falls back to regular slot_deform_tuple.
Signed-off-by: NNikos Armenatzoglou <nikos.armenatzoglou@gmail.com>

tuple.
This closes #649

Squashed commit of the following:

commit 96f415bc1744533e96a74bcc85016ff5df65d06e
Author: Karthikeyan Jambu Rajaraman <karthi.jrk@gmail.com>
Date:   Thu Apr 21 15:08:49 2016 -0700

    Don't call prepare for execution if no generator is registered

commit 5318e9214a3bbc237c0d79017cdef02538c4e711
Author: Karthikeyan Jambu Rajaraman <karthi.jrk@gmail.com>
Date:   Thu Apr 21 11:35:22 2016 -0700

    Adding if codegen protection in starting active manager and while
    deleting manager

commit 12b0045d51a5b056b32a6cc9ef370e6d9af315fd
Author: Karthikeyan Jambu Rajaraman <karthi.jrk@gmail.com>
Date:   Fri Apr 15 14:33:55 2016 -0700

    Updating Readme for Codegen GPDB Function

commit d17119c5d8ce9a706b3321d5e7ad41f419fddbdc
Author: Karthikeyan Jambu Rajaraman <karthi.jrk@gmail.com>
Date:   Wed Apr 13 13:25:14 2016 -0700

    Add documentation for how to codegen in README.md

commit d711be4bb044592e7f97bf31ceb5f32337ad6d03
Author: Karthikeyan Jambu Rajaraman <karthi.jrk@gmail.com>
Date:   Wed Apr 13 11:19:03 2016 -0700

    Fixing doxygen namespace addtogroup

commit e34131bcc9bb3d36c5e8995e1a0efe11fbd55417
Author: Karthikeyan Jambu Rajaraman <karthi.jrk@gmail.com>
Date:   Wed Apr 13 11:17:32 2016 -0700

    Changing CodeGen to Codegen

commit bedfcca80b5f1ff796783ca7c2035d920ae86c4e
Author: Karthikeyan Jambu Rajaraman <karthi.jrk@gmail.com>
Date:   Wed Apr 13 10:51:39 2016 -0700

    Fix cpplint error

commit 32f89feedba655e1242529c78a54faf01ddf32bf
Author: Karthikeyan Jambu Rajaraman <karthi.jrk@gmail.com>
Date:   Wed Apr 13 10:48:49 2016 -0700

    Change regular version of generated to regular version of target in comment

commit d1da2824c08e34eb7139b08cd9d2aff19bf008e7
Author: Karthikeyan Jambu Rajaraman <karthi.jrk@gmail.com>
Date:   Wed Apr 13 10:42:09 2016 -0700

    Made GetFunctionPointer and CreateFunction to take typedef of function type as only public api

commit 9f48f77bd9fc50c7d29a4e8f7700452ef88dc79f
Author: Karthikeyan Jambu Rajaraman <karthi.jrk@gmail.com>
Date:   Tue Apr 12 17:18:24 2016 -0700

    Rename doCodegeneration to GenerateCodeInternal

commit 9884d07f4c6ab08d9ccca8d07b80974081cda1fa
Author: Karthikeyan Jambu Rajaraman <karthi.jrk@gmail.com>
Date:   Tue Apr 12 17:15:55 2016 -0700

    fix cpplint error

commit c21b3bee3e289675fc13017688269b40e26efa11
Author: Karthikeyan Jambu Rajaraman <karthi.jrk@gmail.com>
Date:   Tue Apr 12 17:08:16 2016 -0700

    Update comments

commit f016bb582af65e84def1b55917017008e292d45b
Author: Karthikeyan Jambu Rajaraman <karthi.jrk@gmail.com>
Date:   Tue Apr 12 17:01:19 2016 -0700

    Adding assert in static setToRegular function

commit 93a791a4ec415313933dd5b7c1ae17dbdd6193a6
Author: Karthikeyan Jambu Rajaraman <karthi.jrk@gmail.com>
Date:   Tue Apr 12 16:49:15 2016 -0700

    Removing decltype in Slotdeformtype codegen since we know the functype

commit ed95193d0991cb24fd68f6d9a473de630f297b0c
Author: Karthikeyan Jambu Rajaraman <karthi.jrk@gmail.com>
Date:   Tue Apr 12 16:45:49 2016 -0700

    Fixed compiler / unit test failure
    Changed InitCodeGen to InitCodegen

commit 3c95c9efdd4336e100a86748b0e87d79f07bcc11
Author: Karthikeyan Jambu Rajaraman <karthi.jrk@gmail.com>
Date:   Tue Apr 12 16:28:24 2016 -0700

    Changing CodeGen back to Codegen while cherry-picking

commit 4e24ef22bcf54edc00c2a0d1f9f9703364cdf5ba
Author: Karthikeyan Jambu Rajaraman <krajaraman@pivotal.io>
Date:   Tue Apr 12 11:08:42 2016 -0700

    Changing auto to actual type

commit 1d92fc190dda3ec866edc83d56d9eca9fe90a63a
Author: Karthikeyan Jambu Rajaraman <krajaraman@pivotal.io>
Date:   Tue Apr 12 10:44:49 2016 -0700

    fix indent in codegen_utils

commit ea6fa787a2332f94b8d20d24825ba1f0a240d02e
Author: Karthikeyan Jambu Rajaraman <krajaraman@pivotal.io>
Date:   Mon Apr 11 17:25:53 2016 -0700

    Adding codegen guc to wrapper function so related function not executed when codegen guc is disabled

commit f1cd7d80d8d81f3565f3ab12cb94a3c4cc58c44a
Author: Karthikeyan Jambu Rajaraman <krajaraman@pivotal.io>
Date:   Mon Apr 11 17:13:24 2016 -0700

    Changing bool to unsigned int in codegen_wrapper.h

commit 7ef462540a312d077b332c732c125c55dedb6b14
Author: Karthikeyan Jambu Rajaraman <krajaraman@pivotal.io>
Date:   Mon Apr 11 16:59:08 2016 -0700

    Fix based on lint complain

commit ddc55cec0d579a03cb9e2303efa6d74f5183979f
Author: Karthikeyan Jambu Rajaraman <krajaraman@pivotal.io>
Date:   Mon Apr 11 16:19:48 2016 -0700

    Removing commented code from unittest

commit 28b8886ed2bfdc854aa38f871a41536c1e6cef29
Author: Karthikeyan Jambu Rajaraman <krajaraman@pivotal.io>
Date:   Mon Apr 11 16:17:45 2016 -0700

    Chaging auto in for to actual type

commit ad8d78e07e9b789f4d8ae3c6ccc3cea1a05089c0
Author: Karthikeyan Jambu Rajaraman <krajaraman@pivotal.io>
Date:   Mon Apr 11 16:13:21 2016 -0700

    * Handle return value of SetToGenerated from PrepareForExecution
    * Change size_t to unsigned integer

commit d46ee8be414b3ac2a9ea7ef899f29cef3df145c7
Author: Karthikeyan Jambu Rajaraman <krajaraman@pivotal.io>
Date:   Mon Apr 11 15:30:40 2016 -0700

    Introducing template function to do enroll for any type of generator

commit 83428605f906e28d0fc994dcc311e0cc3e6d9e12
Author: Karthikeyan Jambu Rajaraman <krajaraman@pivotal.io>
Date:   Mon Apr 11 14:41:11 2016 -0700

    Changing manager to take module name as argument

commit b88aa40f89860e7c15503365ea304d2392f71cda
Author: Karthikeyan Jambu Rajaraman <krajaraman@pivotal.io>
Date:   Mon Apr 11 14:27:31 2016 -0700

    adding more comments to SlotDeformTupleCodeGenInfo and unique counter

commit df1b704548fcbc1b43524064eb2f024995eea865
Author: Karthikeyan Jambu Rajaraman <krajaraman@pivotal.io>
Date:   Mon Apr 11 14:08:34 2016 -0700

    Change test target to have ext .t and adding gitignore for that extension

commit 9ee3521ca687c4ea7f13d15893f47146ecdb5277
Author: Karthikeyan Jambu Rajaraman <krajaraman@pivotal.io>
Date:   Mon Apr 11 13:59:57 2016 -0700

    Add target_include_directories for test library header file

commit effb7dc165b4c7094731dc39e0872741b28b2e6a
Author: Karthikeyan Jambu Rajaraman <krajaraman@pivotal.io>
Date:   Mon Apr 11 13:36:50 2016 -0700

    Move all possible ifdef from gpdb to codegen_wrapper based on pull request comments

commit 989b96a91c1e843577b5f24623d1e4a6b810394c
Author: Karthikeyan Jambu Rajaraman <krajaraman@pivotal.io>
Date:   Fri Apr 8 10:43:17 2016 -0700

    fixing comments for GetOrigFuncName in codegen_interface

commit 665de065c34ea13abcec8c5f3a3336558dae501f
Author: Karthikeyan Jambu Rajaraman <krajaraman@pivotal.io>
Date:   Fri Apr 8 10:27:31 2016 -0700

    Disabling codegen guc by default

commit ec34513e86dea95bd36ee528724273c1c19e7062
Author: Karthikeyan Jambu Rajaraman <krajaraman@pivotal.io>
Date:   Thu Apr 7 16:28:43 2016 -0700

    Fixed comments space/tab so it aligned in different editor

commit 35a60fe2fd053a5babd1820302afad95c5c2635f
Author: Karthikeyan Jambu Rajaraman <krajaraman@pivotal.io>
Date:   Thu Apr 7 16:12:08 2016 -0700

    Fixing coding style based on cpplint

commit 9477dcf6c0a7ffcca77b9f2c62c7340d13f474aa
Author: Karthikeyan Jambu Rajaraman <krajaraman@pivotal.io>
Date:   Thu Apr 7 13:37:01 2016 -0700

    Adding USE_CODEGEN ifdef

commit 142ffc1416341a3ac1d0fe37a83577168f2ec166
Author: Shreedhar Hardikar <shardikar@pivotal.io>
Date:   Tue Apr 5 22:21:19 2016 +0000

    Make codegen work with GPDB unit test framework

commit 9dca9507dd933a5517bee3ca34615c09c23c1728
Author: Shreedhar Hardikar <shardikar@pivotal.io>
Date:   Mon Apr 4 22:10:21 2016 +0000

    Add gpcodegen_mock.c

commit 41d515ff0342d3df3b0853ad91731f6d1d75843c
Author: Shreedhar Hardikar <shardikar@pivotal.io>
Date:   Mon Apr 4 21:46:34 2016 +0000

    Remove unneccessary header file includes in codegen

commit 14c5e14b9834197693b1ef56efa96386cfe049c5
Author: Foyzur Rahman <frahman@gmail.com>
Date:   Tue Apr 12 14:49:51 2016 -0700

    Integrate codegen framework and make simple external call to slot deform tuple
Signed-off-by: NKarthikeyan Jambu Rajaraman <karthi.jrk@gmail.com>
Signed-off-by: NFoyzur Rahman <frahman@gmail.com>

`set optimizer_join_arity_for_associativity_commutativity=7` will hint
at the optimizer to stop exploring join associativity and join
commutativity transformations when an n-ary join operator has more than
7 children during optimization, pruning quite a bit of the search space.

It didn't do anything useful AFAICS.

In some cases, the filerep processes get stuck waiting for the spin locks on an
in memory log and the postmaster PANICs if they can't acquire a lock after
timeout period. In order to avoid locking related issues we want to avoid
acquiring the locks in the first place. This patch introduces a new guc which
will cause filerep logs to be logged to server logs and will avoid logging to
the in memory log.

The gp_default_storage_options GUC had two scenarios in which the value between
master and segments could be conflicting which would cause tables to possibly
have different storage parameters between master and segments.

First Scenario:
The below gpconfig example would give newly created user tables a default storage
parameter of AO on segments while master sets the table as heap. Fixed by not
allowing the user to make them different values.
gpconfig -c gp_default_storage_options -v "'appendonly=true'" -m "'appendonly=false'"

Second Scenario:
During a psql session not in transaction, a session-level gp_default_storage_options
GUC value can be set which will also be dispatched to segment processes. Those
segment processes when idle for a period of time will be stopped and the next SQL
(e.g. CREATE TABLE) will spawn new segment processes which will not have the
session-level GUC value set. Fixed by setting the value at cdbgang creation.

Codes of ic_tcp and ic_udp are not maintained any more, ic_udpifc is
the only interconnect type for now. We keep the gp_interconnect_type
guc for backward compatibility.

Remove unnecessary #includes, add #includes that are actually needed by
some headers.

Renaming optimizer_release_mdcache to a more intuitive name: optimizer_metadata_caching. Before this commit, we were caching metadata in optimizer side by setting optimizer_release_mdcache = OFF. Now, we cache metadata in optimizer side by setting optimizer_metadata_caching=ON.