When determine waiting relation in global dead lock detector,
we should also check whether the waiter's lockmode conflict with
holder's holdmask.
Co-authored-by: NNing Yu <nyu@pivotal.io>
Reviewed-by: NZhenghua Lyu <zlv@pivotal.io>

Currently GDD sets DistributedTransactionContext to
DTX_CONTEXT_QD_DISTRIBUTED_CAPABLE and as a result allocates distributed
transaction id. It creates entry in ProcGlobal->allTmGxact with state
DTX_STATE_ACTIVE_NOT_DISTRIBUTED. The effect of this is that any query
taking a snapshot will see this transaction as in progress. Since GDD
transaction is short lived it is not an issue in general, but in CI it
causes flaky behavior for some of the vacuum tests. The flaky behavior
shows up as unvacuumed tables where the vacuum snapshot was taken while
GDD transaction was running thereby forcing vacuum to lower its oldest
XMIN. Current behavior of GDD consuming a distributed transaction id
(every 2 minutes by default) is also wasteful behavior.

Currently GDD also sends a snapshot to QE, but this isn't required and
is wasteful as well.

In this change for GDD we keep DistributedTransactionContext as
DTX_CONTEXT_LOCAL_ONLY and avoid dispatching snapshots to QEs.
Co-authored-by: NAshwin Agrawal <aagrawal@pivotal.io>

We could declare the function as ON SEGMENT, use SPI, and let the
planner/executor dispatch it, but there's no particular reason we would
need to do it that  way. I think dispatching the function call like this
is totally OK.

This reverts commit 4750e1b6.

This is the final batch of commits from PostgreSQL 9.2 development,
up to the point where the REL9_2_STABLE branch was created, and 9.3
development started on the PostgreSQL master branch.

Notable upstream changes:

* Index-only scan was included in the batch of upstream commits. It
  allows queries to retrieve data only from indexes, avoiding heap access.

* Group commit was added to work effectively under heavy load. Previously,
  batching of commits became ineffective as the write workload increased,
  because of internal lock contention.

* A new fast-path lock mechanism was added to reduce the overhead of
  taking and releasing certain types of locks which are taken and released
  very frequently but rarely conflict.

* The new "parameterized path" mechanism was added. It allows inner index
  scans to use values from relations that are more than one join level up
  from the scan. This can greatly improve performance in situations where
  semantic restrictions (such as outer joins) limit the allowed join orderings.

* SP-GiST (Space-Partitioned GiST) index access method was added to support
  unbalanced partitioned search structures. For suitable problems, SP-GiST can
  be faster than GiST in both index build time and search time.

* Checkpoints now are performed by a dedicated background process. Formerly
  the background writer did both dirty-page writing and checkpointing. Separating
  this into two processes allows each goal to be accomplished more predictably.

* Custom plan was supported for specific parameter values even when using
  prepared statements.

* API for FDW was improved to provide multiple access "paths" for their tables,
  allowing more flexibility in join planning.

* Security_barrier option was added for views to prevents optimizations that
  might allow view-protected data to be exposed to users.

* Range data type was added to store a lower and upper bound belonging to its
  base data type.

* CTAS (CREATE TABLE AS/SELECT INTO) is now treated as utility statement. The
  SELECT query is planned during the execution of the utility. To conform to
  this change, GPDB executes the utility statement only on QD and dispatches
  the plan of the SELECT query to QEs.
Co-authored-by: NAdam Lee <ali@pivotal.io>
Co-authored-by: NAlexandra Wang <lewang@pivotal.io>
Co-authored-by: NAshwin Agrawal <aagrawal@pivotal.io>
Co-authored-by: NAsim R P <apraveen@pivotal.io>
Co-authored-by: NDaniel Gustafsson <dgustafsson@pivotal.io>
Co-authored-by: NGang Xiong <gxiong@pivotal.io>
Co-authored-by: NHaozhou Wang <hawang@pivotal.io>
Co-authored-by: NHeikki Linnakangas <hlinnakangas@pivotal.io>
Co-authored-by: NJesse Zhang <sbjesse@gmail.com>
Co-authored-by: NJinbao Chen <jinchen@pivotal.io>
Co-authored-by: NJoao Pereira <jdealmeidapereira@pivotal.io>
Co-authored-by: NMelanie Plageman <mplageman@pivotal.io>
Co-authored-by: NPaul Guo <paulguo@gmail.com>
Co-authored-by: NRichard Guo <guofenglinux@gmail.com>
Co-authored-by: NShujie Zhang <shzhang@pivotal.io>
Co-authored-by: NTaylor Vesely <tvesely@pivotal.io>
Co-authored-by: NZhenghua Lyu <zlv@pivotal.io>

The persistent field of LOCK struct may cause confusion. Its meaning is the lock can only
be released after the transaction ends.  This commit renames this field as holdTillEndXact
and changes some related function's name. Some comments are also added.

Code had these two variables (GUCs), serving same purpose. GpIdentity.segindex
is set to content-id, based on command line argument at start-up and inherited
by all processes from postmaster. Whereas Gp_segment, was session level guc only
set for backends, by dispatching the same from QD. So. essentially Gp_segment
was not available and had incorrect value in auxiliary processes.

Hence replaced all usages with GpIdentity.segindex. As a side effect log files
now have correct value reported for segment number (content-id) in each and
every line of file, irrespective of which process generated the log message.

Discussion:
https://groups.google.com/a/greenplum.org/forum/#!msg/gpdb-dev/Yr8-LZIiNfA/ob4KLgmkAQAJ

To prevent distributed deadlock, in Greenplum DB an exclusive table lock is
held for UPDATE and DELETE commands, so concurrent updates the same table are
actually disabled.

We add a backend process to do global deadlock detect so that we do not lock
the whole table while doing UPDATE/DELETE and this will help improve the
concurrency of Greenplum DB.

The core idea of the algorithm is to divide lock into two types:

- Persistent: the lock can only be released after the transaction is over(abort/commit)
- Otherwise cases

This PR’s implementation adds a persistent flag in the LOCK, and the set rule is:

- Xid lock is always persistent
- Tuple lock is never persistent
- Relation is persistent if it has been closed with NoLock parameter, otherwise
  it is not persistent Other types of locks are not persistent

More details please refer the code and README.

There are several known issues to pay attention to:

- This PR’s implementation only cares about the locks can be shown
  in the view pg_locks.
- This PR’s implementation does not support AO table. We keep upgrading
  the locks for AO table.
- This PR’s implementation does not take networking wait into account.
  Thus we cannot detect the deadlock of GitHub issue #2837.
- SELECT FOR UPDATE still lock the whole table.
Co-authored-by: NZhenghua Lyu <zlv@pivotal.io>
Co-authored-by: NNing Yu <nyu@pivotal.io>