The fault `twophase_transaction_commit_prepared` would cause the segment to
PANIC while writing a `COMMIT PREPARED` record on the segment. The master would
then retry the `COMMIT PREPARED` while the postmaster was still resetting on the
segment causing the master itself to `PANIC`. This patch introduces a new fault
injector `finish_prepared_before_commit`  which will not cause a `PANIC`, but
just error out since the intent of the test is to ensure that the retry works
correctly.

The commit xlog record carries the information for objects which serves us for
crash-recovery till post-commit persistent object work is done, hence cannot
allow checkpoint in between. So, use inCommit flag to achieve the same. This
makes it consistent with FinishPreparedTransaction() as well.

Without the fix leaves out dangling persistent table entries and files on-disk
for following sequence:

- start transaction to drop a table
- `RecordTransactionCommit()`
- Checkpoint before `AtEOXact_smgr()`
- Crash/Recover

In Postgres, it's not supported to have temp table as part of PREPARE
TRANSACTION. Hence, the method `LockTagIsTemp()` to check whether a lock is on
temp table or not got removed from upstream.

It's a requirement for GPDB to allow access to temp table for MPP transactions,
and all transactions go through 2PC.

If we remove the `LockTagIsTemp()`, then the locks on the temp table will be
captured as part of `TwoPhaseRecordOnDisk`, and held during the prepare of 2PC.
That caused a concern during `xact_redo()` if segment crashed before commit. In
that case, the `xact_redo()` will lock and release the locks on the temp table,
which is already deleted by GPDB when handling the crash. That results in a
redundant operations of lock and release on a non-existent temp table object.

To prevent that from happening, we shouldn't copy the lock information on the
temp table to the `TwoPhaseRecordOnDisk`. Hence, we still need the
`LockTagIsTemp()` method.

This commit brings it back.
Signed-off-by: NXin Zhang <xzhang@pivotal.io>

We used to have a separate array of LocalDistributedXactData instances, and
a reference in PGPROC to its associated LocalDistributedXact. That's
unnecessarily complicated: we can store the LocalDistributedXact information
directly in the PGPROC entry, and get rid fo the auxiliary array and the
bookkeeping needed to manage that array.

This doesn't affect the backend-private cache of committed Xids that also
lives in cdblocaldistribxact.c.

Now that the PGPROC->localDistributedXactData fields are never accessed
by other backends, don't protect it with ProcArrayLock anymore. This makes
the code simpler, and potentially improves performance too (ProcArrayLock
can be very heavily contended on a busy system).

Callers to FaultInjector_InjectFaultIfSet() which don't pass neither
databasename nor tablename and that use DDLNotSpecified can instead
use the convenient macro SIMPLE_FAULT_INJECTOR() which cuts down on
the boilerplate in the code. This commit does not bring any changes
in functionality, merely readability.

Removed the locking and some more cleanups.
Avoid looping again in FinishPrepared Transaction. prepare_lsn to commit
the transaction can be found using gxact which we have locked, seems
pointless to loop around again to scan the list.

This is modified patch for GPDB based on postgres patch:
https://github.com/postgres/postgres/commit/bb38fb0d43c8d7ff54072bfd8bd63154e536b384#diff-3ed77c70e54e7f56eff48f6157aba91e
Original Patch commit message:
To lock a prepared transaction's shared memory entry, we used to mark it
with the XID of the backend. When the XID was no longer active according
to the proc array, the entry was implicitly considered as not locked
anymore. However, when preparing a transaction, the backend's proc array
entry was cleared before transfering the locks (and some other state) to
the prepared transaction's dummy PGPROC entry, so there was a window where
another backend could finish the transaction before it was in fact fully
prepared.

To fix, rewrite the locking mechanism of global transaction entries. Instead
of an XID, just have simple locked-or-not flag in each entry (we store the
locking backend's backend id rather than a simple boolean, but that's just
for debugging purposes). The backend is responsible for explicitly unlocking
the entry, and to make sure that that happens, install a callback to unlock
it on abort or process exit.

Introduced by a89b2684.

Just some random things I happened to spot while reading code.

be part of multixacts, so allocate a slot for each prepared transaction in
the "oldest member" array in multixact.c. On PREPARE TRANSACTION, transfer
the oldest member value from the current backends slot to the prepared xact
slot. Also save and recover the value from the 2pc state file.

The symptom of the bug was that after a transaction prepared, a shared lock
still held by the prepared transaction was sometimes ignored by other
transactions.

Fix back to 8.1, where both 2PC and multixact were introduced.

to go beoynd 10MB, as demonstrated by Gavin Sharry's example of dropping a
schema with ~25000 objects. The really bogus thing about the limit was that
it was enforced when a state file file was read in, not when it was written,
so you would end up with a prepared transaction that you can't commit or
abort, and the only recourse was to shut down the server and remove the file
by hand.

Raise the limit to MaxAllocSize, and enforce it also when a state file is
written. We could've removed the limit altogether, but reading in a file
larger than MaxAllocSize would fail anyway because we read it into a
palloc'd buffer.

Backpatch down to 8.1, where 2PC and this issue was introduced.

binary search instead of linear search when checking child-transaction XIDs.
Per example from Robert Treat, the speed of TransactionIdIsCurrentTransactionId
is significantly more important in 8.3 than it was in prior releases, so
this seems worth taking back-patching risk for.

having several of them.  Add two more flags: whether the process is
executing an ANALYZE, and whether a vacuum is for Xid wraparound (which
is obviously only set by autovacuum).

Sneakily move the worker's recently-acquired PostAuthDelay to a more useful
place.

ReadNewTransactionId from GetSnapshotData --- with a "latestCompletedXid"
variable that is updated during transaction commit or abort.  Since
latestCompletedXid is written only in places that had to lock ProcArrayLock
exclusively anyway, and is read only in places that had to lock ProcArrayLock
shared anyway, it adds no new locking requirements to the system despite being
cluster-wide.  Moreover, removing ReadNewTransactionId from snapshot
acquisition eliminates the need to take both XidGenLock and ProcArrayLock at
the same time.  Since XidGenLock is sometimes held across I/O this can be a
significant win.  Some preliminary benchmarking suggested that this patch has
no effect on average throughput but can significantly improve the worst-case
transaction times seen in pgbench.  Concept by Florian Pflug, implementation
by Tom Lane.

so that different prepared xacts can be told apart in the pg_locks
view.  Per suggestion from Florian.

rows will normally never obtain an XID at all.  We already did things this way
for subtransactions, but this patch extends the concept to top-level
transactions.  In applications where there are lots of short read-only
transactions, this should improve performance noticeably; not so much from
removal of the actual XID-assignments, as from reduction of overhead that's
driven by the rate of XID consumption.  We add a concept of a "virtual
transaction ID" so that active transactions can be uniquely identified even
if they don't have a regular XID.  This is a much lighter-weight concept:
uniqueness of VXIDs is only guaranteed over the short term, and no on-disk
record is made about them.

Florian Pflug, with some editorialization by Tom.

before reporting a transaction committed.  Data consistency is still
guaranteed (unlike setting fsync = off), but a crash may lose the effects
of the last few transactions.  Patch by Simon, some editorialization by Tom.

and aborted transactions have different effects; also teach it not to assume
that prepared transactions are always committed.

Along the way, simplify the pgstats API by tying counting directly to
Relations; I cannot detect any redeeming social value in having stats
pointers in HeapScanDesc and IndexScanDesc structures.  And fix a few
corner cases in which counts might be missed because the relation's
pgstat_info pointer hadn't been set.

from time_t to TimestampTz representation.  This provides full gettimeofday()
resolution of the timestamps, which might be useful when attempting to
do point-in-time recovery --- previously it was not possible to specify
the stop point with sub-second resolution.  But mostly this is to get
rid of TimestampTz-to-time_t conversion overhead during commit.  Per my
proposal of a day or two back.

are in their commit critical sections via flags in the ProcArray.  Checkpoint
can watch the ProcArray to determine when it's safe to proceed.  This is
a considerably better solution to the original problem of race conditions
between checkpoint and transaction commit: it speeds up commit, since there's
one less lock to fool with, and it prevents the problem of checkpoint being
delayed indefinitely when there's a constant flow of commits.  Heikki, with
some kibitzing from Tom.

it was executed in.  Someday it might be nice to allow cross-DB commits, but
work would be needed in NOTIFY and perhaps other places.  Per Heikki.

accessing it, like DROP DATABASE.  This allows the regression tests to pass
with autovacuum enabled, which open the gates for finally enabling autovacuum
by default.

back-stamped for this.

Euler Taveira de Oliveira

vacuums.  This allows a OLTP-like system with big tables to continue
regular vacuuming on small-but-frequently-updated tables while the
big tables are being vacuumed.

Original patch from Hannu Krossing, rewritten by Tom Lane and updated
by me.

Strip unused include files out unused include files, and add needed
includes to C files.

The next step is to remove unused include files in C files.

reduce contention for the former single LockMgrLock.  Per my recent
proposal.  I set it up for 16 partitions, but on a pgbench test this
gives only a marginal further improvement over 4 partitions --- we need
to test more scenarios to choose the number of partitions.

comment line where output as too long, and update typedefs for /lib
directory.  Also fix case where identifiers were used as variable names
in the backend, but as typedefs in ecpg (favor the backend for
indenting).

Backpatch to 8.1.X.

etc. match the docs, which talk about "transaction identifier" not
"gid" or "global transaction identifier".

Steve Woodcock