# 25.1. Routine Vacuuming
25.1.3. Updating Planner Statistics
25.1.4. Updating the Visibility Map
25.1.5. Preventing Transaction ID Wraparound Failures
PostgreSQL databases require periodic maintenance known asvacuuming. For many installations, it is sufficient to let vacuuming be performed by theautovacuum daemon, which is described inSection 25.1.6. You might need to adjust the autovacuuming parameters described there to obtain best results for your situation. Some database administrators will want to supplement or replace the daemon's activities with manually-managedVACUUM
commands, which typically are executed according to a schedule by cron or Task Scheduler scripts. To set up manually-managed vacuuming properly, it is essential to understand the issues discussed in the next few subsections. Administrators who rely on autovacuuming may still wish to skim this material to help them understand and adjust autovacuuming.
# 25.1.1. Vacuuming Basics
PostgreSQL'sVACUUM
command has to process each table on a regular basis for several reasons:
To recover or reuse disk space occupied by updated or deleted rows.
To update data statistics used by the PostgreSQL query planner.
To update the visibility map, which speeds upindex-only scans.
To protect against loss of very old data due totransaction ID wraparoundormultixact ID wraparound.
这些原因中的每一个都要求执行
真空
不同频率和范围的操作,如以下小节所述。有两种变体
真空
: 标准真空
和真空已满
.真空已满
可以回收更多磁盘空间,但运行速度要慢得多。此外,标准形式真空
可以与生产数据库操作并行运行。(命令如选择
,插入
,更新
, 和删除
将继续正常运行,但您将无法使用以下命令修改表的定义更改表
当它被吸尘时。)真空已满
需要一个访问独家
锁定它正在处理的表,因此不能与表的其他用途并行完成。因此,一般来说,管理员应努力使用标准真空
并避免VACUUM FULL
.
VACUUM
creates a substantial amount of I/O traffic, which can cause poor performance for other active sessions. There are configuration parameters that can be adjusted to reduce the performance impact of background vacuuming — seeSection 20.4.4.
# 25.1.2. Recovering Disk Space
In PostgreSQL, anUPDATE
orDELETE
of a row does not immediately remove the old version of the row. This approach is necessary to gain the benefits of multiversion concurrency control (MVCC, seeChapter 13): the row version must not be deleted while it is still potentially visible to other transactions. But eventually, an outdated or deleted row version is no longer of interest to any transaction. The space it occupies must then be reclaimed for reuse by new rows, to avoid unbounded growth of disk space requirements. This is done by runningVACUUM
.
The standard form ofVACUUM
removes dead row versions in tables and indexes and marks the space available for future reuse. However, it will not return the space to the operating system, except in the special case where one or more pages at the end of a table become entirely free and an exclusive table lock can be easily obtained. In contrast,VACUUM FULL
actively compacts tables by writing a complete new version of the table file with no dead space. This minimizes the size of the table, but can take a long time. It also requires extra disk space for the new copy of the table, until the operation completes.
The usual goal of routine vacuuming is to do standardVACUUM
s often enough to avoid needingVACUUM FULL
. The autovacuum daemon attempts to work this way, and in fact will never issueVACUUM FULL
. In this approach, the idea is not to keep tables at their minimum size, but to maintain steady-state usage of disk space: each table occupies space equivalent to its minimum size plus however much space gets used up between vacuum runs. AlthoughVACUUM FULL
can be used to shrink a table back to its minimum size and return the disk space to the operating system, there is not much point in this if the table will just grow again in the future. Thus, moderately-frequent standard真空
运行是比不频繁运行更好的方法真空已满
运行以维护大量更新的表。
一些管理员更喜欢自己安排吸尘时间,例如在晚上负载较低的时候完成所有工作。根据固定时间表进行清理的困难在于,如果表在更新活动中出现意外的峰值,它可能会变得臃肿到以至于真空已满
确实需要回收空间。使用 autovacuum 守护进程可以缓解这个问题,因为守护进程会动态调度清理以响应更新活动。除非您的工作负载非常可预测,否则完全禁用守护程序是不明智的。一种可能的折衷方案是设置守护程序的参数,使其仅对异常繁重的更新活动做出反应,从而防止事情在计划时失控真空
当负载很典型时,预计 s 将完成大部分工作。
对于那些不使用 autovacuum 的人,一种典型的方法是安排一个数据库范围的真空
在低使用率期间每天一次,并根据需要更频繁地清理大量更新的表。(一些更新率极高的安装每隔几分钟就会清理一次最繁忙的表。)如果集群中有多个数据库,请不要忘记真空
每一个;该程序真空数据库可能会有所帮助。
# 提示
清楚的真空
当由于大量更新或删除活动而导致表包含大量死行版本时,可能无法令人满意。如果你有这样一张表,并且需要回收它占用的多余磁盘空间,则需要使用真空已满
, 或者簇
或表重写变体之一更改表
.这些命令重写表的全新副本并为其构建新索引。所有这些选项都需要一个访问独家
锁。请注意,它们还临时使用大约等于表大小的额外磁盘空间,因为表和索引的旧副本在新副本完成之前无法释放。
# 提示
如果您有一个表,其全部内容会定期删除,请考虑使用TRUNCATE
rather than usingDELETE
followed byVACUUM
.TRUNCATE
removes the entire content of the table immediately, without requiring a subsequentVACUUM
orVACUUM FULL
to reclaim the now-unused disk space. The disadvantage is that strict MVCC semantics are violated.
# 25.1.3. Updating Planner Statistics
The PostgreSQL query planner relies on statistical information about the contents of tables in order to generate good plans for queries. These statistics are gathered by theANALYZE
command, which can be invoked by itself or as an optional step inVACUUM
. It is important to have reasonably accurate statistics, otherwise poor choices of plans might degrade database performance.
The autovacuum daemon, if enabled, will automatically issueANALYZE
commands whenever the content of a table has changed sufficiently. However, administrators might prefer to rely on manually-scheduledANALYZE
operations, particularly if it is known that update activity on a table will not affect the statistics of “interesting” columns. The daemon schedulesANALYZE
strictly as a function of the number of rows inserted or updated; it has no knowledge of whether that will lead to meaningful statistical changes.
As with vacuuming for space recovery, frequent updates of statistics are more useful for heavily-updated tables than for seldom-updated ones. But even for a heavily-updated table, there might be no need for statistics updates if the statistical distribution of the data is not changing much. A simple rule of thumb is to think about how much the minimum and maximum values of the columns in the table change. For example, atimestamp
column that contains the time of row update will have a constantly-increasing maximum value as rows are added and updated; such a column will probably need more frequent statistics updates than, say, a column containing URLs for pages accessed on a website. The URL column might receive changes just as often, but the statistical distribution of its values probably changes relatively slowly.
It is possible to runANALYZE
on specific tables and even just specific columns of a table, so the flexibility exists to update some statistics more frequently than others if your application requires it. In practice, however, it is usually best to just analyze the entire database, because it is a fast operation.ANALYZE
uses a statistically random sampling of the rows of a table rather than reading every single row.
# Tip
Although per-column tweaking ofANALYZE
frequency might not be very productive, you might find it worthwhile to do per-column adjustment of the level of detail of the statistics collected byANALYZE
. Columns that are heavily used inWHERE
clauses and have highly irregular data distributions might require a finer-grain data histogram than other columns. SeeALTER TABLE SET STATISTICS
, or change the database-wide default using thedefault_statistics_targetconfiguration parameter.
Also, by default there is limited information available about the selectivity of functions. However, if you create a statistics object or an expression index that uses a function call, useful statistics will be gathered about the function, which can greatly improve query plans that use the expression index.
# Tip
The autovacuum daemon does not issueANALYZE
commands for foreign tables, since it has no means of determining how often that might be useful. If your queries require statistics on foreign tables for proper planning, it's a good idea to run manually-managedANALYZE
commands on those tables on a suitable schedule.
# 25.1.4. Updating the Visibility Map
Vacuum maintains avisibility mapfor each table to keep track of which pages contain only tuples that are known to be visible to all active transactions (and all future transactions, until the page is again modified). This has two purposes. First, vacuum itself can skip such pages on the next run, since there is nothing to clean up.
Second, it allows PostgreSQL to answer some queries using only the index, without reference to the underlying table. Since PostgreSQL indexes don't contain tuple visibility information, a normal index scan fetches the heap tuple for each matching index entry, to check whether it should be seen by the current transaction. Anindex-only scan, on the other hand, checks the visibility map first. If it's known that all tuples on the page are visible, the heap fetch can be skipped. This is most useful on large data sets where the visibility map can prevent disk accesses. The visibility map is vastly smaller than the heap, so it can easily be cached even when the heap is very large.
# 25.1.5. Preventing Transaction ID Wraparound Failures
PostgreSQL'sMVCCtransaction semantics depend on being able to compare transaction ID (XID) numbers: a row version with an insertion XID greater than the current transaction's XID is “in the future” and should not be visible to the current transaction. But since transaction IDs have limited size (32 bits) a cluster that runs for a long time (more than 4 billion transactions) would suffertransaction ID wraparound: the XID counter wraps around to zero, and all of a sudden transactions that were in the past appear to be in the future — which means their output become invisible. In short, catastrophic data loss. (Actually the data is still there, but that's cold comfort if you cannot get at it.) To avoid this, it is necessary to vacuum every table in every database at least once every two billion transactions.
The reason that periodic vacuuming solves the problem is thatVACUUM
will mark rows asfrozen, indicating that they were inserted by a transaction that committed sufficiently far in the past that the effects of the inserting transaction are certain to be visible to all current and future transactions. Normal XIDs are compared using modulo-232arithmetic. This means that for every normal XID, there are two billion XIDs that are “older” and two billion that are “newer”; another way to say it is that the normal XID space is circular with no endpoint. Therefore, once a row version has been created with a particular normal XID, the row version will appear to be “in the past” for the next two billion transactions, no matter which normal XID we are talking about. If the row version still exists after more than two billion transactions, it will suddenly appear to be in the future. To prevent this, PostgreSQL reserves a special XID,FrozenTransactionId
, which does not follow the normal XID comparison rules and is always considered older than every normal XID. Frozen row versions are treated as if the inserting XID wereFrozenTransactionId
, so that they will appear to be “in the past” to all normal transactions regardless of wraparound issues, and so such row versions will be valid until deleted, no matter how long that is.
# Note
In PostgreSQL versions before 9.4, freezing was implemented by actually replacing a row's insertion XID withFrozenTransactionId
, which was visible in the row'sxmin
system column. Newer versions just set a flag bit, preserving the row's originalxmin
for possible forensic use. However, rows withxmin
equal to冻结的交易 ID
(2) 仍可能在数据库 pg 中找到_从 9.4 之前的版本升级。
此外,系统目录可能包含带有xmin
等于BootstrapTransactionId
(1),表示它们是在initdb的第一阶段插入的。像冻结的交易 ID
, 这个特殊的 XID 被视为比每个普通 XID 都旧。
真空_冻结_分钟_年龄控制 XID 值在冻结带有该 XID 的行之前必须存在多长时间。如果将很快再次修改原本会被冻结的行,则增加此设置可能会避免不必要的工作,但减少此设置会增加必须再次清理表之前可以经过的事务数。
真空
使用能见度图以确定必须扫描表的哪些页面。通常,它会跳过没有任何死行版本的页面,即使这些页面可能仍有带有旧 XID 值的行版本。因此,正常真空
s 不会总是冻结表中的每个旧行版本。定期,真空
将执行一个侵略性真空,只跳过那些既不包含死行也不包含任何未冻结的 XID 或 MXID 值的页面。真空_冻结_桌子_年龄控制何时真空
这样做:如果自上次此类扫描以来通过的事务数大于vacuum_freeze_table_age
减vacuum_freeze_min_age
.环境vacuum_freeze_table_age
为 0 力真空
对所有扫描使用这种更积极的策略。
一个表可以取消真空的最长时间是 20 亿个事务减去vacuum_freeze_min_age
最后一次激进真空时的值。如果它的空置时间超过此时间,可能会导致数据丢失。为确保不会发生这种情况,将对任何可能包含未冻结行且 XID 早于配置参数指定的年龄的表调用 autovacuum自动真空_冻结_最大限度_年龄.(即使禁用了 autovacuum,也会发生这种情况。)
这意味着如果一个表没有被清理,autovacuum 将在其上大约每调用一次autovacuum_freeze_max_age
减vacuum_freeze_min_age
交易。对于出于空间回收目的而定期清理的表,这无关紧要。但是,对于静态表(包括接收插入但没有更新或删除的表),不需要清理空间来回收空间,因此尝试在非常大的静态表上最大化强制自动清理之间的间隔会很有用。显然,可以通过增加autovacuum_freeze_max_age
或减少vacuum_freeze_min_age
.
有效最大值为vacuum_freeze_table_age
是 0.95* autovacuum_freeze_max_age
;高于该值的设置将被限制为最大值。一个高于autovacuum_freeze_max_age
没有意义,因为无论如何都会在那个时候触发反环绕自动真空,并且 0.95 乘数为运行手册留下了一些喘息的空间真空
在此之前。根据经验,vacuum_freeze_table_age
应设置为略低于autovacuum_freeze_max_age
,留下足够的间隙,以便定期安排真空
或在该窗口中运行由正常删除和更新活动触发的自动清理。将其设置得太近可能会导致反环绕自动清理,即使表最近被清理以回收空间,而较低的值会导致更频繁的激进清理。
增加的唯一缺点autovacuum_freeze_max_age
(和vacuum_freeze_table_age
连同它)是pg_xact
和pg_commit_ts
数据库集群的子目录将占用更多空间,因为它必须存储提交状态和(如果track_commit_timestamp
已启用)所有事务的时间戳返回到autovacuum_freeze_max_age
地平线。提交状态每个事务使用两个位,所以如果autovacuum_freeze_max_age
设置为最大允许值 20 亿,pg_xact
预计将增长到大约半 GB 和pg_commit_ts
约20GB。如果这与您的总数据库大小相比微不足道,请设置autovacuum_freeze_max_age
建议将其设置为最大允许值。否则,请根据您愿意允许的情况进行设置pg_xact
和pg_commit_ts
贮存。(默认 2 亿笔交易,转换为大约 50MBpg_xact
存储空间和大约 2GBpg_commit_ts
贮存。)
减少的缺点之一vacuum_freeze_min_age
是它可能导致真空
做无用的工作:如果行之后很快被修改(导致它获取新的 XID),冻结行版本是浪费时间。所以设置应该足够大,以至于行不会被冻结,直到它们不太可能再发生变化。
要跟踪数据库中最旧的未冻结 XID 的年龄,真空
在系统表中存储 XID 统计信息pg_class
和pg_database
.特别是,再冷冻西德
表的列pg_class
行包含上次激进使用的冻结截止 XID真空
为那张桌子。XID 早于此截止 XID 的事务插入的所有行都保证已被冻结。同样,冰冻素
数据库的列pg_database
row 是该数据库中出现的未冻结 XID 的下限——它只是每个表的最小值再冷冻西德
数据库中的值。检查此信息的一种便捷方法是执行查询,例如:
SELECT c.oid::regclass as table_name,
greatest(age(c.relfrozenxid),age(t.relfrozenxid)) as age
FROM pg_class c
LEFT JOIN pg_class t ON c.reltoastrelid = t.oid
WHERE c.relkind IN ('r', 'm');
SELECT datname, age(datfrozenxid) FROM pg_database;
这年龄
列测量从截止 XID 到当前事务的 XID 的事务数。
真空
通常只扫描自上次清理后修改过的页面,但再冷冻西德
只能在扫描可能包含未冻结 XID 的表的每一页时才可以前进。这发生在再冷冻西德
超过vacuum_freeze_table_age
交易旧,何时真空
的冻结
使用选项,或者当所有尚未完全冻结的页面碰巧需要清理以删除死行版本时。什么时候真空
扫描表中尚未完全冻结的每个页面,它应该设置年龄(relfrozenxid)
值只是比vacuum_freeze_min_age
使用的设置(更多的是自真空
开始)。如果不再冷冻西德
-前进真空
在桌子上发出直到autovacuum_freeze_max_age
达到时,将很快对表强制执行自动清空。
如果由于某种原因 autovacuum 无法从表中清除旧的 XID,当数据库最旧的 XID 从回绕点达到四千万事务时,系统将开始发出类似这样的警告消息:
WARNING: database "mydb" must be vacuumed within 39985967 transactions
HINT: To avoid a database shutdown, execute a database-wide VACUUM in that database.
(一本手册真空
应该按照提示的建议解决问题;但请注意,真空
必须由超级用户执行,否则将无法处理系统目录,因此无法推进数据库的冰冻素
.) 如果这些警告被忽略,系统将关闭并拒绝启动任何新事务,一旦剩下的事务少于 300 万,直到结束:
ERROR: database is not accepting commands to avoid wraparound data loss in database "mydb"
HINT: Stop the postmaster and vacuum that database in single-user mode.
300万次交易的安全边际,让管理员在不丢失数据的情况下恢复,通过手动执行所需的真空
命令。但是,由于系统一旦进入安全关机模式就不会执行命令,唯一的办法就是停止服务器并以单用户模式启动服务器执行真空
.在单用户模式下不强制执行关闭模式。见postgres有关使用单用户模式的详细信息,请参阅参考页。
# 25.1.5.1. Multixacts and Wraparound
Multixact IDsare used to support row locking by multiple transactions. Since there is only limited space in a tuple header to store lock information, that information is encoded as a “multiple transaction ID”, or multixact ID for short, whenever there is more than one transaction concurrently locking a row. Information about which transaction IDs are included in any particular multixact ID is stored separately in thepg_multixact
subdirectory, and only the multixact ID appears in thexmax
field in the tuple header. Like transaction IDs, multixact IDs are implemented as a 32-bit counter and corresponding storage, all of which requires careful aging management, storage cleanup, and wraparound handling. There is a separate storage area which holds the list of members in each multixact, which also uses a 32-bit counter and which must also be managed.
WheneverVACUUM
scans any part of a table, it will replace any multixact ID it encounters which is older thanvacuum_multixact_freeze_min_ageby a different value, which can be the zero value, a single transaction ID, or a newer multixact ID. For each table,pg_class
.relminmxid
stores the oldest possible multixact ID still appearing in any tuple of that table. If this value is older thanvacuum_multixact_freeze_桌子_年龄,一个具有攻击性的真空被强迫。正如前一节所讨论的,激进的真空意味着只有那些已知已全部冻结的页面才会被跳过。mxid_age()
可用于pg_类
.relminmxid
寻找它的年龄。
侵略性的真空
不管是什么原因导致扫描,都可以使该表的值提前。最终,由于所有数据库中的所有表都会被扫描,并且它们最旧的multixact值会被提升,因此可以删除旧的multixact的磁盘存储。
作为一种安全装置,任何具有多层面X射线扫描年龄的手术台都会进行积极的真空扫描(参见第25.1.5.1节)大于自动真空_多X射线_冻结_最大值_年龄。此外,如果multixact成员占用的存储空间超过2GB,则会更频繁地对所有表进行主动真空扫描,从那些具有最早multixact年龄的表开始。即使autovacuum名义上被禁用,这两种攻击性扫描也会发生。
# 25.1.6.自动真空守护进程
PostgreSQL有一个可选但强烈推荐的功能,名为自动真空,其目的是自动执行真空
和分析
命令。启用后,autovacuum 会检查已插入、更新或删除大量元组的表。这些检查使用统计信息收集工具;因此,除非追踪_计数设定为真的
.在默认配置中,自动清理已启用,相关配置参数已适当设置。
“autovacuum daemon”实际上由多个进程组成。有一个持久的守护进程,称为自动真空发射器, 负责启动自动吸尘器所有数据库的进程。启动器将跨时间分配工作,尝试在每个数据库中启动一个工作人员自动真空_午觉时间秒。(因此,如果安装有*ñ
数据库,每启动一个新的workerautovacuum_naptime
/ñ
*秒。)最多自动真空_最大限度_工人允许工作进程同时运行。如果有超过autovacuum_max_workers
要处理的数据库,下一个数据库将在第一个工作人员完成后立即处理。每个工作进程将检查其数据库中的每个表并执行真空
和/或分析
如所须。日志_自动真空_分钟_期间可以设置为监视 autovacuum 工作人员的活动。
如果几个大表都可以在短时间内进行清理,那么所有 autovacuum 工作人员可能会在很长一段时间内忙于清理这些表。这将导致其他表和数据库在工作人员可用之前不会被清理。单个数据库中可能有多少工作人员没有限制,但工作人员确实会尽量避免重复其他工作人员已经完成的工作。请注意,正在运行的工人数量不计入最大限度_连接或者超级用户_预订的_连接限制。
表再冷冻西德
值大于自动真空_freeze_max_agetransactions old are always vacuumed (this also applies to those tables whose freeze max age has been modified via storage parameters; see below). Otherwise, if the number of tuples obsoleted since the lastVACUUM
exceeds the “vacuum threshold”, the table is vacuumed. The vacuum threshold is defined as:
vacuum threshold = vacuum base threshold + vacuum scale factor * number of tuples
where the vacuum base threshold isautovacuum_vacuum_threshold, the vacuum scale factor isautovacuum_vacuum_scale_factor, and the number of tuples ispg_class
.reltuples
.
The table is also vacuumed if the number of tuples inserted since the last vacuum has exceeded the defined insert threshold, which is defined as:
vacuum insert threshold = vacuum base insert threshold + vacuum insert scale factor * number of tuples
where the vacuum insert base threshold is自动真空_真空_插入_临界点, 真空插入比例因子是自动真空_真空_插入_规模_因素.这种真空可能允许将桌子的某些部分标记为全部可见并且还允许冻结元组,这可以减少后续真空所需的工作。对于接收表插入
操作但没有或几乎没有更新
/删除
操作,它可能是有益的,以降低表的自动真空_冻结_分钟_年龄因为这可能允许元组被早期的真空冻结。废弃元组的数量和插入的元组数量从统计收集器中获得;它是每个更新的半准确计数更新
,删除
和插入
手术。(这只是半准确的,因为某些信息可能会在重负载下丢失。)如果再冷冻西德
表的值大于vacuum_freeze_table_age
旧事务,执行积极的真空以冻结旧元组并推进再冷冻西德
;否则,仅扫描自上次清理后修改过的页面。
对于分析,使用类似的条件:阈值,定义为:
analyze threshold = analyze base threshold + analyze scale factor * number of tuples
与自上次以来插入、更新或删除的元组总数进行比较分析
.
autovacuum 无法访问临时表。因此,应通过会话 SQL 命令执行适当的清理和分析操作。
默认阈值和比例因子取自postgresql.conf
,但可以在每个表的基础上覆盖它们(以及许多其他自动真空控制参数);看存储参数了解更多信息。如果通过表的存储参数更改了设置,则在处理该表时使用该值;否则使用全局设置。看第 20.10 节有关全局设置的更多详细信息。
当多个工作人员正在运行时,autovacuum 成本延迟参数(请参阅第 20.4.4 节) 在所有正在运行的工作人员之间是“平衡的”,因此无论实际运行的工作人员数量如何,对系统的总 I/O 影响都是相同的。但是,任何处理每个表的表的工作人员autovacuum_vacuum_cost_delay
或者autovacuum_vacuum_cost_limit
平衡算法中不考虑已设置的存储参数。
Autovacuum 工作人员通常不会阻止其他命令。如果一个进程试图获取一个与共享更新独家
由 autovacuum 持有的锁,获取锁会中断 autovacuum。有关冲突的锁定模式,请参阅表 13.2.但是,如果 autovacuum 正在运行以防止事务 ID 回绕(即,autovacuum 查询名称在pg_stat_activity
视图以(以防止环绕)
),autovacuum 不会自动中断。
# 警告
定期运行获取与 a 冲突的锁的命令共享更新独家
锁定(例如,ANALYZE)可以有效地防止自动清空完成。