mirrors/postgres
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Add a new WAL summarizer process.

Browse Source 
When active, this process writes WAL summary files to
$PGDATA/pg_wal/summaries. Each summary file contains information for a
certain range of LSNs on a certain TLI. For each relation, it stores a
"limit block" which is 0 if a relation is created or destroyed within
a certain range of WAL records, or otherwise the shortest length to
which the relation was truncated during that range of WAL records, or
otherwise InvalidBlockNumber. In addition, it stores a list of blocks
which have been modified during that range of WAL records, but
excluding blocks which were removed by truncation after they were
modified and never subsequently modified again.

In other words, it tells us which blocks need to copied in case of an
incremental backup covering that range of WAL records. But this
doesn't yet add the capability to actually perform an incremental
backup; the next patch will do that.

A new parameter summarize_wal enables or disables this new background
process.  The background process also automatically deletes summary
files that are older than wal_summarize_keep_time, if that parameter
has a non-zero value and the summarizer is configured to run.

Patch by me, with some design help from Dilip Kumar and Andres Freund.
Reviewed by Matthias van de Meent, Dilip Kumar, Jakub Wartak, Peter
Eisentraut, and Álvaro Herrera.

Discussion: http://postgr.es/m/CA+TgmoYOYZfMCyOXFyC-P+-mdrZqm5pP2N7S-r0z3_402h9rsA@mail.gmail.com
This commit is contained in:
Robert Haas 2023-12-20 08:41:09 -05:00
parent 00498b7185
commit 174c480508
30 changed files with 3743 additions and 11 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

61
doc/src/sgml/config.sgml
View File

@ -4150,6 +4150,67 @@ restore_command = 'copy "C:\\server\\archivedir\\%f" "%p"' # Windows
</variablelist>
</sect2>
<sect2 id="runtime-config-wal-summarization">
<title>WAL Summarization</title>
<!--
<para>
These settings control WAL summarization, a feature which must be
enabled in order to perform an
<link linkend="backup-incremental-backup">incremental backup</link>.
</para>
-->
<variablelist>
<varlistentry id="guc-summarize-wal" xreflabel="summarize_wal">
<term><varname>summarize_wal</varname> (<type>boolean</type>)
<indexterm>
<primary><varname>summarize_wal</varname> configuration parameter</primary>
</indexterm>
</term>
<listitem>
<para>
Enables the WAL summarizer process. Note that WAL summarization can
be enabled either on a primary or on a standby. WAL summarization
cannot be enabled when <varname>wal_level</varname> is set to
<literal>minimal</literal>. This parameter can only be set in the
<filename>postgresql.conf</filename> file or on the server command line.
The default is <literal>off</literal>.
</para>
</listitem>
</varlistentry>
<varlistentry id="guc-wal-summary-keep-time" xreflabel="wal_summary_keep_time">
<term><varname>wal_summary_keep_time</varname> (<type>boolean</type>)
<indexterm>
<primary><varname>wal_summary_keep_time</varname> configuration parameter</primary>
</indexterm>
</term>
<listitem>
<para>
Configures the amount of time after which the WAL summarizer
automatically removes old WAL summaries. The file timestamp is used to
determine which files are old enough to remove. Typically, you should set
this comfortably higher than the time that could pass between a backup
and a later incremental backup that depends on it. WAL summaries must
be available for the entire range of WAL records between the preceding
backup and the new one being taken; if not, the incremental backup will
fail. If this parameter is set to zero, WAL summaries will not be
automatically deleted, but it is safe to manually remove files that you
know will not be required for future incremental backups.
This parameter can only be set in the
<filename>postgresql.conf</filename> file or on the server command line.
The default is 10 days. If <literal>summarize_wal = off</literal>,
existing WAL summaries will not be removed regardless of the value of
this parameter, because the WAL summarizer will not run.
</para>
</listitem>
</varlistentry>
</variablelist>
</sect2>
</sect1>
<sect1 id="runtime-config-replication">

101
src/backend/access/transam/xlog.c
View File

@ -77,6 +77,7 @@
#include "port/pg_iovec.h"
#include "postmaster/bgwriter.h"
#include "postmaster/startup.h"
#include "postmaster/walsummarizer.h"
#include "postmaster/walwriter.h"
#include "replication/logical.h"
#include "replication/origin.h"
@ -3592,6 +3593,43 @@ XLogGetLastRemovedSegno(void)
return lastRemovedSegNo;
}
/*
* Return the oldest WAL segment on the given TLI that still exists in
* XLOGDIR, or 0 if none.
*/
XLogSegNo
XLogGetOldestSegno(TimeLineID tli)
{
DIR *xldir;
struct dirent *xlde;
XLogSegNo oldest_segno = 0;
xldir = AllocateDir(XLOGDIR);
while ((xlde = ReadDir(xldir, XLOGDIR)) != NULL)
{
TimeLineID file_tli;
XLogSegNo file_segno;
/* Ignore files that are not XLOG segments. */
if (!IsXLogFileName(xlde->d_name))
continue;
/* Parse filename to get TLI and segno. */
XLogFromFileName(xlde->d_name, &file_tli, &file_segno,
wal_segment_size);
/* Ignore anything that's not from the TLI of interest. */
if (tli != file_tli)
continue;
/* If it's the oldest so far, update oldest_segno. */
if (oldest_segno == 0 || file_segno < oldest_segno)
oldest_segno = file_segno;
}
FreeDir(xldir);
return oldest_segno;
}
/*
* Update the last removed segno pointer in shared memory, to reflect that the
@ -3872,8 +3910,8 @@ RemoveXlogFile(const struct dirent *segment_de,
}
/*
* Verify whether pg_wal and pg_wal/archive_status exist.
* If the latter does not exist, recreate it.
* Verify whether pg_wal, pg_wal/archive_status, and pg_wal/summaries exist.
* If the latter do not exist, recreate them.
*
* It is not the goal of this function to verify the contents of these
* directories, but to help in cases where someone has performed a cluster
@ -3916,6 +3954,26 @@ ValidateXLOGDirectoryStructure(void)
(errmsg("could not create missing directory \"%s\": %m",
path)));
}
/* Check for summaries */
snprintf(path, MAXPGPATH, XLOGDIR "/summaries");
if (stat(path, &stat_buf) == 0)
{
/* Check for weird cases where it exists but isn't a directory */
if (!S_ISDIR(stat_buf.st_mode))
ereport(FATAL,
(errmsg("required WAL directory \"%s\" does not exist",
path)));
}
else
{
ereport(LOG,
(errmsg("creating missing WAL directory \"%s\"", path)));
if (MakePGDirectory(path) < 0)
ereport(FATAL,
(errmsg("could not create missing directory \"%s\": %m",
path)));
}
}
/*
@ -5243,9 +5301,9 @@ StartupXLOG(void)
#endif
/*
* Verify that pg_wal and pg_wal/archive_status exist. In cases where
* someone has performed a copy for PITR, these directories may have been
* excluded and need to be re-created.
* Verify that pg_wal, pg_wal/archive_status, and pg_wal/summaries exist.
* In cases where someone has performed a copy for PITR, these directories
* may have been excluded and need to be re-created.
*/
ValidateXLOGDirectoryStructure();
@ -6962,6 +7020,25 @@ CreateCheckPoint(int flags)
*/
END_CRIT_SECTION();
/*
* WAL summaries end when the next XLOG_CHECKPOINT_REDO or
* XLOG_CHECKPOINT_SHUTDOWN record is reached. This is the first point
* where (a) we're not inside of a critical section and (b) we can be
* certain that the relevant record has been flushed to disk, which must
* happen before it can be summarized.
*
* If this is a shutdown checkpoint, then this happens reasonably
* promptly: we've only just inserted and flushed the
* XLOG_CHECKPOINT_SHUTDOWN record. If this is not a shutdown checkpoint,
* then this might not be very prompt at all: the XLOG_CHECKPOINT_REDO
* record was written before we began flushing data to disk, and that
* could be many minutes ago at this point. However, we don't XLogFlush()
* after inserting that record, so we're not guaranteed that it's on disk
* until after the above call that flushes the XLOG_CHECKPOINT_ONLINE
* record.
*/
SetWalSummarizerLatch();
/*
* Let smgr do post-checkpoint cleanup (eg, deleting old files).
*/
@ -7636,6 +7713,20 @@ KeepLogSeg(XLogRecPtr recptr, XLogSegNo *logSegNo)
}
}
/*
* If WAL summarization is in use, don't remove WAL that has yet to be
* summarized.
*/
keep = GetOldestUnsummarizedLSN(NULL, NULL, false);
if (keep != InvalidXLogRecPtr)
{
XLogSegNo unsummarized_segno;
XLByteToSeg(keep, unsummarized_segno, wal_segment_size);
if (unsummarized_segno < segno)
segno = unsummarized_segno;
}
/* but, keep at least wal_keep_size if that's set */
if (wal_keep_size_mb > 0)
{

4
src/backend/backup/Makefile
View File

@ -25,6 +25,8 @@ OBJS = \
basebackup_server.o \
basebackup_sink.o \
basebackup_target.o \
basebackup_throttle.o
basebackup_throttle.o \
walsummary.o \
walsummaryfuncs.o
include $(top_srcdir)/src/backend/common.mk

2
src/backend/backup/meson.build
View File

@ -12,4 +12,6 @@ backend_sources += files(
'basebackup_target.c',
'basebackup_throttle.c',
'basebackup_zstd.c',
'walsummary.c',
'walsummaryfuncs.c',
)

356
src/backend/backup/walsummary.c Normal file
View File

@ -0,0 +1,356 @@
/*-------------------------------------------------------------------------
*
* walsummary.c
* Functions for accessing and managing WAL summary data.
*
* Portions Copyright (c) 2010-2022, PostgreSQL Global Development Group
*
* src/backend/backup/walsummary.c
*
*-------------------------------------------------------------------------
*/
#include "postgres.h"
#include <sys/stat.h>
#include <unistd.h>
#include "access/xlog_internal.h"
#include "backup/walsummary.h"
#include "utils/wait_event.h"
static bool IsWalSummaryFilename(char *filename);
static int ListComparatorForWalSummaryFiles(const ListCell *a,
const ListCell *b);
/*
* Get a list of WAL summaries.
*
* If tli != 0, only WAL summaries with the indicated TLI will be included.
*
* If start_lsn != InvalidXLogRecPtr, only summaries that end after the
* indicated LSN will be included.
*
* If end_lsn != InvalidXLogRecPtr, only summaries that start before the
* indicated LSN will be included.
*
* The intent is that you can call GetWalSummaries(tli, start_lsn, end_lsn)
* to get all WAL summaries on the indicated timeline that overlap the
* specified LSN range.
*/
List *
GetWalSummaries(TimeLineID tli, XLogRecPtr start_lsn, XLogRecPtr end_lsn)
{
DIR *sdir;
struct dirent *dent;
List *result = NIL;
sdir = AllocateDir(XLOGDIR "/summaries");
while ((dent = ReadDir(sdir, XLOGDIR "/summaries")) != NULL)
{
WalSummaryFile *ws;
uint32 tmp[5];
TimeLineID file_tli;
XLogRecPtr file_start_lsn;
XLogRecPtr file_end_lsn;
/* Decode filename, or skip if it's not in the expected format. */
if (!IsWalSummaryFilename(dent->d_name))
continue;
sscanf(dent->d_name, "%08X%08X%08X%08X%08X",
&tmp[0], &tmp[1], &tmp[2], &tmp[3], &tmp[4]);
file_tli = tmp[0];
file_start_lsn = ((uint64) tmp[1]) << 32 | tmp[2];
file_end_lsn = ((uint64) tmp[3]) << 32 | tmp[4];
/* Skip if it doesn't match the filter criteria. */
if (tli != 0 && tli != file_tli)
continue;
if (!XLogRecPtrIsInvalid(start_lsn) && start_lsn >= file_end_lsn)
continue;
if (!XLogRecPtrIsInvalid(end_lsn) && end_lsn <= file_start_lsn)
continue;
/* Add it to the list. */
ws = palloc(sizeof(WalSummaryFile));
ws->tli = file_tli;
ws->start_lsn = file_start_lsn;
ws->end_lsn = file_end_lsn;
result = lappend(result, ws);
}
FreeDir(sdir);
return result;
}
/*
* Build a new list of WAL summaries based on an existing list, but filtering
* out summaries that don't match the search parameters.
*
* If tli != 0, only WAL summaries with the indicated TLI will be included.
*
* If start_lsn != InvalidXLogRecPtr, only summaries that end after the
* indicated LSN will be included.
*
* If end_lsn != InvalidXLogRecPtr, only summaries that start before the
* indicated LSN will be included.
*/
List *
FilterWalSummaries(List *wslist, TimeLineID tli,
XLogRecPtr start_lsn, XLogRecPtr end_lsn)
{
List *result = NIL;
ListCell *lc;
/* Loop over input. */
foreach(lc, wslist)
{
WalSummaryFile *ws = lfirst(lc);
/* Skip if it doesn't match the filter criteria. */
if (tli != 0 && tli != ws->tli)
continue;
if (!XLogRecPtrIsInvalid(start_lsn) && start_lsn > ws->end_lsn)
continue;
if (!XLogRecPtrIsInvalid(end_lsn) && end_lsn < ws->start_lsn)
continue;
/* Add it to the result list. */
result = lappend(result, ws);
}
return result;
}
/*
* Check whether the supplied list of WalSummaryFile objects covers the
* whole range of LSNs from start_lsn to end_lsn. This function ignores
* timelines, so the caller should probably filter using the appropriate
* timeline before calling this.
*
* If the whole range of LSNs is covered, returns true, otherwise false.
* If false is returned, *missing_lsn is set either to InvalidXLogRecPtr
* if there are no WAL summary files in the input list, or to the first LSN
* in the range that is not covered by a WAL summary file in the input list.
*/
bool
WalSummariesAreComplete(List *wslist, XLogRecPtr start_lsn,
XLogRecPtr end_lsn, XLogRecPtr *missing_lsn)
{
XLogRecPtr current_lsn = start_lsn;
ListCell *lc;
/* Special case for empty list. */
if (wslist == NIL)
{
*missing_lsn = InvalidXLogRecPtr;
return false;
}
/* Make a private copy of the list and sort it by start LSN. */
wslist = list_copy(wslist);
list_sort(wslist, ListComparatorForWalSummaryFiles);
/*
* Consider summary files in order of increasing start_lsn, advancing the
* known-summarized range from start_lsn toward end_lsn.
*
* Normally, the summary files should cover non-overlapping WAL ranges,
* but this algorithm is intended to be correct even in case of overlap.
*/
foreach(lc, wslist)
{
WalSummaryFile *ws = lfirst(lc);
if (ws->start_lsn > current_lsn)
{
/* We found a gap. */
break;
}
if (ws->end_lsn > current_lsn)
{
/*
* Next summary extends beyond end of previous summary, so extend
* the end of the range known to be summarized.
*/
current_lsn = ws->end_lsn;
/*
* If the range we know to be summarized has reached the required
* end LSN, we have proved completeness.
*/
if (current_lsn >= end_lsn)
return true;
}
}
/*
* We either ran out of summary files without reaching the end LSN, or we
* hit a gap in the sequence that resulted in us bailing out of the loop
* above.
*/
*missing_lsn = current_lsn;
return false;
}
/*
* Open a WAL summary file.
*
* This will throw an error in case of trouble. As an exception, if
* missing_ok = true and the trouble is specifically that the file does
* not exist, it will not throw an error and will return a value less than 0.
*/
File
OpenWalSummaryFile(WalSummaryFile *ws, bool missing_ok)
{
char path[MAXPGPATH];
File file;
snprintf(path, MAXPGPATH,
XLOGDIR "/summaries/%08X%08X%08X%08X%08X.summary",
ws->tli,
LSN_FORMAT_ARGS(ws->start_lsn),
LSN_FORMAT_ARGS(ws->end_lsn));
file = PathNameOpenFile(path, O_RDONLY);
if (file < 0 && (errno != EEXIST || !missing_ok))
ereport(ERROR,
(errcode_for_file_access(),
errmsg("could not open file \"%s\": %m", path)));
return file;
}
/*
* Remove a WAL summary file if the last modification time precedes the
* cutoff time.
*/
void
RemoveWalSummaryIfOlderThan(WalSummaryFile *ws, time_t cutoff_time)
{
char path[MAXPGPATH];
struct stat statbuf;
snprintf(path, MAXPGPATH,
XLOGDIR "/summaries/%08X%08X%08X%08X%08X.summary",
ws->tli,
LSN_FORMAT_ARGS(ws->start_lsn),
LSN_FORMAT_ARGS(ws->end_lsn));
if (lstat(path, &statbuf) != 0)
{
if (errno == ENOENT)
return;
ereport(ERROR,
(errcode_for_file_access(),
errmsg("could not stat file \"%s\": %m", path)));
}
if (statbuf.st_mtime >= cutoff_time)
return;
if (unlink(path) != 0)
ereport(ERROR,
(errcode_for_file_access(),
errmsg("could not stat file \"%s\": %m", path)));
ereport(DEBUG2,
(errmsg_internal("removing file \"%s\"", path)));
}
/*
* Test whether a filename looks like a WAL summary file.
*/
static bool
IsWalSummaryFilename(char *filename)
{
return strspn(filename, "0123456789ABCDEF") == 40 &&
strcmp(filename + 40, ".summary") == 0;
}
/*
* Data read callback for use with CreateBlockRefTableReader.
*/
int
ReadWalSummary(void *wal_summary_io, void *data, int length)
{
WalSummaryIO *io = wal_summary_io;
int nbytes;
nbytes = FileRead(io->file, data, length, io->filepos,
WAIT_EVENT_WAL_SUMMARY_READ);
if (nbytes < 0)
ereport(ERROR,
(errcode_for_file_access(),
errmsg("could not read file \"%s\": %m",
FilePathName(io->file))));
io->filepos += nbytes;
return nbytes;
}
/*
* Data write callback for use with WriteBlockRefTable.
*/
int
WriteWalSummary(void *wal_summary_io, void *data, int length)
{
WalSummaryIO *io = wal_summary_io;
int nbytes;
nbytes = FileWrite(io->file, data, length, io->filepos,
WAIT_EVENT_WAL_SUMMARY_WRITE);
if (nbytes < 0)
ereport(ERROR,
(errcode_for_file_access(),
errmsg("could not write file \"%s\": %m",
FilePathName(io->file))));
if (nbytes != length)
ereport(ERROR,
(errcode_for_file_access(),
errmsg("could not write file \"%s\": wrote only %d of %d bytes at offset %u",
FilePathName(io->file), nbytes,
length, (unsigned) io->filepos),
errhint("Check free disk space.")));
io->filepos += nbytes;
return nbytes;
}
/*
* Error-reporting callback for use with CreateBlockRefTableReader.
*/
void
ReportWalSummaryError(void *callback_arg, char *fmt,...)
{
StringInfoData buf;
va_list ap;
int needed;
initStringInfo(&buf);
for (;;)
{
va_start(ap, fmt);
needed = appendStringInfoVA(&buf, fmt, ap);
va_end(ap);
if (needed == 0)
break;
enlargeStringInfo(&buf, needed);
}
ereport(ERROR,
errcode(ERRCODE_DATA_CORRUPTED),
errmsg_internal("%s", buf.data));
}
/*
* Comparator to sort a List of WalSummaryFile objects by start_lsn.
*/
static int
ListComparatorForWalSummaryFiles(const ListCell *a, const ListCell *b)
{
WalSummaryFile *ws1 = lfirst(a);
WalSummaryFile *ws2 = lfirst(b);
if (ws1->start_lsn < ws2->start_lsn)
return -1;
if (ws1->start_lsn > ws2->start_lsn)
return 1;
return 0;
}

169
src/backend/backup/walsummaryfuncs.c Normal file
View File

@ -0,0 +1,169 @@
/*-------------------------------------------------------------------------
*
* walsummaryfuncs.c
* SQL-callable functions for accessing WAL summary data.
*
* Portions Copyright (c) 2010-2022, PostgreSQL Global Development Group
*
* src/backend/backup/walsummaryfuncs.c
*
*-------------------------------------------------------------------------
*/
#include "postgres.h"
#include "backup/walsummary.h"
#include "common/blkreftable.h"
#include "funcapi.h"
#include "miscadmin.h"
#include "utils/fmgrprotos.h"
#include "utils/pg_lsn.h"
#define NUM_WS_ATTS 3
#define NUM_SUMMARY_ATTS 6
#define MAX_BLOCKS_PER_CALL 256
/*
* List the WAL summary files available in pg_wal/summaries.
*/
Datum
pg_available_wal_summaries(PG_FUNCTION_ARGS)
{
ReturnSetInfo *rsi;
List *wslist;
ListCell *lc;
Datum values[NUM_WS_ATTS];
bool nulls[NUM_WS_ATTS];
InitMaterializedSRF(fcinfo, 0);
rsi = (ReturnSetInfo *) fcinfo->resultinfo;
memset(nulls, 0, sizeof(nulls));
wslist = GetWalSummaries(0, InvalidXLogRecPtr, InvalidXLogRecPtr);
foreach(lc, wslist)
{
WalSummaryFile *ws = (WalSummaryFile *) lfirst(lc);
HeapTuple tuple;
CHECK_FOR_INTERRUPTS();
values[0] = Int64GetDatum((int64) ws->tli);
values[1] = LSNGetDatum(ws->start_lsn);
values[2] = LSNGetDatum(ws->end_lsn);
tuple = heap_form_tuple(rsi->setDesc, values, nulls);
tuplestore_puttuple(rsi->setResult, tuple);
}
return (Datum) 0;
}
/*
* List the contents of a WAL summary file identified by TLI, start LSN,
* and end LSN.
*/
Datum
pg_wal_summary_contents(PG_FUNCTION_ARGS)
{
ReturnSetInfo *rsi;
Datum values[NUM_SUMMARY_ATTS];
bool nulls[NUM_SUMMARY_ATTS];
WalSummaryFile ws;
WalSummaryIO io;
BlockRefTableReader *reader;
int64 raw_tli;
RelFileLocator rlocator;
ForkNumber forknum;
BlockNumber limit_block;
InitMaterializedSRF(fcinfo, 0);
rsi = (ReturnSetInfo *) fcinfo->resultinfo;
memset(nulls, 0, sizeof(nulls));
/*
* Since the timeline could at least in theory be more than 2^31, and
* since we don't have unsigned types at the SQL level, it is passed as a
* 64-bit integer. Test whether it's out of range.
*/
raw_tli = PG_GETARG_INT64(0);
if (raw_tli < 1 || raw_tli > PG_INT32_MAX)
ereport(ERROR,
errcode(ERRCODE_INVALID_PARAMETER_VALUE),
errmsg("invalid timeline %lld", (long long) raw_tli));
/* Prepare to read the specified WAL summry file. */
ws.tli = (TimeLineID) raw_tli;
ws.start_lsn = PG_GETARG_LSN(1);
ws.end_lsn = PG_GETARG_LSN(2);
io.filepos = 0;
io.file = OpenWalSummaryFile(&ws, false);
reader = CreateBlockRefTableReader(ReadWalSummary, &io,
FilePathName(io.file),
ReportWalSummaryError, NULL);
/* Loop over relation forks. */
while (BlockRefTableReaderNextRelation(reader, &rlocator, &forknum,
&limit_block))
{
BlockNumber blocks[MAX_BLOCKS_PER_CALL];
HeapTuple tuple;
CHECK_FOR_INTERRUPTS();
values[0] = ObjectIdGetDatum(rlocator.relNumber);
values[1] = ObjectIdGetDatum(rlocator.spcOid);
values[2] = ObjectIdGetDatum(rlocator.dbOid);
values[3] = Int16GetDatum((int16) forknum);
/* Loop over blocks within the current relation fork. */
while (1)
{
unsigned nblocks;
unsigned i;
CHECK_FOR_INTERRUPTS();
nblocks = BlockRefTableReaderGetBlocks(reader, blocks,
MAX_BLOCKS_PER_CALL);
if (nblocks == 0)
break;
/*
* For each block that we specifically know to have been modified,
* emit a row with that block number and limit_block = false.
*/
values[5] = BoolGetDatum(false);
for (i = 0; i < nblocks; ++i)
{
values[4] = Int64GetDatum((int64) blocks[i]);
tuple = heap_form_tuple(rsi->setDesc, values, nulls);
tuplestore_puttuple(rsi->setResult, tuple);
}
/*
* If the limit block is not InvalidBlockNumber, emit an exta row
* with that block number and limit_block = true.
*
* There is no point in doing this when the limit_block is
* InvalidBlockNumber, because no block with that number or any
* higher number can ever exist.
*/
if (BlockNumberIsValid(limit_block))
{
values[4] = Int64GetDatum((int64) limit_block);
values[5] = BoolGetDatum(true);
tuple = heap_form_tuple(rsi->setDesc, values, nulls);
tuplestore_puttuple(rsi->setResult, tuple);
}
}
}
/* Cleanup */
DestroyBlockRefTableReader(reader);
FileClose(io.file);
return (Datum) 0;
}

1
src/backend/postmaster/Makefile
View File

@ -24,6 +24,7 @@ OBJS = \
postmaster.o \
startup.o \
syslogger.o \
walsummarizer.o \
walwriter.o
include $(top_srcdir)/src/backend/common.mk

8
src/backend/postmaster/auxprocess.c
View File

@ -21,6 +21,7 @@
#include "postmaster/auxprocess.h"
#include "postmaster/bgwriter.h"
#include "postmaster/startup.h"
#include "postmaster/walsummarizer.h"
#include "postmaster/walwriter.h"
#include "replication/walreceiver.h"
#include "storage/bufmgr.h"
@ -80,6 +81,9 @@ AuxiliaryProcessMain(AuxProcType auxtype)
case WalReceiverProcess:
MyBackendType = B_WAL_RECEIVER;
break;
case WalSummarizerProcess:
MyBackendType = B_WAL_SUMMARIZER;
break;
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
MyBackendType = B_INVALID;
@ -158,6 +162,10 @@ AuxiliaryProcessMain(AuxProcType auxtype)
WalReceiverMain();
proc_exit(1);
case WalSummarizerProcess:
WalSummarizerMain();
proc_exit(1);
default:
elog(PANIC, "unrecognized process type: %d", (int) MyAuxProcType);
proc_exit(1);

1
src/backend/postmaster/meson.build
View File

@ -12,5 +12,6 @@ backend_sources += files(
'postmaster.c',
'startup.c',
'syslogger.c',
'walsummarizer.c',
'walwriter.c',
)

56
src/backend/postmaster/postmaster.c
View File

@ -113,6 +113,7 @@
#include "postmaster/pgarch.h"
#include "postmaster/postmaster.h"
#include "postmaster/syslogger.h"
#include "postmaster/walsummarizer.h"
#include "replication/logicallauncher.h"
#include "replication/walsender.h"
#include "storage/fd.h"
@ -250,6 +251,7 @@ static pid_t StartupPID = 0,
CheckpointerPID = 0,
WalWriterPID = 0,
WalReceiverPID = 0,
WalSummarizerPID = 0,
AutoVacPID = 0,
PgArchPID = 0,
SysLoggerPID = 0;
@ -441,6 +443,7 @@ static bool CreateOptsFile(int argc, char *argv[], char *fullprogname);
static pid_t StartChildProcess(AuxProcType type);
static void StartAutovacuumWorker(void);
static void MaybeStartWalReceiver(void);
static void MaybeStartWalSummarizer(void);
static void InitPostmasterDeathWatchHandle(void);
/*
@ -564,6 +567,7 @@ static void ShmemBackendArrayRemove(Backend *bn);
#define StartCheckpointer() StartChildProcess(CheckpointerProcess)
#define StartWalWriter() StartChildProcess(WalWriterProcess)
#define StartWalReceiver() StartChildProcess(WalReceiverProcess)
#define StartWalSummarizer() StartChildProcess(WalSummarizerProcess)
/* Macros to check exit status of a child process */
#define EXIT_STATUS_0(st) ((st) == 0)
@ -933,6 +937,9 @@ PostmasterMain(int argc, char *argv[])
if (max_wal_senders > 0 && wal_level == WAL_LEVEL_MINIMAL)
ereport(ERROR,
(errmsg("WAL streaming (max_wal_senders > 0) requires wal_level \"replica\" or \"logical\"")));
if (summarize_wal && wal_level == WAL_LEVEL_MINIMAL)
ereport(ERROR,
(errmsg("WAL cannot be summarized when wal_level is \"minimal\"")));
/*
* Other one-time internal sanity checks can go here, if they are fast.
@ -1835,6 +1842,9 @@ ServerLoop(void)
if (WalReceiverRequested)
MaybeStartWalReceiver();
/* If we need to start a WAL summarizer, try to do that now */
MaybeStartWalSummarizer();
/* Get other worker processes running, if needed */
if (StartWorkerNeeded || HaveCrashedWorker)
maybe_start_bgworkers();
@ -2659,6 +2669,8 @@ process_pm_reload_request(void)
signal_child(WalWriterPID, SIGHUP);
if (WalReceiverPID != 0)
signal_child(WalReceiverPID, SIGHUP);
if (WalSummarizerPID != 0)
signal_child(WalSummarizerPID, SIGHUP);
if (AutoVacPID != 0)
signal_child(AutoVacPID, SIGHUP);
if (PgArchPID != 0)
@ -3012,6 +3024,7 @@ process_pm_child_exit(void)
BgWriterPID = StartBackgroundWriter();
if (WalWriterPID == 0)
WalWriterPID = StartWalWriter();
MaybeStartWalSummarizer();
/*
* Likewise, start other special children as needed. In a restart
@ -3130,6 +3143,20 @@ process_pm_child_exit(void)
continue;
}
/*
* Was it the wal summarizer? Normal exit can be ignored; we'll start
* a new one at the next iteration of the postmaster's main loop, if
* necessary. Any other exit condition is treated as a crash.
*/
if (pid == WalSummarizerPID)
{
WalSummarizerPID = 0;
if (!EXIT_STATUS_0(exitstatus))
HandleChildCrash(pid, exitstatus,
_("WAL summarizer process"));
continue;
}
/*
* Was it the autovacuum launcher? Normal exit can be ignored; we'll
* start a new one at the next iteration of the postmaster's main
@ -3525,6 +3552,12 @@ HandleChildCrash(int pid, int exitstatus, const char *procname)
else if (WalReceiverPID != 0 && take_action)
sigquit_child(WalReceiverPID);
/* Take care of the walsummarizer too */
if (pid == WalSummarizerPID)
WalSummarizerPID = 0;
else if (WalSummarizerPID != 0 && take_action)
sigquit_child(WalSummarizerPID);
/* Take care of the autovacuum launcher too */
if (pid == AutoVacPID)
AutoVacPID = 0;
@ -3675,6 +3708,8 @@ PostmasterStateMachine(void)
signal_child(StartupPID, SIGTERM);
if (WalReceiverPID != 0)
signal_child(WalReceiverPID, SIGTERM);
if (WalSummarizerPID != 0)
signal_child(WalSummarizerPID, SIGTERM);
/* checkpointer, archiver, stats, and syslogger may continue for now */
/* Now transition to PM_WAIT_BACKENDS state to wait for them to die */
@ -3701,6 +3736,7 @@ PostmasterStateMachine(void)
if (CountChildren(BACKEND_TYPE_ALL - BACKEND_TYPE_WALSND) == 0 &&
StartupPID == 0 &&
WalReceiverPID == 0 &&
WalSummarizerPID == 0 &&
BgWriterPID == 0 &&
(CheckpointerPID == 0 ||
(!FatalError && Shutdown < ImmediateShutdown)) &&
@ -3798,6 +3834,7 @@ PostmasterStateMachine(void)
/* These other guys should be dead already */
Assert(StartupPID == 0);
Assert(WalReceiverPID == 0);
Assert(WalSummarizerPID == 0);
Assert(BgWriterPID == 0);
Assert(CheckpointerPID == 0);
Assert(WalWriterPID == 0);
@ -4019,6 +4056,8 @@ TerminateChildren(int signal)
signal_child(WalWriterPID, signal);
if (WalReceiverPID != 0)
signal_child(WalReceiverPID, signal);
if (WalSummarizerPID != 0)
signal_child(WalSummarizerPID, signal);
if (AutoVacPID != 0)
signal_child(AutoVacPID, signal);
if (PgArchPID != 0)
@ -5326,6 +5365,10 @@ StartChildProcess(AuxProcType type)
ereport(LOG,
(errmsg("could not fork WAL receiver process: %m")));
break;
case WalSummarizerProcess:
ereport(LOG,
(errmsg("could not fork WAL summarizer process: %m")));
break;
default:
ereport(LOG,
(errmsg("could not fork process: %m")));
@ -5462,6 +5505,19 @@ MaybeStartWalReceiver(void)
}
}
/*
* MaybeStartWalSummarizer
* Start the WAL summarizer process, if not running and our state allows.
*/
static void
MaybeStartWalSummarizer(void)
{
if (summarize_wal && WalSummarizerPID == 0 &&
(pmState == PM_RUN || pmState == PM_HOT_STANDBY) &&
Shutdown <= SmartShutdown)
WalSummarizerPID = StartWalSummarizer();
}
/*
* Create the opts file

1398
src/backend/postmaster/walsummarizer.c Normal file
View File

File diff suppressed because it is too large Load Diff

1
src/backend/storage/lmgr/lwlocknames.txt
View File

@ -54,3 +54,4 @@ XactTruncationLock 44
WrapLimitsVacuumLock 46
NotifyQueueTailLock 47
WaitEventExtensionLock 48
WALSummarizerLock 49

4
src/backend/utils/activity/pgstat_io.c
View File

@ -306,7 +306,8 @@ pgstat_io_snapshot_cb(void)
* - Syslogger because it is not connected to shared memory
* - Archiver because most relevant archiving IO is delegated to a
* specialized command or module
* - WAL Receiver and WAL Writer IO is not tracked in pg_stat_io for now
* - WAL Receiver, WAL Writer, and WAL Summarizer IO are not tracked in
* pg_stat_io for now
*
* Function returns true if BackendType participates in the cumulative stats
* subsystem for IO and false if it does not.
@ -328,6 +329,7 @@ pgstat_tracks_io_bktype(BackendType bktype)
case B_LOGGER:
case B_WAL_RECEIVER:
case B_WAL_WRITER:
case B_WAL_SUMMARIZER:
return false;
case B_AUTOVAC_LAUNCHER:

5
src/backend/utils/activity/wait_event_names.txt
View File

@ -56,6 +56,7 @@ RECOVERY_WAL_STREAM "Waiting in main loop of startup process for WAL to arrive,
SYSLOGGER_MAIN "Waiting in main loop of syslogger process."
WAL_RECEIVER_MAIN "Waiting in main loop of WAL receiver process."
WAL_SENDER_MAIN "Waiting in main loop of WAL sender process."
WAL_SUMMARIZER_WAL "Waiting in WAL summarizer for more WAL to be generated."
WAL_WRITER_MAIN "Waiting in main loop of WAL writer process."
@ -142,6 +143,7 @@ SAFE_SNAPSHOT "Waiting to obtain a valid snapshot for a <literal>READ ONLY DEFER
SYNC_REP "Waiting for confirmation from a remote server during synchronous replication."
WAL_RECEIVER_EXIT "Waiting for the WAL receiver to exit."
WAL_RECEIVER_WAIT_START "Waiting for startup process to send initial data for streaming replication."
WAL_SUMMARY_READY "Waiting for a new WAL summary to be generated."
XACT_GROUP_UPDATE "Waiting for the group leader to update transaction status at end of a parallel operation."
@ -162,6 +164,7 @@ REGISTER_SYNC_REQUEST "Waiting while sending synchronization requests to the che
SPIN_DELAY "Waiting while acquiring a contended spinlock."
VACUUM_DELAY "Waiting in a cost-based vacuum delay point."
VACUUM_TRUNCATE "Waiting to acquire an exclusive lock to truncate off any empty pages at the end of a table vacuumed."
WAL_SUMMARIZER_ERROR "Waiting after a WAL summarizer error."
#
@ -243,6 +246,8 @@ WAL_COPY_WRITE "Waiting for a write when creating a new WAL segment by copying a
WAL_INIT_SYNC "Waiting for a newly initialized WAL file to reach durable storage."
WAL_INIT_WRITE "Waiting for a write while initializing a new WAL file."
WAL_READ "Waiting for a read from a WAL file."
WAL_SUMMARY_READ "Waiting for a read from a WAL summary file."
WAL_SUMMARY_WRITE "Waiting for a write to a WAL summary file."
WAL_SYNC "Waiting for a WAL file to reach durable storage."
WAL_SYNC_METHOD_ASSIGN "Waiting for data to reach durable storage while assigning a new WAL sync method."
WAL_WRITE "Waiting for a write to a WAL file."

3
src/backend/utils/init/miscinit.c
View File

@ -305,6 +305,9 @@ GetBackendTypeDesc(BackendType backendType)
case B_WAL_SENDER:
backendDesc = "walsender";
break;
case B_WAL_SUMMARIZER:
backendDesc = "walsummarizer";
break;
case B_WAL_WRITER:
backendDesc = "walwriter";
break;

26
src/backend/utils/misc/guc_tables.c
View File

@ -63,6 +63,7 @@
#include "postmaster/postmaster.h"
#include "postmaster/startup.h"
#include "postmaster/syslogger.h"
#include "postmaster/walsummarizer.h"
#include "postmaster/walwriter.h"
#include "replication/logicallauncher.h"
#include "replication/slot.h"
@ -703,6 +704,8 @@ const char *const config_group_names[] =
gettext_noop("Write-Ahead Log / Archive Recovery"),
/* WAL_RECOVERY_TARGET */
gettext_noop("Write-Ahead Log / Recovery Target"),
/* WAL_SUMMARIZATION */
gettext_noop("Write-Ahead Log / Summarization"),
/* REPLICATION_SENDING */
gettext_noop("Replication / Sending Servers"),
/* REPLICATION_PRIMARY */
@ -1786,6 +1789,16 @@ struct config_bool ConfigureNamesBool[] =
NULL, NULL, NULL
},
{
{"summarize_wal", PGC_SIGHUP, WAL_SUMMARIZATION,
gettext_noop("Starts the WAL summarizer process to enable incremental backup."),
NULL
},
&summarize_wal,
false,
NULL, NULL, NULL
},
{
{"hot_standby", PGC_POSTMASTER, REPLICATION_STANDBY,
gettext_noop("Allows connections and queries during recovery."),
@ -3200,6 +3213,19 @@ struct config_int ConfigureNamesInt[] =
check_wal_segment_size, NULL, NULL
},
{
{"wal_summary_keep_time", PGC_SIGHUP, WAL_SUMMARIZATION,
gettext_noop("Time for which WAL summary files should be kept."),
NULL,
GUC_UNIT_MIN,
},
&wal_summary_keep_time,
10 * 24 * 60, /* 10 days */
0,
INT_MAX,
NULL, NULL, NULL
},
{
{"autovacuum_naptime", PGC_SIGHUP, AUTOVACUUM,
gettext_noop("Time to sleep between autovacuum runs."),

5
src/backend/utils/misc/postgresql.conf.sample
View File

@ -302,6 +302,11 @@
#recovery_target_action = 'pause' # 'pause', 'promote', 'shutdown'
# (change requires restart)
# - WAL Summarization -
#summarize_wal = off # run WAL summarizer process?
#wal_summary_keep_time = '10d' # when to remove old summary files, 0 = never
#------------------------------------------------------------------------------
# REPLICATION

1
src/bin/initdb/initdb.c
View File

@ -227,6 +227,7 @@ static char *extra_options = "";
static const char *const subdirs[] = {
"global",
"pg_wal/archive_status",
"pg_wal/summaries",
"pg_commit_ts",
"pg_dynshmem",
"pg_notify",

1
src/common/Makefile
View File

@ -47,6 +47,7 @@ OBJS_COMMON = \
archive.o \
base64.o \
binaryheap.o \
blkreftable.o \
checksum_helper.o \
compression.o \
config_info.o \

1308
src/common/blkreftable.c Normal file
View File

File diff suppressed because it is too large Load Diff

1
src/common/meson.build
View File

@ -4,6 +4,7 @@ common_sources = files(
'archive.c',
'base64.c',
'binaryheap.c',
'blkreftable.c',
'checksum_helper.c',
'compression.c',
'controldata_utils.c',

1
src/include/access/xlog.h
View File

@ -209,6 +209,7 @@ extern int XLogFileOpen(XLogSegNo segno, TimeLineID tli);
extern void CheckXLogRemoved(XLogSegNo segno, TimeLineID tli);
extern XLogSegNo XLogGetLastRemovedSegno(void);
extern XLogSegNo XLogGetOldestSegno(TimeLineID tli);
extern void XLogSetAsyncXactLSN(XLogRecPtr asyncXactLSN);
extern void XLogSetReplicationSlotMinimumLSN(XLogRecPtr lsn);

49
src/include/backup/walsummary.h Normal file
View File

@ -0,0 +1,49 @@
/*-------------------------------------------------------------------------
*
* walsummary.h
* WAL summary management
*
* Portions Copyright (c) 2010-2022, PostgreSQL Global Development Group
*
* src/include/backup/walsummary.h
*
*-------------------------------------------------------------------------
*/
#ifndef WALSUMMARY_H
#define WALSUMMARY_H
#include <time.h>
#include "access/xlogdefs.h"
#include "nodes/pg_list.h"
#include "storage/fd.h"
typedef struct WalSummaryIO
{
File file;
off_t filepos;
} WalSummaryIO;
typedef struct WalSummaryFile
{
XLogRecPtr start_lsn;
XLogRecPtr end_lsn;
TimeLineID tli;
} WalSummaryFile;
extern List *GetWalSummaries(TimeLineID tli, XLogRecPtr start_lsn,
XLogRecPtr end_lsn);
extern List *FilterWalSummaries(List *wslist, TimeLineID tli,
XLogRecPtr start_lsn, XLogRecPtr end_lsn);
extern bool WalSummariesAreComplete(List *wslist,
XLogRecPtr start_lsn, XLogRecPtr end_lsn,
XLogRecPtr *missing_lsn);
extern File OpenWalSummaryFile(WalSummaryFile *ws, bool missing_ok);
extern void RemoveWalSummaryIfOlderThan(WalSummaryFile *ws,
time_t cutoff_time);
extern int ReadWalSummary(void *wal_summary_io, void *data, int length);
extern int WriteWalSummary(void *wal_summary_io, void *data, int length);
extern void ReportWalSummaryError(void *callback_arg, char *fmt,...) pg_attribute_printf(2, 3);
#endif /* WALSUMMARY_H */

19
src/include/catalog/pg_proc.dat
View File

@ -12099,4 +12099,23 @@
proname => 'any_value_transfn', prorettype => 'anyelement',
proargtypes => 'anyelement anyelement', prosrc => 'any_value_transfn' },
{ oid => '8436',
descr => 'list of available WAL summary files',
proname => 'pg_available_wal_summaries', prorows => '100',
proretset => 't', provolatile => 'v', proparallel => 's',
prorettype => 'record', proargtypes => '',
proallargtypes => '{int8,pg_lsn,pg_lsn}',
proargmodes => '{o,o,o}',
proargnames => '{tli,start_lsn,end_lsn}',
prosrc => 'pg_available_wal_summaries' },
{ oid => '8437',
descr => 'contents of a WAL sumamry file',
proname => 'pg_wal_summary_contents', prorows => '100',
proretset => 't', provolatile => 'v', proparallel => 's',
prorettype => 'record', proargtypes => 'int8 pg_lsn pg_lsn',
proallargtypes => '{int8,pg_lsn,pg_lsn,oid,oid,oid,int2,int8,bool}',
proargmodes => '{i,i,i,o,o,o,o,o,o}',
proargnames => '{tli,start_lsn,end_lsn,relfilenode,reltablespace,reldatabase,relforknumber,relblocknumber,is_limit_block}',
prosrc => 'pg_wal_summary_contents' },
]

116
src/include/common/blkreftable.h Normal file
View File

@ -0,0 +1,116 @@
/*-------------------------------------------------------------------------
*
* blkreftable.h
* Block reference tables.
*
* A block reference table is used to keep track of which blocks have
* been modified by WAL records within a certain LSN range.
*
* For each relation fork, there is a "limit block number". All existing
* blocks greater than or equal to the limit block number must be
* considered modified; for those less than the limit block number,
* we maintain a bitmap. When a relation fork is created or dropped,
* the limit block number should be set to 0. When it's truncated,
* the limit block number should be set to the length in blocks to
* which it was truncated.
*
* Portions Copyright (c) 2010-2022, PostgreSQL Global Development Group
*
* src/include/common/blkreftable.h
*
*-------------------------------------------------------------------------
*/
#ifndef BLKREFTABLE_H
#define BLKREFTABLE_H
#include "storage/block.h"
#include "storage/relfilelocator.h"
/* Magic number for serialization file format. */
#define BLOCKREFTABLE_MAGIC 0x652b137b
typedef struct BlockRefTable BlockRefTable;
typedef struct BlockRefTableEntry BlockRefTableEntry;
typedef struct BlockRefTableReader BlockRefTableReader;
typedef struct BlockRefTableWriter BlockRefTableWriter;
/*
* The return value of io_callback_fn should be the number of bytes read
* or written. If an error occurs, the functions should report it and
* not return. When used as a write callback, short writes should be retried
* or treated as errors, so that if the callback returns, the return value
* is always the request length.
*
* report_error_fn should not return.
*/
typedef int (*io_callback_fn) (void *callback_arg, void *data, int length);
typedef void (*report_error_fn) (void *calblack_arg, char *msg,...) pg_attribute_printf(2, 3);
/*
* Functions for manipulating an entire in-memory block reference table.
*/
extern BlockRefTable *CreateEmptyBlockRefTable(void);
extern void BlockRefTableSetLimitBlock(BlockRefTable *brtab,
const RelFileLocator *rlocator,
ForkNumber forknum,
BlockNumber limit_block);
extern void BlockRefTableMarkBlockModified(BlockRefTable *brtab,
const RelFileLocator *rlocator,
ForkNumber forknum,
BlockNumber blknum);
extern void WriteBlockRefTable(BlockRefTable *brtab,
io_callback_fn write_callback,
void *write_callback_arg);
extern BlockRefTableEntry *BlockRefTableGetEntry(BlockRefTable *brtab,
const RelFileLocator *rlocator,
ForkNumber forknum,
BlockNumber *limit_block);
extern int BlockRefTableEntryGetBlocks(BlockRefTableEntry *entry,
BlockNumber start_blkno,
BlockNumber stop_blkno,
BlockNumber *blocks,
int nblocks);
/*
* Functions for reading a block reference table incrementally from disk.
*/
extern BlockRefTableReader *CreateBlockRefTableReader(io_callback_fn read_callback,
void *read_callback_arg,
char *error_filename,
report_error_fn error_callback,
void *error_callback_arg);
extern bool BlockRefTableReaderNextRelation(BlockRefTableReader *reader,
RelFileLocator *rlocator,
ForkNumber *forknum,
BlockNumber *limit_block);
extern unsigned BlockRefTableReaderGetBlocks(BlockRefTableReader *reader,
BlockNumber *blocks,
int nblocks);
extern void DestroyBlockRefTableReader(BlockRefTableReader *reader);
/*
* Functions for writing a block reference table incrementally to disk.
*
* Note that entries must be written in the proper order, that is, sorted by
* database, then tablespace, then relfilenumber, then fork number. Caller
* is responsible for supplying data in the correct order. If that seems hard,
* use an in-memory BlockRefTable instead.
*/
extern BlockRefTableWriter *CreateBlockRefTableWriter(io_callback_fn write_callback,
void *write_callback_arg);
extern void BlockRefTableWriteEntry(BlockRefTableWriter *writer,
BlockRefTableEntry *entry);
extern void DestroyBlockRefTableWriter(BlockRefTableWriter *writer);
extern BlockRefTableEntry *CreateBlockRefTableEntry(RelFileLocator rlocator,
ForkNumber forknum);
extern void BlockRefTableEntrySetLimitBlock(BlockRefTableEntry *entry,
BlockNumber limit_block);
extern void BlockRefTableEntryMarkBlockModified(BlockRefTableEntry *entry,
ForkNumber forknum,
BlockNumber blknum);
extern void BlockRefTableFreeEntry(BlockRefTableEntry *entry);
#endif /* BLKREFTABLE_H */

3
src/include/miscadmin.h
View File

@ -336,6 +336,7 @@ typedef enum BackendType
B_STARTUP,
B_WAL_RECEIVER,
B_WAL_SENDER,
B_WAL_SUMMARIZER,
B_WAL_WRITER,
} BackendType;
@ -442,6 +443,7 @@ typedef enum
CheckpointerProcess,
WalWriterProcess,
WalReceiverProcess,
WalSummarizerProcess,
NUM_AUXPROCTYPES /* Must be last! */
} AuxProcType;
@ -454,6 +456,7 @@ extern PGDLLIMPORT AuxProcType MyAuxProcType;
#define AmCheckpointerProcess() (MyAuxProcType == CheckpointerProcess)
#define AmWalWriterProcess() (MyAuxProcType == WalWriterProcess)
#define AmWalReceiverProcess() (MyAuxProcType == WalReceiverProcess)
#define AmWalSummarizerProcess() (MyAuxProcType == WalSummarizerProcess)
/*****************************************************************************

33
src/include/postmaster/walsummarizer.h Normal file
View File

@ -0,0 +1,33 @@
/*-------------------------------------------------------------------------
*
* walsummarizer.h
*
* Header file for background WAL summarization process.
*
* Portions Copyright (c) 1996-2022, PostgreSQL Global Development Group
*
* IDENTIFICATION
* src/include/postmaster/walsummarizer.h
*
*-------------------------------------------------------------------------
*/
#ifndef WALSUMMARIZER_H
#define WALSUMMARIZER_H
#include "access/xlogdefs.h"
extern bool summarize_wal;
extern int wal_summary_keep_time;
extern Size WalSummarizerShmemSize(void);
extern void WalSummarizerShmemInit(void);
extern void WalSummarizerMain(void) pg_attribute_noreturn();
extern XLogRecPtr GetOldestUnsummarizedLSN(TimeLineID *tli,
bool *lsn_is_exact,
bool reset_pending_lsn);
extern void SetWalSummarizerLatch(void);
extern XLogRecPtr WaitForWalSummarization(XLogRecPtr lsn, long timeout,
XLogRecPtr *pending_lsn);
#endif

9
src/include/storage/proc.h
View File

@ -417,11 +417,12 @@ extern PGDLLIMPORT PGPROC *PreparedXactProcs;
* We set aside some extra PGPROC structures for auxiliary processes,
* ie things that aren't full-fledged backends but need shmem access.
*
* Background writer, checkpointer, WAL writer and archiver run during normal
* operation. Startup process and WAL receiver also consume 2 slots, but WAL
* writer is launched only after startup has exited, so we only need 5 slots.
* Background writer, checkpointer, WAL writer, WAL summarizer, and archiver
* run during normal operation. Startup process and WAL receiver also consume
* 2 slots, but WAL writer is launched only after startup has exited, so we
* only need 6 slots.
*/
#define NUM_AUXILIARY_PROCS 5
#define NUM_AUXILIARY_PROCS 6
/* configurable options */
extern PGDLLIMPORT int DeadlockTimeout;

1
src/include/utils/guc_tables.h
View File

@ -72,6 +72,7 @@ enum config_group
WAL_RECOVERY,
WAL_ARCHIVE_RECOVERY,
WAL_RECOVERY_TARGET,
WAL_SUMMARIZATION,
REPLICATION_SENDING,
REPLICATION_PRIMARY,
REPLICATION_STANDBY,

11
src/tools/pgindent/typedefs.list
View File

@ -4012,3 +4012,14 @@ yyscan_t
z_stream
z_streamp
zic_t
BlockRefTable
BlockRefTableBuffer
BlockRefTableEntry
BlockRefTableKey
BlockRefTableReader
BlockRefTableSerializedEntry
BlockRefTableWriter
SummarizerReadLocalXLogPrivate
WalSummarizerData
WalSummaryFile
WalSummaryIO