mirrors/postgres
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Custom WAL Resource Managers.

Browse Source 
Allow extensions to specify a new custom resource manager (rmgr),
which allows specialized WAL. This is meant to be used by a Table
Access Method or Index Access Method.

Prior to this commit, only Generic WAL was available, which offers
support for recovery and physical replication but not logical
replication.

Reviewed-by: Julien Rouhaud, Bharath Rupireddy, Andres Freund
Discussion: https://postgr.es/m/ed1fb2e22d15d3563ae0eb610f7b61bb15999c0a.camel%40j-davis.com
This commit is contained in:
Jeff Davis 2022-04-06 22:26:43 -07:00
parent a8cfb0c1a9
commit 5c279a6d35
24 changed files with 526 additions and 70 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

4
doc/src/sgml/config.sgml
View File

@ -11189,8 +11189,8 @@ LOG: CleanUpLock: deleting: lock(0xb7acd844) id(24688,24696,0,0,0,1)
<literal>heap2</literal>, <literal>btree</literal>, <literal>hash</literal>,
<literal>gin</literal>, <literal>gist</literal>, <literal>sequence</literal>,
<literal>spgist</literal>, <literal>brin</literal>, and <literal>generic</literal>.
Only superusers and users with the appropriate <literal>SET</literal>
privilege can change this setting.
Extensions may define additional resource managers. Only superusers and users with
the appropriate <literal>SET</literal> privilege can change this setting.
</para>
</listitem>
</varlistentry>

98
doc/src/sgml/custom-rmgr.sgml Normal file
View File

@ -0,0 +1,98 @@
<!-- doc/src/sgml/custom-rmgr.sgml -->
<chapter id="custom-rmgr">
<title>Custom WAL Resource Managers</title>
<para>
This chapter explains the interface between the core
<productname>PostgreSQL</productname> system and custom WAL resource
managers, which enable extensions to integrate directly with the <link
linkend="wal"><acronym>WAL</acronym></link>.
</para>
<para>
An extension, especially a <link linkend="tableam">Table Access
Method</link> or <link linkend="indexam">Index Access Method</link>, may
need to use WAL for recovery, replication, and/or <link
linkend="logicaldecoding">Logical Decoding</link>. Custom resource managers
are a more flexible alternative to <link linkend="generic-wal">Generic
WAL</link> (which does not support logical decoding), but more complex for
an extension to implement.
</para>
<para>
To create a new custom WAL resouce manager, first define an
<structname>RmgrData</structname> structure with implementations for the
resource manager methods. Refer to
<filename>src/backend/access/transam/README</filename> and
<filename>src/include/access/xlog_internal.h</filename> in the
<productname>PostgreSQL</productname> source.
<programlisting>
/*
* Method table for resource managers.
*
* This struct must be kept in sync with the PG_RMGR definition in
* rmgr.c.
*
* rm_identify must return a name for the record based on xl_info (without
* reference to the rmid). For example, XLOG_BTREE_VACUUM would be named
* "VACUUM". rm_desc can then be called to obtain additional detail for the
* record, if available (e.g. the last block).
*
* rm_mask takes as input a page modified by the resource manager and masks
* out bits that shouldn't be flagged by wal_consistency_checking.
*
* RmgrTable[] is indexed by RmgrId values (see rmgrlist.h). If rm_name is
* NULL, the corresponding RmgrTable entry is considered invalid.
*/
typedef struct RmgrData
{
const char *rm_name;
void (*rm_redo) (XLogReaderState *record);
void (*rm_desc) (StringInfo buf, XLogReaderState *record);
const char *(*rm_identify) (uint8 info);
void (*rm_startup) (void);
void (*rm_cleanup) (void);
void (*rm_mask) (char *pagedata, BlockNumber blkno);
void (*rm_decode) (struct LogicalDecodingContext *ctx,
struct XLogRecordBuffer *buf);
} RmgrData;
</programlisting>
</para>
<para>
Then, register your new resource
manager.
<programlisting>
/*
* Register a new custom WAL resource manager.
*
* Resource manager IDs must be globally unique across all extensions. Refer
* to https://wiki.postgresql.org/wiki/CustomWALResourceManager to reserve a
* unique RmgrId for your extension, to avoid conflicts with other extension
* developers. During development, use RM_EXPERIMENTAL_ID to avoid needlessly
* reserving a new ID.
*/
extern void RegisterCustomRmgr(RmgrId rmid, RmgrData *rmgr);
</programlisting>
<function>RegisterCustomRmgr</function> must be called from the
extension module's <link linkend="xfunc-c-dynload">_PG_init</link> function.
While developing a new extension, use <literal>RM_EXPERIMENTAL_ID</literal>
for <parameter>rmid</parameter>. When you ready to release the extension to
users, reserve a new resource manager ID at the <ulink
url="https://wiki.postgresql.org/wiki/CustomWALResourceManagers">Custom WAL
Resource Manager</ulink> page.
</para>
<para>
Place the extension module implementing the custom resource manager in <xref
linkend="guc-shared-preload-libraries"/> so that it will be loaded early
during <productname>PostgreSQL</productname> startup.
</para>
<note>
<para>
The extension must remain in shared_preload_libraries as long as any
custom WAL records may exist in the system. Otherwise
<productname>PostgreSQL</productname> will not be able to apply or decode
the custom WAL records, which may prevent the server from starting.
</para>
</note>
</chapter>

1
doc/src/sgml/filelist.sgml
View File

@ -105,6 +105,7 @@
<!ENTITY storage SYSTEM "storage.sgml">
<!ENTITY tablesample-method SYSTEM "tablesample-method.sgml">
<!ENTITY generic-wal SYSTEM "generic-wal.sgml">
<!ENTITY custom-rmgr SYSTEM "custom-rmgr.sgml">
<!ENTITY backup-manifest SYSTEM "backup-manifest.sgml">
<!-- contrib information -->

19
doc/src/sgml/func.sgml
View File

@ -26001,6 +26001,25 @@ postgres=# SELECT * FROM pg_walfile_name_offset((pg_backup_stop()).lsn);
without recovery, the function returns <literal>NULL</literal>.
</para></entry>
</row>
<row>
<entry role="func_table_entry"><para role="func_signature">
<indexterm>
<primary>pg_get_wal_resource_managers</primary>
</indexterm>
<function>pg_get_wal_resource_managers</function> ()
<returnvalue>setof record</returnvalue>
( <parameter>rm_id</parameter> <type>integer</type>,
<parameter>rm_name</parameter> <type>text</type>,
<parameter>rm_builtin</parameter> <type>boolean</type> )
</para>
<para>
Returns the currently-loaded WAL resource managers in the system. The
column <parameter>rm_builtin</parameter> indicates whether it's a
built-in resource manager, or a custom resource manager loaded by an
extension.
</para></entry>
</row>
</tbody>
</tgroup>
</table>

20
doc/src/sgml/generic-wal.sgml
View File

@ -6,11 +6,25 @@
<para>
Although all built-in WAL-logged modules have their own types of WAL
records, there is also a generic WAL record type, which describes changes
to pages in a generic way. This is useful for extensions that provide
custom access methods, because they cannot register their own WAL redo
routines.
to pages in a generic way. This is useful for extensions that provide
custom access methods.
</para>
<para>
In comparison with <link linkend="custom-rmgr">Custom WAL Resource
Managers</link>, Generic WAL is simpler for an extension to implement and
does not require the extension library to be loaded in order to apply the
records.
</para>
<note>
<para>
Generic WAL records are ignored during <link
linkend="logicaldecoding">Logical Decoding</link>. If logical decoding is
required for your extension, consider a Custom WAL Resource Manager.
</para>
</note>
<para>
The API for constructing generic WAL records is defined in
<filename>access/generic_xlog.h</filename> and implemented

1
doc/src/sgml/postgres.sgml
View File

@ -262,6 +262,7 @@ break is not needed in a wider output rendering.
&tableam;
&indexam;
&generic-wal;
&custom-rmgr;
&btree;
&gist;
&spgist;

8
doc/src/sgml/ref/pg_waldump.sgml
View File

@ -173,6 +173,14 @@ PostgreSQL documentation
If <literal>list</literal> is passed as name, print a list of valid resource manager
names, and exit.
</para>
<para>
Extensions may define custom resource managers, but pg_waldump does
not load the extension module and therefore does not recognize custom
resource managers by name. Instead, you can specify the custom
resource managers as <literal>custom###</literal> where
"<literal>###</literal>" is the three-digit resource manager ID. Names
of this form will always be considered valid.
</para>
</listitem>
</varlistentry>

124
src/backend/access/transam/rmgr.c
View File

@ -24,16 +24,138 @@
#include "commands/dbcommands_xlog.h"
#include "commands/sequence.h"
#include "commands/tablespace.h"
#include "fmgr.h"
#include "funcapi.h"
#include "miscadmin.h"
#include "replication/decode.h"
#include "replication/message.h"
#include "replication/origin.h"
#include "storage/standby.h"
#include "utils/builtins.h"
#include "utils/relmapper.h"
/* must be kept in sync with RmgrData definition in xlog_internal.h */
#define PG_RMGR(symname,name,redo,desc,identify,startup,cleanup,mask,decode) \
{ name, redo, desc, identify, startup, cleanup, mask, decode },
const RmgrData RmgrTable[RM_MAX_ID + 1] = {
RmgrData RmgrTable[RM_MAX_ID + 1] = {
#include "access/rmgrlist.h"
};
/*
* Start up all resource managers.
*/
void
RmgrStartup(void)
{
for (int rmid = 0; rmid <= RM_MAX_ID; rmid++)
{
if (!RmgrIdExists(rmid))
continue;
if (RmgrTable[rmid].rm_startup != NULL)
RmgrTable[rmid].rm_startup();
}
}
/*
* Clean up all resource managers.
*/
void
RmgrCleanup(void)
{
for (int rmid = 0; rmid <= RM_MAX_ID; rmid++)
{
if (!RmgrIdExists(rmid))
continue;
if (RmgrTable[rmid].rm_cleanup != NULL)
RmgrTable[rmid].rm_cleanup();
}
}
/*
* Emit ERROR when we encounter a record with an RmgrId we don't
* recognize.
*/
void
RmgrNotFound(RmgrId rmid)
{
ereport(ERROR, (errmsg("resource manager with ID %d not registered", rmid),
errhint("Include the extension module that implements this resource manager in shared_preload_libraries.")));
}
/*
* Register a new custom WAL resource manager.
*
* Resource manager IDs must be globally unique across all extensions. Refer
* to https://wiki.postgresql.org/wiki/CustomWALResourceManager to reserve a
* unique RmgrId for your extension, to avoid conflicts with other extension
* developers. During development, use RM_EXPERIMENTAL_ID to avoid needlessly
* reserving a new ID.
*/
void
RegisterCustomRmgr(RmgrId rmid, RmgrData *rmgr)
{
if (rmgr->rm_name == NULL || strlen(rmgr->rm_name) == 0)
ereport(ERROR, (errmsg("custom resource manager name is invalid"),
errhint("Provide a non-empty name for the custom resource manager.")));
if (!RMID_IS_CUSTOM(rmid))
ereport(ERROR, (errmsg("custom resource manager ID %d is out of range", rmid),
errhint("Provide a custom resource manager ID between %d and %d.",
RM_MIN_CUSTOM_ID, RM_MAX_CUSTOM_ID)));
if (!process_shared_preload_libraries_in_progress)
ereport(ERROR,
(errmsg("failed to register custom resource manager \"%s\" with ID %d", rmgr->rm_name, rmid),
errdetail("Custom resource manager must be registered while initializing modules in shared_preload_libraries.")));
if (RmgrTable[rmid].rm_name != NULL)
ereport(ERROR,
(errmsg("failed to register custom resource manager \"%s\" with ID %d", rmgr->rm_name, rmid),
errdetail("Custom resource manager \"%s\" already registered with the same ID.",
RmgrTable[rmid].rm_name)));
/* check for existing rmgr with the same name */
for (int existing_rmid = 0; existing_rmid <= RM_MAX_ID; existing_rmid++)
{
if (!RmgrIdExists(existing_rmid))
continue;
if (!pg_strcasecmp(RmgrTable[existing_rmid].rm_name, rmgr->rm_name))
ereport(ERROR,
(errmsg("failed to register custom resource manager \"%s\" with ID %d", rmgr->rm_name, rmid),
errdetail("Existing resource manager with ID %d has the same name.", existing_rmid)));
}
/* register it */
RmgrTable[rmid] = *rmgr;
ereport(LOG,
(errmsg("registered custom resource manager \"%s\" with ID %d",
rmgr->rm_name, rmid)));
}
/* SQL SRF showing loaded resource managers */
Datum
pg_get_wal_resource_managers(PG_FUNCTION_ARGS)
{
#define PG_GET_RESOURCE_MANAGERS_COLS 3
ReturnSetInfo *rsinfo = (ReturnSetInfo *) fcinfo->resultinfo;
Datum values[PG_GET_RESOURCE_MANAGERS_COLS];
bool nulls[PG_GET_RESOURCE_MANAGERS_COLS] = {0};
SetSingleFuncCall(fcinfo, 0);
for (int rmid = 0; rmid <= RM_MAX_ID; rmid++)
{
if (!RmgrIdExists(rmid))
continue;
values[0] = Int32GetDatum(rmid);
values[1] = CStringGetTextDatum(GetRmgr(rmid).rm_name);
values[2] = BoolGetDatum(RMID_IS_BUILTIN(rmid));
tuplestore_putvalues(rsinfo->setResult, rsinfo->setDesc, values, nulls);
}
return (Datum) 0;
}

2
src/backend/access/transam/xlogreader.c
View File

@ -1102,7 +1102,7 @@ ValidXLogRecordHeader(XLogReaderState *state, XLogRecPtr RecPtr,
(uint32) SizeOfXLogRecord, record->xl_tot_len);
return false;
}
if (record->xl_rmid > RM_MAX_ID)
if (!RMID_IS_VALID(record->xl_rmid))
{
report_invalid_record(state,
"invalid resource manager ID %u at %X/%X",

33
src/backend/access/transam/xlogrecovery.c
View File

@ -1541,7 +1541,6 @@ ShutdownWalRecovery(void)
void
PerformWalRecovery(void)
{
int rmid;
XLogRecord *record;
bool reachedRecoveryTarget = false;
TimeLineID replayTLI;
@ -1614,12 +1613,7 @@ PerformWalRecovery(void)
InRedo = true;
/* Initialize resource managers */
for (rmid = 0; rmid <= RM_MAX_ID; rmid++)
{
if (RmgrTable[rmid].rm_startup != NULL)
RmgrTable[rmid].rm_startup();
}
RmgrStartup();
ereport(LOG,
(errmsg("redo starts at %X/%X",
@ -1756,12 +1750,7 @@ PerformWalRecovery(void)
}
}
/* Allow resource managers to do any required cleanup. */
for (rmid = 0; rmid <= RM_MAX_ID; rmid++)
{
if (RmgrTable[rmid].rm_cleanup != NULL)
RmgrTable[rmid].rm_cleanup();
}
RmgrCleanup();
ereport(LOG,
(errmsg("redo done at %X/%X system usage: %s",
@ -1881,7 +1870,7 @@ ApplyWalRecord(XLogReaderState *xlogreader, XLogRecord *record, TimeLineID *repl
xlogrecovery_redo(xlogreader, *replayTLI);
/* Now apply the WAL record itself */
RmgrTable[record->xl_rmid].rm_redo(xlogreader);
GetRmgr(record->xl_rmid).rm_redo(xlogreader);
/*
* After redo, check whether the backup pages associated with the WAL
@ -2111,20 +2100,20 @@ rm_redo_error_callback(void *arg)
void
xlog_outdesc(StringInfo buf, XLogReaderState *record)
{
RmgrId rmid = XLogRecGetRmid(record);
RmgrData rmgr = GetRmgr(XLogRecGetRmid(record));
uint8 info = XLogRecGetInfo(record);
const char *id;
appendStringInfoString(buf, RmgrTable[rmid].rm_name);
appendStringInfoString(buf, rmgr.rm_name);
appendStringInfoChar(buf, '/');
id = RmgrTable[rmid].rm_identify(info);
id = rmgr.rm_identify(info);
if (id == NULL)
appendStringInfo(buf, "UNKNOWN (%X): ", info & ~XLR_INFO_MASK);
else
appendStringInfo(buf, "%s: ", id);
RmgrTable[rmid].rm_desc(buf, record);
rmgr.rm_desc(buf, record);
}
#ifdef WAL_DEBUG
@ -2273,7 +2262,7 @@ getRecordTimestamp(XLogReaderState *record, TimestampTz *recordXtime)
static void
verifyBackupPageConsistency(XLogReaderState *record)
{
RmgrId rmid = XLogRecGetRmid(record);
RmgrData rmgr = GetRmgr(XLogRecGetRmid(record));
RelFileNode rnode;
ForkNumber forknum;
BlockNumber blkno;
@ -2353,10 +2342,10 @@ verifyBackupPageConsistency(XLogReaderState *record)
* If masking function is defined, mask both the primary and replay
* images
*/
if (RmgrTable[rmid].rm_mask != NULL)
if (rmgr.rm_mask != NULL)
{
RmgrTable[rmid].rm_mask(replay_image_masked, blkno);
RmgrTable[rmid].rm_mask(primary_image_masked, blkno);
rmgr.rm_mask(replay_image_masked, blkno);
rmgr.rm_mask(primary_image_masked, blkno);
}
/* Time to compare the primary and replay images. */

6
src/backend/postmaster/postmaster.c
View File

@ -1039,6 +1039,12 @@ PostmasterMain(int argc, char *argv[])
*/
InitializeShmemGUCs();
/*
* Now that modules have been loaded, we can process any custom resource
* managers specified in the wal_consistency_checking GUC.
*/
InitializeWalConsistencyChecking();
/*
* If -C was specified with a runtime-computed GUC, we held off printing
* the value earlier, as the GUC was not yet initialized. We handle -C

8
src/backend/replication/logical/decode.c
View File

@ -94,7 +94,7 @@ LogicalDecodingProcessRecord(LogicalDecodingContext *ctx, XLogReaderState *recor
{
XLogRecordBuffer buf;
TransactionId txid;
RmgrId rmid;
RmgrData rmgr;
buf.origptr = ctx->reader->ReadRecPtr;
buf.endptr = ctx->reader->EndRecPtr;
@ -115,10 +115,10 @@ LogicalDecodingProcessRecord(LogicalDecodingContext *ctx, XLogReaderState *recor
buf.origptr);
}
rmid = XLogRecGetRmid(record);
rmgr = GetRmgr(XLogRecGetRmid(record));
if (RmgrTable[rmid].rm_decode != NULL)
RmgrTable[rmid].rm_decode(ctx, &buf);
if (rmgr.rm_decode != NULL)
rmgr.rm_decode(ctx, &buf);
else
{
/* just deal with xid, and done */

2
src/backend/utils/init/miscinit.c
View File

@ -1610,6 +1610,7 @@ char *local_preload_libraries_string = NULL;
/* Flag telling that we are loading shared_preload_libraries */
bool process_shared_preload_libraries_in_progress = false;
bool process_shared_preload_libraries_done = false;
/*
* load the shared libraries listed in 'libraries'
@ -1677,6 +1678,7 @@ process_shared_preload_libraries(void)
"shared_preload_libraries",
false);
process_shared_preload_libraries_in_progress = false;
process_shared_preload_libraries_done = true;
}
/*

77
src/backend/utils/misc/guc.c
View File

@ -245,6 +245,11 @@ static bool check_default_with_oids(bool *newval, void **extra, GucSource source
static ConfigVariable *ProcessConfigFileInternal(GucContext context,
bool applySettings, int elevel);
/*
* Track whether there were any deferred checks for custom resource managers
* specified in wal_consistency_checking.
*/
static bool check_wal_consistency_checking_deferred = false;
/*
* Options for enum values defined in this module.
@ -5835,6 +5840,36 @@ InitializeGUCOptions(void)
InitializeGUCOptionsFromEnvironment();
}
/*
* If any custom resource managers were specified in the
* wal_consistency_checking GUC, processing was deferred. Now that
* shared_preload_libraries have been loaded, process wal_consistency_checking
* again.
*/
void
InitializeWalConsistencyChecking(void)
{
Assert(process_shared_preload_libraries_done);
if (check_wal_consistency_checking_deferred)
{
struct config_generic *guc;
guc = find_option("wal_consistency_checking", false, false, ERROR);
check_wal_consistency_checking_deferred = false;
set_config_option("wal_consistency_checking",
wal_consistency_checking_string,
PGC_POSTMASTER, guc->source,
GUC_ACTION_SET, true, ERROR, false);
/* checking should not be deferred again */
Assert(!check_wal_consistency_checking_deferred);
}
}
/*
* Assign any GUC values that can come from the server's environment.
*
@ -11882,13 +11917,13 @@ check_wal_consistency_checking(char **newval, void **extra, GucSource source)
{
char *tok = (char *) lfirst(l);
bool found = false;
RmgrId rmid;
int rmid;
/* Check for 'all'. */
if (pg_strcasecmp(tok, "all") == 0)
{
for (rmid = 0; rmid <= RM_MAX_ID; rmid++)
if (RmgrTable[rmid].rm_mask != NULL)
if (RmgrIdExists(rmid) && GetRmgr(rmid).rm_mask != NULL)
newwalconsistency[rmid] = true;
found = true;
}
@ -11900,8 +11935,8 @@ check_wal_consistency_checking(char **newval, void **extra, GucSource source)
*/
for (rmid = 0; rmid <= RM_MAX_ID; rmid++)
{
if (pg_strcasecmp(tok, RmgrTable[rmid].rm_name) == 0 &&
RmgrTable[rmid].rm_mask != NULL)
if (RmgrIdExists(rmid) && GetRmgr(rmid).rm_mask != NULL &&
pg_strcasecmp(tok, GetRmgr(rmid).rm_name) == 0)
{
newwalconsistency[rmid] = true;
found = true;
@ -11912,10 +11947,21 @@ check_wal_consistency_checking(char **newval, void **extra, GucSource source)
/* If a valid resource manager is found, check for the next one. */
if (!found)
{
GUC_check_errdetail("Unrecognized key word: \"%s\".", tok);
pfree(rawstring);
list_free(elemlist);
return false;
/*
* Perhaps it's a custom resource manager. If so, defer checking
* until InitializeWalConsistencyChecking().
*/
if (!process_shared_preload_libraries_done)
{
check_wal_consistency_checking_deferred = true;
}
else
{
GUC_check_errdetail("Unrecognized key word: \"%s\".", tok);
pfree(rawstring);
list_free(elemlist);
return false;
}
}
}
@ -11931,7 +11977,20 @@ check_wal_consistency_checking(char **newval, void **extra, GucSource source)
static void
assign_wal_consistency_checking(const char *newval, void *extra)
{
wal_consistency_checking = (bool *) extra;
/*
* If some checks were deferred, it's possible that the checks will fail
* later during InitializeWalConsistencyChecking(). But in that case, the
* postmaster will exit anyway, so it's safe to proceed with the
* assignment.
*
* Any built-in resource managers specified are assigned immediately,
* which affects WAL created before shared_preload_libraries are
* processed. Any custom resource managers specified won't be assigned
* until after shared_preload_libraries are processed, but that's OK
* because WAL for a custom resource manager can't be written before the
* module is loaded anyway.
*/
wal_consistency_checking = extra;
}
static bool

11
src/bin/pg_rewind/parsexlog.c
View File

@ -25,8 +25,8 @@
#include "pg_rewind.h"
/*
* RmgrNames is an array of resource manager names, to make error messages
* a bit nicer.
* RmgrNames is an array of the built-in resource manager names, to make error
* messages a bit nicer.
*/
#define PG_RMGR(symname,name,redo,desc,identify,startup,cleanup,mask,decode) \
name,
@ -35,6 +35,9 @@ static const char *RmgrNames[RM_MAX_ID + 1] = {
#include "access/rmgrlist.h"
};
#define RmgrName(rmid) (((rmid) <= RM_MAX_BUILTIN_ID) ? \
RmgrNames[rmid] : "custom")
static void extractPageInfo(XLogReaderState *record);
static int xlogreadfd = -1;
@ -436,9 +439,9 @@ extractPageInfo(XLogReaderState *record)
* track that change.
*/
pg_fatal("WAL record modifies a relation, but record type is not recognized: "
"lsn: %X/%X, rmgr: %s, info: %02X",
"lsn: %X/%X, rmid: %d, rmgr: %s, info: %02X",
LSN_FORMAT_ARGS(record->ReadRecPtr),
RmgrNames[rmid], info);
rmid, RmgrName(rmid), info);
}
for (block_id = 0; block_id <= XLogRecMaxBlockId(record); block_id++)

66
src/bin/pg_waldump/pg_waldump.c
View File

@ -80,8 +80,8 @@ typedef struct XLogDumpStats
uint64 count;
XLogRecPtr startptr;
XLogRecPtr endptr;
Stats rmgr_stats[RM_NEXT_ID];
Stats record_stats[RM_NEXT_ID][MAX_XLINFO_TYPES];
Stats rmgr_stats[RM_MAX_ID + 1];
Stats record_stats[RM_MAX_ID + 1][MAX_XLINFO_TYPES];
} XLogDumpStats;
#define fatal_error(...) do { pg_log_fatal(__VA_ARGS__); exit(EXIT_FAILURE); } while(0)
@ -104,9 +104,9 @@ print_rmgr_list(void)
{
int i;
for (i = 0; i <= RM_MAX_ID; i++)
for (i = 0; i <= RM_MAX_BUILTIN_ID; i++)
{
printf("%s\n", RmgrDescTable[i].rm_name);
printf("%s\n", GetRmgrDesc(i)->rm_name);
}
}
@ -535,7 +535,7 @@ static void
XLogDumpDisplayRecord(XLogDumpConfig *config, XLogReaderState *record)
{
const char *id;
const RmgrDescData *desc = &RmgrDescTable[XLogRecGetRmid(record)];
const RmgrDescData *desc = GetRmgrDesc(XLogRecGetRmid(record));
uint32 rec_len;
uint32 fpi_len;
RelFileNode rnode;
@ -720,7 +720,7 @@ XLogDumpDisplayStats(XLogDumpConfig *config, XLogDumpStats *stats)
* calculate column totals.
*/
for (ri = 0; ri < RM_NEXT_ID; ri++)
for (ri = 0; ri < RM_MAX_ID; ri++)
{
total_count += stats->rmgr_stats[ri].count;
total_rec_len += stats->rmgr_stats[ri].rec_len;
@ -741,13 +741,18 @@ XLogDumpDisplayStats(XLogDumpConfig *config, XLogDumpStats *stats)
"Type", "N", "(%)", "Record size", "(%)", "FPI size", "(%)", "Combined size", "(%)",
"----", "-", "---", "-----------", "---", "--------", "---", "-------------", "---");
for (ri = 0; ri < RM_NEXT_ID; ri++)
for (ri = 0; ri <= RM_MAX_ID; ri++)
{
uint64 count,
rec_len,
fpi_len,
tot_len;
const RmgrDescData *desc = &RmgrDescTable[ri];
const RmgrDescData *desc;
if (!RMID_IS_VALID(ri))
continue;
desc = GetRmgrDesc(ri);
if (!config->stats_per_record)
{
@ -756,6 +761,9 @@ XLogDumpDisplayStats(XLogDumpConfig *config, XLogDumpStats *stats)
fpi_len = stats->rmgr_stats[ri].fpi_len;
tot_len = rec_len + fpi_len;
if (RMID_IS_CUSTOM(ri) && count == 0)
continue;
XLogDumpStatsRow(desc->rm_name,
count, total_count, rec_len, total_rec_len,
fpi_len, total_fpi_len, tot_len, total_len);
@ -1000,7 +1008,7 @@ main(int argc, char **argv)
break;
case 'r':
{
int i;
int rmid;
if (pg_strcasecmp(optarg, "list") == 0)
{
@ -1008,20 +1016,42 @@ main(int argc, char **argv)
exit(EXIT_SUCCESS);
}
for (i = 0; i <= RM_MAX_ID; i++)
/*
* First look for the generated name of a custom rmgr, of
* the form "custom###". We accept this form, because the
* custom rmgr module is not loaded, so there's no way to
* know the real name. This convention should be
* consistent with that in rmgrdesc.c.
*/
if (sscanf(optarg, "custom%03d", &rmid) == 1)
{
if (pg_strcasecmp(optarg, RmgrDescTable[i].rm_name) == 0)
if (!RMID_IS_CUSTOM(rmid))
{
config.filter_by_rmgr[i] = true;
config.filter_by_rmgr_enabled = true;
break;
pg_log_error("custom resource manager \"%s\" does not exist",
optarg);
goto bad_argument;
}
config.filter_by_rmgr[rmid] = true;
config.filter_by_rmgr_enabled = true;
}
if (i > RM_MAX_ID)
else
{
pg_log_error("resource manager \"%s\" does not exist",
optarg);
goto bad_argument;
/* then look for builtin rmgrs */
for (rmid = 0; rmid <= RM_MAX_BUILTIN_ID; rmid++)
{
if (pg_strcasecmp(optarg, GetRmgrDesc(rmid)->rm_name) == 0)
{
config.filter_by_rmgr[rmid] = true;
config.filter_by_rmgr_enabled = true;
break;
}
}
if (rmid > RM_MAX_BUILTIN_ID)
{
pg_log_error("resource manager \"%s\" does not exist",
optarg);
goto bad_argument;
}
}
}
break;

61
src/bin/pg_waldump/rmgrdesc.c
View File

@ -35,6 +35,65 @@
#define PG_RMGR(symname,name,redo,desc,identify,startup,cleanup,mask,decode) \
{ name, desc, identify},
const RmgrDescData RmgrDescTable[RM_MAX_ID + 1] = {
static const RmgrDescData RmgrDescTable[RM_N_BUILTIN_IDS] = {
#include "access/rmgrlist.h"
};
#define CUSTOM_NUMERIC_NAME_LEN sizeof("custom###")
static char CustomNumericNames[RM_N_CUSTOM_IDS][CUSTOM_NUMERIC_NAME_LEN] = {0};
static RmgrDescData CustomRmgrDesc[RM_N_CUSTOM_IDS] = {0};
static bool CustomRmgrDescInitialized = false;
/*
* No information on custom resource managers; just print the ID.
*/
static void
default_desc(StringInfo buf, XLogReaderState *record)
{
appendStringInfo(buf, "rmid: %d", XLogRecGetRmid(record));
}
/*
* No information on custom resource managers; just return NULL and let the
* caller handle it.
*/
static const char *
default_identify(uint8 info)
{
return NULL;
}
/*
* We are unable to get the real name of a custom rmgr because the module is
* not loaded. Generate a table of rmgrs with numeric names of the form
* "custom###", where "###" is the 3-digit resource manager ID.
*/
static void
initialize_custom_rmgrs(void)
{
for (int i = 0; i < RM_N_CUSTOM_IDS; i++)
{
snprintf(CustomNumericNames[i], CUSTOM_NUMERIC_NAME_LEN,
"custom%03d", i + RM_MIN_CUSTOM_ID);
CustomRmgrDesc[i].rm_name = CustomNumericNames[i];
CustomRmgrDesc[i].rm_desc = default_desc;
CustomRmgrDesc[i].rm_identify = default_identify;
}
CustomRmgrDescInitialized = true;
}
const RmgrDescData *
GetRmgrDesc(RmgrId rmid)
{
Assert(RMID_IS_VALID(rmid));
if (RMID_IS_BUILTIN(rmid))
return &RmgrDescTable[rmid];
else
{
if (!CustomRmgrDescInitialized)
initialize_custom_rmgrs();
return &CustomRmgrDesc[rmid - RM_MIN_CUSTOM_ID];
}
}

2
src/bin/pg_waldump/rmgrdesc.h
View File

@ -18,6 +18,6 @@ typedef struct RmgrDescData
const char *(*rm_identify) (uint8 info);
} RmgrDescData;
extern const RmgrDescData RmgrDescTable[];
extern const RmgrDescData *GetRmgrDesc(RmgrId rmid);
#endif /* RMGRDESC_H */

19
src/include/access/rmgr.h
View File

@ -30,6 +30,23 @@ typedef enum RmgrIds
#undef PG_RMGR
#define RM_MAX_ID (RM_NEXT_ID - 1)
#define RM_MAX_ID UINT8_MAX
#define RM_MAX_BUILTIN_ID (RM_NEXT_ID - 1)
#define RM_MIN_CUSTOM_ID 128
#define RM_MAX_CUSTOM_ID UINT8_MAX
#define RM_N_IDS (UINT8_MAX + 1)
#define RM_N_BUILTIN_IDS (RM_MAX_BUILTIN_ID + 1)
#define RM_N_CUSTOM_IDS (RM_MAX_CUSTOM_ID - RM_MIN_CUSTOM_ID + 1)
#define RMID_IS_BUILTIN(rmid) ((rmid) <= RM_MAX_BUILTIN_ID)
#define RMID_IS_CUSTOM(rmid) ((rmid) >= RM_MIN_CUSTOM_ID && \
(rmid) <= RM_MAX_CUSTOM_ID)
#define RMID_IS_VALID(rmid) (RMID_IS_BUILTIN((rmid)) || RMID_IS_CUSTOM((rmid)))
/*
* RmgrId to use for extensions that require an RmgrId, but are still in
* development and have not reserved their own unique RmgrId yet. See:
* https://wiki.postgresql.org/wiki/CustomWALResourceManagers
*/
#define RM_EXPERIMENTAL_ID 128
#endif /* RMGR_H */

23
src/include/access/xlog_internal.h
View File

@ -304,7 +304,8 @@ struct XLogRecordBuffer;
* rm_mask takes as input a page modified by the resource manager and masks
* out bits that shouldn't be flagged by wal_consistency_checking.
*
* RmgrTable[] is indexed by RmgrId values (see rmgrlist.h).
* RmgrTable[] is indexed by RmgrId values (see rmgrlist.h). If rm_name is
* NULL, the corresponding RmgrTable entry is considered invalid.
*/
typedef struct RmgrData
{
@ -319,7 +320,25 @@ typedef struct RmgrData
struct XLogRecordBuffer *buf);
} RmgrData;
extern const RmgrData RmgrTable[];
extern RmgrData RmgrTable[];
extern void RmgrStartup(void);
extern void RmgrCleanup(void);
extern void RmgrNotFound(RmgrId rmid);
extern void RegisterCustomRmgr(RmgrId rmid, RmgrData *rmgr);
static inline bool
RmgrIdExists(RmgrId rmid)
{
return RmgrTable[rmid].rm_name != NULL;
}
static inline RmgrData
GetRmgr(RmgrId rmid)
{
if (unlikely(!RmgrIdExists(rmid)))
RmgrNotFound(rmid);
return RmgrTable[rmid];
}
/*
* Exported to support xlog switching from checkpointer

2
src/include/catalog/catversion.h
View File

@ -53,6 +53,6 @@
*/
/* yyyymmddN */
#define CATALOG_VERSION_NO 202204071
#define CATALOG_VERSION_NO 202204072
#endif

7
src/include/catalog/pg_proc.dat
View File

@ -6356,6 +6356,13 @@
prorettype => 'text', proargtypes => '',
prosrc => 'pg_get_wal_replay_pause_state' },
{ oid => '8189', descr => 'get resource managers loaded in system',
proname => 'pg_get_wal_resource_managers', prorows => '50', proretset => 't',
provolatile => 'v', prorettype => 'record', proargtypes => '',
proallargtypes => '{int4,text,bool}', proargmodes => '{o,o,o}',
proargnames => '{rm_id, rm_name, rm_builtin}',
prosrc => 'pg_get_wal_resource_managers' },
{ oid => '2621', descr => 'reload configuration files',
proname => 'pg_reload_conf', provolatile => 'v', prorettype => 'bool',
proargtypes => '', prosrc => 'pg_reload_conf' },

1
src/include/miscadmin.h
View File

@ -465,6 +465,7 @@ extern void BaseInit(void);
/* in utils/init/miscinit.c */
extern bool IgnoreSystemIndexes;
extern PGDLLIMPORT bool process_shared_preload_libraries_in_progress;
extern bool process_shared_preload_libraries_done;
extern char *session_preload_libraries_string;
extern char *shared_preload_libraries_string;
extern char *local_preload_libraries_string;

1
src/include/utils/guc.h
View File

@ -367,6 +367,7 @@ extern void ProcessConfigFile(GucContext context);
extern char *convert_GUC_name_for_parameter_acl(const char *name);
extern bool check_GUC_name_for_parameter_acl(const char *name);
extern void InitializeGUCOptions(void);
extern void InitializeWalConsistencyChecking(void);
extern bool SelectConfigFiles(const char *userDoption, const char *progname);
extern void ResetAllOptions(void);
extern void AtStart_GUC(void);