mirrors/postgres
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Add pg_basebackup tool for streaming base backups

Browse Source 
This tool makes it possible to do the pg_start_backup/
copy files/pg_stop_backup step in a single command.

There are still some steps to be done before this is a
complete backup solution, such as the ability to stream
the required WAL logs, but it's still usable, and
could do with some buildfarm coverage.

In passing, make the checkpoint request optionally
fast instead of hardcoding it.

Magnus Hagander, reviewed by Fujii Masao and Dimitri Fontaine
This commit is contained in:
Magnus Hagander 2011-01-23 12:21:23 +01:00
parent 6f59777c65
commit 048d148fe6
17 changed files with 1528 additions and 13 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

10
doc/src/sgml/backup.sgml
View File

@ -812,6 +812,16 @@ SELECT pg_stop_backup();
</orderedlist>
</para>
<para>
You can also use the <xref linkend="app-pgbasebackup"> tool to take
the backup, instead of manually copying the files. This tool will take
care of the <function>pg_start_backup()</>, copy and
<function>pg_stop_backup()</> steps automatically, and transfers the
backup over a regular <productname>PostgreSQL</productname> connection
using the replication protocol, instead of requiring filesystem level
access.
</para>
<para>
Some file system backup tools emit warnings or errors
if the files they are trying to copy change while the copy proceeds.

3
doc/src/sgml/config.sgml
View File

@ -1921,7 +1921,8 @@ SET ENABLE_SEQSCAN TO OFF;
<listitem>
<para>
Specifies the maximum number of concurrent connections from standby
servers (i.e., the maximum number of simultaneously running WAL sender
servers or streaming base backup clients (i.e., the maximum number of
simultaneously running WAL sender
processes). The default is zero. This parameter can only be set at
server start. <varname>wal_level</> must be set to <literal>archive</>
or <literal>hot_standby</> to allow connections from standby servers.

11
doc/src/sgml/protocol.sgml
View File

@ -1460,7 +1460,7 @@ The commands accepted in walsender mode are:
</varlistentry>
<varlistentry>
<term>BASE_BACKUP [<literal>LABEL</literal> <replaceable>'label'</replaceable>] [<literal>PROGRESS</literal>]</term>
<term>BASE_BACKUP [<literal>LABEL</literal> <replaceable>'label'</replaceable>] [<literal>PROGRESS</literal>] [<literal>FAST</literal>]</term>
<listitem>
<para>
Instructs the server to start streaming a base backup.
@ -1496,6 +1496,15 @@ The commands accepted in walsender mode are:
</para>
</listitem>
</varlistentry>
<varlistentry>
<term><literal>FAST</></term>
<listitem>
<para>
Request a fast checkpoint.
</para>
</listitem>
</varlistentry>
</variablelist>
</para>
<para>

1
doc/src/sgml/ref/allfiles.sgml
View File

@ -160,6 +160,7 @@ Complete list of usable sgml source files in this directory.
<!entity dropuser system "dropuser.sgml">
<!entity ecpgRef system "ecpg-ref.sgml">
<!entity initdb system "initdb.sgml">
<!entity pgBasebackup system "pg_basebackup.sgml">
<!entity pgConfig system "pg_config-ref.sgml">
<!entity pgControldata system "pg_controldata.sgml">
<!entity pgCtl system "pg_ctl-ref.sgml">

397
doc/src/sgml/ref/pg_basebackup.sgml Normal file
View File

@ -0,0 +1,397 @@
<!--
doc/src/sgml/ref/pg_basebackup.sgml
PostgreSQL documentation
-->
<refentry id="app-pgbasebackup">
<refmeta>
<refentrytitle>pg_basebackup</refentrytitle>
<manvolnum>1</manvolnum>
<refmiscinfo>Application</refmiscinfo>
</refmeta>
<refnamediv>
<refname>pg_basebackup</refname>
<refpurpose>take a base backup of a <productname>PostgreSQL</productname> cluster</refpurpose>
</refnamediv>
<indexterm zone="app-pgbasebackup">
<primary>pg_basebackup</primary>
</indexterm>
<refsynopsisdiv>
<cmdsynopsis>
<command>pg_basebackup</command>
<arg rep="repeat"><replaceable>option</></arg>
</cmdsynopsis>
</refsynopsisdiv>
<refsect1>
<title>
Description
</title>
<para>
<application>pg_basebackup</application> is used to take base backups of
a running <productname>PostgreSQL</productname> database cluster. These
are taken without affecting other clients to the database, and can be used
both for point-in-time recovery (see <xref linkend="continuous-archiving">)
and as the starting point for a log shipping or streaming replication standby
servers (see <xref linkend="warm-standby">).
</para>
<para>
<application>pg_basebackup</application> makes a binary copy of the database
cluster files, while making sure the system is automatically put in and
out of backup mode automatically. Backups are always taken of the entire
database cluster, it is not possible to back up individual databases or
database objects. For individual database backups, a tool such as
<xref linkend="APP-PGDUMP"> must be used.
</para>
<para>
The backup is made over a regular <productname>PostgreSQL</productname>
connection, and uses the replication protocol. The connection must be
made with a user having <literal>REPLICATION</literal> permissions (see
<xref linkend="role-attributes">), and the user must be granted explicit
permissions in <filename>pg_hba.conf</filename>. The server must also
be configured with <xref linkend="guc-max-wal-senders"> set high enough
to leave at least one session available for the backup.
</para>
<para>
Only one backup can be concurrently active in
<productname>PostgreSQL</productname>, meaning that only one instance of
<application>pg_basebackup</application> can run at the same time
against a single database cluster.
</para>
</refsect1>
<refsect1>
<title>Options</title>
<para>
The following command-line options control the location and format of the
output.
<variablelist>
<varlistentry>
<term><option>-D <replaceable class="parameter">directory</replaceable></option></term>
<term><option>--pgdata=<replaceable class="parameter">directory</replaceable></option></term>
<listitem>
<para>
Directory to write the output to.
</para>
<para>
When the backup is in tar mode, and the directory is specified as
<literal>-</literal> (dash), the tar file will be written to
<literal>stdout</literal>.
</para>
<para>
This parameter is required.
</para>
</listitem>
</varlistentry>
<varlistentry>
<term><option>-F <replaceable class="parameter">format</replaceable></option></term>
<term><option>--format=<replaceable class="parameter">format</replaceable></option></term>
<listitem>
<para>
Selects the format for the output. <replaceable>format</replaceable>
can be one of the following:
<variablelist>
<varlistentry>
<term><literal>p</literal></term>
<term><literal>plain</literal></term>
<listitem>
<para>
Write the output as plain files, with the same layout as the
current data directory and tablespaces. When the cluster has
no additional tablespaces, the whole database will be placed in
the target directory. If the cluster contains additional
tablespaces, the main data directory will be placed in the
target directory, but all other tablespaces will be placed
in the same absolute path as they have on the server.
</para>
<para>
This is the default format.
</para>
</listitem>
</varlistentry>
<varlistentry>
<term><literal>t</literal></term>
<term><literal>tar</literal></term>
<listitem>
<para>
Write the output as tar files in the target directory. The main
data directory will be written to a file named
<filename>base.tar</filename>, and all other tablespaces will
be named after the tablespace oid.
</para>
<para>
If the value <literal>-</literal> (dash) is specified as
target directory, the tar contents will be written to
standard output, suitable for piping to for example
<productname>gzip</productname>. This is only possible if
the cluster has no additional tablespaces.
</para>
</listitem>
</varlistentry>
</variablelist>
</para>
</listitem>
</varlistentry>
<varlistentry>
<term><option>-Z <replaceable class="parameter">level</replaceable></option></term>
<term><option>--compress=<replaceable class="parameter">level</replaceable></option></term>
<listitem>
<para>
Enables gzip compression of tar file output. Compression is only
available when generating tar files, and is not available when sending
output to standard output.
</para>
</listitem>
</varlistentry>
</variablelist>
</para>
<para>
The following command-line options control the generation of the
backup and the running of the program.
<variablelist>
<varlistentry>
<term><option>-c <replaceable class="parameter">fast|spread</replaceable></option></term>
<term><option>--checkpoint <replaceable class="parameter">fast|spread</replaceable></option></term>
<listitem>
<para>
Sets checkpoint mode to fast or spread (default).
</para>
</listitem>
</varlistentry>
<varlistentry>
<term><option>-l <replaceable class="parameter">label</replaceable></option></term>
<term><option>--label=<replaceable class="parameter">label</replaceable></option></term>
<listitem>
<para>
Sets the label for the backup. If none is specified, a default value of
<literal>pg_basebackup base backup</literal> will be used.
</para>
</listitem>
</varlistentry>
<varlistentry>
<term><option>-P</option></term>
<term><option>--progress</option></term>
<listitem>
<para>
Enables progress reporting. Turning this on will deliver an approximate
progress report during the backup. Since the database may change during
the backup, this is only an approximation and may not end at exactly
<literal>100%</literal>.
</para>
<para>
When this is enabled, the backup will start by enumerating the size of
the entire database, and then go back and send the actual contents.
This may make the backup take slightly longer, and in particular it
will take longer before the first data is sent.
</para>
</listitem>
</varlistentry>
<varlistentry>
<term><option>-v</option></term>
<term><option>--verbose</option></term>
<listitem>
<para>
Enables verbose mode. Will output some extra steps during startup and
shutdown, as well as show the exact filename that is currently being
processed if progress reporting is also enabled.
</para>
</listitem>
</varlistentry>
</variablelist>
</para>
<para>
The following command-line options control the database connection parameters.
<variablelist>
<varlistentry>
<term><option>-h <replaceable class="parameter">host</replaceable></option></term>
<term><option>--host=<replaceable class="parameter">host</replaceable></option></term>
<listitem>
<para>
Specifies the host name of the machine on which the server is
running. If the value begins with a slash, it is used as the
directory for the Unix domain socket. The default is taken
from the <envar>PGHOST</envar> environment variable, if set,
else a Unix domain socket connection is attempted.
</para>
</listitem>
</varlistentry>
<varlistentry>
<term><option>-p <replaceable class="parameter">port</replaceable></option></term>
<term><option>--port=<replaceable class="parameter">port</replaceable></option></term>
<listitem>
<para>
Specifies the TCP port or local Unix domain socket file
extension on which the server is listening for connections.
Defaults to the <envar>PGPORT</envar> environment variable, if
set, or a compiled-in default.
</para>
</listitem>
</varlistentry>
<varlistentry>
<term><option>-U <replaceable>username</replaceable></option></term>
<term><option>--username=<replaceable class="parameter">username</replaceable></option></term>
<listitem>
<para>
User name to connect as.
</para>
</listitem>
</varlistentry>
<varlistentry>
<term><option>-w</></term>
<term><option>--no-password</></term>
<listitem>
<para>
Never issue a password prompt. If the server requires
password authentication and a password is not available by
other means such as a <filename>.pgpass</filename> file, the
connection attempt will fail. This option can be useful in
batch jobs and scripts where no user is present to enter a
password.
</para>
</listitem>
</varlistentry>
<varlistentry>
<term><option>-W</option></term>
<term><option>--password</option></term>
<listitem>
<para>
Force <application>pg_basebackup</application> to prompt for a
password before connecting to a database.
</para>
<para>
This option is never essential, since
<application>pg_bsaebackup</application> will automatically prompt
for a password if the server demands password authentication.
However, <application>pg_basebackup</application> will waste a
connection attempt finding out that the server wants a password.
In some cases it is worth typing <option>-W</> to avoid the extra
connection attempt.
</para>
</listitem>
</varlistentry>
</variablelist>
</para>
<para>
Other, less commonly used, parameters are also available:
<variablelist>
<varlistentry>
<term><option>-V</></term>
<term><option>--version</></term>
<listitem>
<para>
Print the <application>pg_basebackup</application> version and exit.
</para>
</listitem>
</varlistentry>
<varlistentry>
<term><option>-?</></term>
<term><option>--help</></term>
<listitem>
<para>
Show help about <application>pg_basebackup</application> command line
arguments, and exit.
</para>
</listitem>
</varlistentry>
</variablelist>
</para>
</refsect1>
<refsect1>
<title>Environment</title>
<para>
This utility, like most other <productname>PostgreSQL</> utilities,
uses the environment variables supported by <application>libpq</>
(see <xref linkend="libpq-envars">).
</para>
</refsect1>
<refsect1>
<title>Notes</title>
<para>
The backup will include all files in the data directory and tablespaces,
including the configuration files and any additional files placed in the
directory by third parties. Only regular files and directories are allowed
in the data directory, no symbolic links or special device files.
</para>
<para>
The way <productname>PostgreSQL</productname> manages tablespaces, the path
for all additional tablespaces must be identical whenever a backup is
restored. The main data directory, however, is relocatable to any location.
</para>
</refsect1>
<refsect1>
<title>Examples</title>
<para>
To create a base backup of the server at <literal>mydbserver</literal>
and store it in the local directory
<filename>/usr/local/pgsql/data</filename>:
<screen>
<prompt>$</prompt> <userinput>pg_basebackup -h mydbserver -D /usr/local/pgsql/data</userinput>
</screen>
</para>
<para>
To create a backup of the local server with one maximum compressed
tar file for each tablespace, and store it in the directory
<filename>backup</filename>, showing a progress report while running:
<screen>
<prompt>$</prompt> <userinput>pg_basebackup -D backup -Ft -Z9 -P</userinput>
</screen>
</para>
<para>
To create a backup of a single-tablespace local database and compress
this with <productname>bzip2</productname>:
<screen>
<prompt>$</prompt> <userinput>pg_basebackup -D - -Ft | bzip2 > backup.tar.bz2</userinput>
</screen>
(this command will fail if there are multiple tablespaces in the
database)
</para>
</refsect1>
<refsect1>
<title>See Also</title>
<simplelist type="inline">
<member><xref linkend="APP-PGDUMP"></member>
</simplelist>
</refsect1>
</refentry>

1
doc/src/sgml/reference.sgml
View File

@ -202,6 +202,7 @@
&droplang;
&dropuser;
&ecpgRef;
&pgBasebackup;
&pgConfig;
&pgDump;
&pgDumpall;

19
src/backend/replication/basebackup.c
View File

@ -40,7 +40,7 @@ static void send_int8_string(StringInfoData *buf, int64 intval);
static void SendBackupHeader(List *tablespaces);
static void SendBackupDirectory(char *location, char *spcoid);
static void base_backup_cleanup(int code, Datum arg);
static void perform_base_backup(const char *backup_label, bool progress, DIR *tblspcdir);
static void perform_base_backup(const char *backup_label, bool progress, DIR *tblspcdir, bool fastcheckpoint);
typedef struct
{
@ -67,9 +67,9 @@ base_backup_cleanup(int code, Datum arg)
* clobbered by longjmp" from stupider versions of gcc.
*/
static void
perform_base_backup(const char *backup_label, bool progress, DIR *tblspcdir)
perform_base_backup(const char *backup_label, bool progress, DIR *tblspcdir, bool fastcheckpoint)
{
do_pg_start_backup(backup_label, true);
do_pg_start_backup(backup_label, fastcheckpoint);
PG_ENSURE_ERROR_CLEANUP(base_backup_cleanup, (Datum) 0);
{
@ -135,7 +135,7 @@ perform_base_backup(const char *backup_label, bool progress, DIR *tblspcdir)
* pg_stop_backup() for the user.
*/
void
SendBaseBackup(const char *backup_label, bool progress)
SendBaseBackup(const char *backup_label, bool progress, bool fastcheckpoint)
{
DIR *dir;
MemoryContext backup_context;
@ -168,7 +168,7 @@ SendBaseBackup(const char *backup_label, bool progress)
ereport(ERROR,
(errmsg("unable to open directory pg_tblspc: %m")));
perform_base_backup(backup_label, progress, dir);
perform_base_backup(backup_label, progress, dir, fastcheckpoint);
FreeDir(dir);
@ -333,7 +333,16 @@ sendDir(char *path, int basepathlen, bool sizeonly)
if (strcmp(pathbuf, "./pg_xlog") == 0)
{
if (!sizeonly)
{
/* If pg_xlog is a symlink, write it as a directory anyway */
#ifndef WIN32
if (S_ISLNK(statbuf.st_mode))
#else
if (pgwin32_is_junction(pathbuf))
#endif
statbuf.st_mode = S_IFDIR | S_IRWXU;
_tarWriteHeader(pathbuf + basepathlen + 1, NULL, &statbuf);
}
size += 512; /* Size of the header just added */
continue; /* don't recurse into pg_xlog */
}

11
src/backend/replication/repl_gram.y
View File

@ -66,11 +66,12 @@ Node *replication_parse_result;
%token K_IDENTIFY_SYSTEM
%token K_LABEL
%token K_PROGRESS
%token K_FAST
%token K_START_REPLICATION
%type <node> command
%type <node> base_backup start_replication identify_system
%type <boolval> opt_progress
%type <boolval> opt_progress opt_fast
%type <str> opt_label
%%
@ -102,15 +103,16 @@ identify_system:
;
/*
* BASE_BACKUP [LABEL <label>] [PROGRESS]
* BASE_BACKUP [LABEL <label>] [PROGRESS] [FAST]
*/
base_backup:
K_BASE_BACKUP opt_label opt_progress
K_BASE_BACKUP opt_label opt_progress opt_fast
{
BaseBackupCmd *cmd = (BaseBackupCmd *) makeNode(BaseBackupCmd);
cmd->label = $2;
cmd->progress = $3;
cmd->fastcheckpoint = $4;
$$ = (Node *) cmd;
}
@ -123,6 +125,9 @@ opt_label: K_LABEL SCONST { $$ = $2; }
opt_progress: K_PROGRESS { $$ = true; }
| /* EMPTY */ { $$ = false; }
;
opt_fast: K_FAST { $$ = true; }
| /* EMPTY */ { $$ = false; }
;
/*
* START_REPLICATION %X/%X

1
src/backend/replication/repl_scanner.l
View File

@ -57,6 +57,7 @@ quotestop {quote}
%%
BASE_BACKUP { return K_BASE_BACKUP; }
FAST { return K_FAST; }
IDENTIFY_SYSTEM { return K_IDENTIFY_SYSTEM; }
LABEL { return K_LABEL; }
PROGRESS { return K_PROGRESS; }

2
src/backend/replication/walsender.c
View File

@ -402,7 +402,7 @@ HandleReplicationCommand(const char *cmd_string)
{
BaseBackupCmd *cmd = (BaseBackupCmd *) cmd_node;
SendBaseBackup(cmd->label, cmd->progress);
SendBaseBackup(cmd->label, cmd->progress, cmd->fastcheckpoint);
/* Send CommandComplete and ReadyForQuery messages */
EndCommand("SELECT", DestRemote);

2
src/bin/Makefile
View File

@ -14,7 +14,7 @@ top_builddir = ../..
include $(top_builddir)/src/Makefile.global
SUBDIRS = initdb pg_ctl pg_dump \
psql scripts pg_config pg_controldata pg_resetxlog
psql scripts pg_config pg_controldata pg_resetxlog pg_basebackup
ifeq ($(PORTNAME), win32)
SUBDIRS+=pgevent
endif

38
src/bin/pg_basebackup/Makefile Normal file
View File

@ -0,0 +1,38 @@
#-------------------------------------------------------------------------
#
# Makefile for src/bin/pg_basebackup
#
# Portions Copyright (c) 1996-2011, PostgreSQL Global Development Group
# Portions Copyright (c) 1994, Regents of the University of California
#
# src/bin/pg_basebackup/Makefile
#
#-------------------------------------------------------------------------
PGFILEDESC = "pg_basebackup - takes a streaming base backup of a PostgreSQL instance"
PGAPPICON=win32
subdir = src/bin/pg_basebackup
top_builddir = ../../..
include $(top_builddir)/src/Makefile.global
override CPPFLAGS := -I$(libpq_srcdir) $(CPPFLAGS)
OBJS= pg_basebackup.o $(WIN32RES)
all: pg_basebackup
pg_basebackup: $(OBJS) | submake-libpq submake-libpgport
$(CC) $(CFLAGS) $(OBJS) $(libpq_pgport) $(LDFLAGS) $(LDFLAGS_EX) $(LIBS) -o $@$(X)
install: all installdirs
$(INSTALL_PROGRAM) pg_basebackup$(X) '$(DESTDIR)$(bindir)/pg_basebackup$(X)'
installdirs:
$(MKDIR_P) '$(DESTDIR)$(bindir)'
uninstall:
rm -f '$(DESTDIR)$(bindir)/pg_basebackup$(X)'
clean distclean maintainer-clean:
rm -f pg_basebackup$(X) $(OBJS)

5
src/bin/pg_basebackup/nls.mk Normal file
View File

@ -0,0 +1,5 @@
# src/bin/pg_basebackup/nls.mk
CATALOG_NAME := pg_basebackup
AVAIL_LANGUAGES :=
GETTEXT_FILES := pg_basebackup.c
GETTEXT_TRIGGERS:= _

1035
src/bin/pg_basebackup/pg_basebackup.c Normal file
View File

File diff suppressed because it is too large Load Diff

2
src/include/replication/basebackup.h
View File

@ -12,6 +12,6 @@
#ifndef _BASEBACKUP_H
#define _BASEBACKUP_H
extern void SendBaseBackup(const char *backup_label, bool progress);
extern void SendBaseBackup(const char *backup_label, bool progress, bool fastcheckpoint);
#endif /* _BASEBACKUP_H */

1
src/include/replication/replnodes.h
View File

@ -47,6 +47,7 @@ typedef struct BaseBackupCmd
NodeTag type;
char *label;
bool progress;
bool fastcheckpoint;
} BaseBackupCmd;

2
src/tools/msvc/Mkvcbuild.pm
View File

@ -275,6 +275,8 @@ sub mkvcbuild
$initdb->AddLibrary('wsock32.lib');
$initdb->AddLibrary('ws2_32.lib');
my $pgbasebackup = AddSimpleFrontend('pg_basebackup', 1);
my $pgconfig = AddSimpleFrontend('pg_config');
my $pgcontrol = AddSimpleFrontend('pg_controldata');