41898 Commits

Author SHA1 Message Date
Noah Misch
13503eb590 Diagnose !indisvalid in more SQL functions.
pgstatindex failed with ERRCODE_DATA_CORRUPTED, of the "can't-happen"
class XX.  The other functions succeeded on an empty index; they might
have malfunctioned if the failed index build left torn I/O or other
complex state.  Report an ERROR in statistics functions pgstatindex,
pgstatginindex, pgstathashindex, and pgstattuple.  Report DEBUG1 and
skip all index I/O in maintenance functions brin_desummarize_range,
brin_summarize_new_values, brin_summarize_range, and
gin_clean_pending_list.  Back-patch to v11 (all supported versions).

Discussion: https://postgr.es/m/20231001195309.a3@google.com
2023-10-30 14:46:05 -07:00
Bruce Momjian
56b30e266e pgindent run to fix commits de64268561 and 5ae2087202a
Reported-by: Michael Paquier

Discussion: https://postgr.es/m/ZT9YH7-TTx27V3yW@paquier.xyz

Backpatch-through: master
2023-10-30 14:52:35 -04:00
Peter Eisentraut
0c60e8ba80 Fill in more of ObjectProperty
Fill in .objtype field where an appropriate value exists.

These cases are currently not used (see also comments at
get_object_type()), but we might as well fill in what's possible in
case additional uses arise.

Discussion: https://www.postgresql.org/message-id/flat/75ae5875-3abc-dafc-8aec-73247ed41cde@eisentraut.org
2023-10-30 06:08:53 -04:00
Michael Paquier
5b2147d9fc Expand regression tests for pg_stat_reset_shared()
This commit adds coverage for the stats reset of recovery_prefetch, and
for the case where an invalid value is given in input of the function.

Author: Bharath Rupireddy
Discussion: https://postgr.es/m/CALj2ACW9Uk7x61oSix9qK0xR4Jhy3cgg6pobQ-Q3GNsUbFrn8A@mail.gmail.com
2023-10-30 17:19:24 +09:00
Michael Paquier
dc5bd38894 Delay recovery mode LOG after reading backup_label and/or checkpoint record
When beginning recovery, a LOG is displayed by the startup process to
show which recovery mode will be used depending on the .signal file(s)
set in the data folder, like "standby mode", recovery up to a given
target type and value, or archive recovery.

A different patch is under discussion to simplify the startup code by
requiring the presence of recovery.signal and/or standby.signal when a
backup_label file is read.  Delaying a bit this LOG ensures that the
correct recovery mode would be reported, and putting it at this position
does not make it lose its value.

While on it, this commit adds a few comments documenting a bit more the
initial recovery steps and their dependencies, and fixes an incorrect
comment format.  This introduces no behavior changes.

Extracted from a larger patch by me.

Reviewed-by: David Steele, Bowen Shi
Discussion: https://postgr.es/m/ZArVOMifjzE7f8W7@paquier.xyz
2023-10-30 15:28:20 +09:00
Michael Paquier
1ffdc03c21 Mention standby.signal in FATALs for checkpoint record missing at recovery
When beginning recovery from a base backup by reading a backup_label
file, it may be possible that no checkpoint record is available
depending on the method used when the case backup was taken, which would
prevent recovery from beginning.  In this case, the FATAL messages
issued, initially added by c900c15269f0f, mentioned recovery.signal as
an option to do recovery but not standby.signal.  Let's add it as an
available option, for clarity.

Per suggestion from Bowen Shi, extracted from a larger patch by me.

Author: Michael Paquier
Discussion: https://postgr.es/m/CAM_vCudkSjr7NsNKSdjwtfAm9dbzepY6beZ5DP177POKy8=2aw@mail.gmail.com
2023-10-30 13:56:02 +09:00
Michael Paquier
96f052613f Introduce pg_stat_checkpointer
Historically, the statistics of the checkpointer have been always part
of pg_stat_bgwriter.  This commit removes a few columns from
pg_stat_bgwriter, and introduces pg_stat_checkpointer with equivalent,
renamed columns (plus a new one for the reset timestamp):
- checkpoints_timed -> num_timed
- checkpoints_req -> num_requested
- checkpoint_write_time -> write_time
- checkpoint_sync_time -> sync_time
- buffers_checkpoint -> buffers_written

The fields of PgStat_CheckpointerStats and its SQL functions are renamed
to match with the new field names, for consistency.  Note that
background writer and checkpointer have been split into two different
processes in commits 806a2aee3791 and bf405ba8e460.  The pgstat
structures were already split, making this change straight-forward.

Bump catalog version.

Author: Bharath Rupireddy
Reviewed-by: Bertrand Drouvot, Andres Freund, Michael Paquier
Discussion: https://postgr.es/m/CALj2ACVxX2ii=66RypXRweZe2EsBRiPMj0aHfRfHUeXJcC7kHg@mail.gmail.com
2023-10-30 09:47:16 +09:00
Michael Paquier
bf01e1ba96 Refactor some code related to transaction-level statistics for relations
This commit refactors find_tabstat_entry() so as transaction counters
for inserted, updated and deleted tuples are included in the result
returned.   If a shared entry is found for a relation, its result is now
a copy of the PgStat_TableStatus entry retrieved from shared memory.
This idea has been proposed by Andres Freund.

While on it, the following SQL functions, used in system views, are
refactored with macros, in the same spirit as 83a1a1b56645, reducing the
amount of code:
- pg_stat_get_xact_tuples_deleted()
- pg_stat_get_xact_tuples_inserted()
- pg_stat_get_xact_tuples_updated()

There is now only one caller of find_tabstat_entry() in the tree.

Author: Bertrand Drouvot
Discussion: https://postgr.es/m/b9e1f543-ee93-8168-d530-d961708ad9d3@gmail.com
2023-10-30 08:23:39 +09:00
Alexander Korotkov
06be01eb26 Fix instable 006_login_trigger.pl test
Handling of login trigger FATAL error could cause a timing-dependant panic of
IPC::Run.  This commit excludes checks which involves handling of such errors.

Reported-by: Tom Lane
Discussion: https://postgr.es/m/2268825.1698618066%40sss.pgh.pa.us
2023-10-30 01:28:18 +03:00
Tom Lane
5ba4cc3090 Teach pg_dump about the new pg_subscription.subrunasowner option.
Among numerous other oversights, commit 482675987 neglected to fix
pg_dump to dump this new subscription option.  Since the new default
is "false" while the previous behavior corresponds to "true", this
would cause legacy subscriptions to silently change behavior during
dump/reload or pg_upgrade.  That seems like a bad idea.  Even if it
was intended, failing to preserve the option once set in a new
installation is certainly not OK.

While here, reorder associated stanzas in pg_dump to match the
field order in pg_subscription, in hopes of reducing the impression
that all this code was written with the aid of a dartboard.

Back-patch to v16 where this new field was added.

Philip Warner (cosmetic tweaks by me)

Discussion: https://postgr.es/m/20231027042539.01A3A220F0A@thebes.rime.com.au
2023-10-29 12:56:35 -04:00
Dean Rasheed
b2d55447a5 Guard against overflow in make_interval().
The original code did very little to guard against integer or floating
point overflow when computing the interval's fields.  Detect any such
overflows and error out, rather than silently returning bogus results.

Joseph Koshakow, reviewed by Ashutosh Bapat and me.

Discussion: https://postgr.es/m/CAAvxfHcm1TPwH_zaGWuFoL8pZBestbRZTU6Z%3D-RvAdSXTPbKfg%40mail.gmail.com
2023-10-29 15:51:53 +00:00
Tom Lane
237f8765df Fix intra-query memory leak when a SRF returns zero rows.
When looping around after finding that the set-returning function
returned zero rows for the current input tuple, ExecProjectSet
neglected to reset either of the two memory contexts it's
responsible for cleaning out.  Typically this wouldn't cause much
problem, because once the SRF does return at least one row, the
contexts would get reset on the next call.  However, if the SRF
returns no rows for many input tuples in succession, quite a lot
of memory could be transiently consumed.

To fix, make sure we reset both contexts while looping around.

Per bug #18172 from Sergei Kornilov.  Back-patch to all supported
branches.

Discussion: https://postgr.es/m/18172-9b8c5fc1d676ded3@postgresql.org
2023-10-28 14:05:01 -04:00
Bruce Momjian
de64268561 doc: comment wording improvement
Discussion: https://postgr.es/m/CAEG8a3L7UoZXH1VmzpV-VDkex2kt68nWKuW1WiohoT=RrzYKWA@mail.gmail.com

Author: Junwang Zhao

Backpatch-through: master
2023-10-28 12:59:00 -04:00
Tom Lane
5fd3e06f6a Remove PHOT from our default timezone abbreviations list.
Debian recently decided to split out a bunch of "obsolete" timezone
names into a new tzdata-legacy package, which isn't installed by
default.  One of these zone names is Pacific/Enderbury, and that
breaks our regression tests (on --with-system-tzdata builds)
because our default timezone abbreviations list defines PHOT as
Pacific/Enderbury.

Pacific/Enderbury got renamed to Pacific/Kanton in tzdata 2021b,
so that in distros that still have this entry it's just a symlink
to Pacific/Kanton anyway.  So one answer would be to redefine PHOT
as Pacific/Kanton.  However, then things would fail if the
installed tzdata predates 2021b, which is recent enough that that
seems like a real problem.

Instead, let's just remove PHOT from the default list.  That seems
likely to affect nobody in the real world, because (a) it was an
abbreviation that the tzdb crew made up in the first place, with
no evidence of real-world usage, and (b) the total human population
of the Phoenix Islands is less than two dozen persons, per Wikipedia.
If anyone does use this zone abbreviation they can easily put it back
via a custom abbreviations file.

We'll keep PHOT in the Pacific.txt reference file, but change it
to Pacific/Kanton there, as that definition seems more likely to
be useful to future readers of that file.

Per report from Victor Wagner.  Back-patch to all supported
branches.

Discussion: https://postgr.es/m/20231027152049.4b5c8044@wagner.wagner.home
2023-10-28 11:54:40 -04:00
Tom Lane
4b9fbd6be4 Allow relkind 'I' in type_sanity test.
This is cosmetic at present, since no partitioned indexes have
been created yet at the stage of the regression tests when
type_sanity runs.  But it's confusing that this list of allowed
values doesn't match reality, so fix it.

Noted by Jian He.

Discussion: https://postgr.es/m/CACJufxHhr58q-YoBeqnY5P-u8Xe2X5sMtsr3LYE1nQDwdqjB4w@mail.gmail.com
2023-10-27 21:50:44 -04:00
Bruce Momjian
12cf3ac7f3 doc Improve C GUC-related comments
Discussion: https://postgr.es/m/CAEG8a3LZHTR5S+OPZCbZvECwsqdbx=pBRFZZyDjKaAtgoALOQQ@mail.gmail.com

Author: Junwang Zhao

Backpatch-through: master
2023-10-27 19:05:25 -04:00
Alexander Korotkov
5ae2087202 Teach contrib/amcheck to check the unique constraint violation
Add the 'checkunique' argument to bt_index_check() and bt_index_parent_check().
When the flag is specified the procedures will check the unique constraint
violation for unique indexes.  Only one heap entry for all equal keys in
the index should be visible (including posting list entries).  Report an error
otherwise.

pg_amcheck called with the --checkunique option will do the same check for all
the indexes it checks.

Author: Anastasia Lubennikova <lubennikovaav@gmail.com>
Author: Pavel Borisov <pashkin.elfe@gmail.com>
Author: Maxim Orlov <orlovmg@gmail.com>
Reviewed-by: Mark Dilger <mark.dilger@enterprisedb.com>
Reviewed-by: Zhihong Yu <zyu@yugabyte.com>
Reviewed-by: Peter Geoghegan <pg@bowt.ie>
Reviewed-by: Aleksander Alekseev <aleksander@timescale.com>
Discussion: https://postgr.es/m/CALT9ZEHRn5xAM5boga0qnrCmPV52bScEK2QnQ1HmUZDD301JEg%40mail.gmail.com
2023-10-28 00:21:23 +03:00
Alexander Korotkov
8b62b441ff Split event_trigger_login test from event_trigger test
That allows to still run event_trigger test in parallel with oidjoins test,
and save ~50ms of tests runtime.

Discussion: https://postgr.es/m/202310271047.mnwkql6nhbwi%40alvherre.pgsql
Author: Alvaro Herrera, Alexander Korotkov
2023-10-27 23:50:07 +03:00
Tomas Vondra
c6cf6d353c Fix minmax-multi distance for extreme interval values
When calculating distance for interval values, the code mostly mimicked
interval_mi, i.e. it built a new interval value for the difference.
That however does not work for sufficiently distant interval values,
when the difference overflows the interval range.

Instead, we can calculate the distance directly, without constructing
the intermediate (and unnecessary) interval value.

Backpatch to 14, where minmax-multi indexes were introduced.

Reported-by: Dean Rasheed
Reviewed-by: Ashutosh Bapat, Dean Rasheed
Backpatch-through: 14
Discussion: https://postgr.es/m/eef0ea8c-4aaa-8d0d-027f-58b1f35dd170@enterprisedb.com
2023-10-27 18:15:37 +02:00
Tomas Vondra
8da86d62a1 Fix minmax-multi on infinite date/timestamp values
Make sure that infinite values in date/timestamp columns are treated as
if in infinite distance. Infinite values should not be merged with other
values, leaving them as outliers. The code however returned distance 0
in this case, so that infinite values were merged first. While this does
not break the index (i.e. it still produces correct query results), it
may make it much less efficient.

We don't need explicit handling of infinite date/timestamp values when
calculating distances, because those values are represented as extreme
but regular values (e.g. INT64_MIN/MAX for the timestamp type).

We don't need an exact distance, just a value that is much larger than
distanced between regular values. With the added cast to double values,
we can simply subtract the values.

The regression test queries a value in the "gap" and checks the range
was properly eliminated by the BRIN index.

This only affects minmax-multi indexes on timestamp/date columns with
infinite values, which is not very common in practice. The affected
indexes may need to be rebuilt.

Backpatch to 14, where minmax-multi indexes were introduced.

Reported-by: Ashutosh Bapat
Reviewed-by: Ashutosh Bapat, Dean Rasheed
Backpatch-through: 14
Discussion: https://postgr.es/m/eef0ea8c-4aaa-8d0d-027f-58b1f35dd170@enterprisedb.com
2023-10-27 18:15:37 +02:00
Tomas Vondra
394d517314 Fix calculation in brin_minmax_multi_distance_date
When calculating the distance between date values, make sure to subtract
them in the right order, i.e. (larger - smaller).

The distance is used to determine which values to merge, and is expected
to be a positive value. The code unfortunately did the subtraction in
the opposite order, i.e. (smaller - larger), thus producing negative
values and merging values the most distant values first.

The resulting index is correct (i.e. produces correct results), but may
be significantly less efficient. This affects all minmax-multi indexes
on date columns.

Backpatch to 14, where minmax-multi indexes were introduced.

Reported-by: Ashutosh Bapat
Reviewed-by: Ashutosh Bapat, Dean Rasheed
Backpatch-through: 14
Discussion: https://postgr.es/m/eef0ea8c-4aaa-8d0d-027f-58b1f35dd170@enterprisedb.com
2023-10-27 18:15:37 +02:00
Tomas Vondra
b5489b75c6 Fix overflow when calculating timestamp distance in BRIN
When calculating distances for timestamp values for BRIN minmax-multi
indexes, we need to be careful about overflows for extreme values. If
the value overflows into a negative value, the index may be inefficient.

The new regression test checks this for the timestamp type by adding a
table with enough values to force range compaction/merging. The values
are close to min/max, which means a risk of overflow.

Fixed by converting the int64 values to double first, before calculating
the distance. This prevents the overflow. We may lose some precision, of
course, but that's good enough. In the worst case we build a slightly
less efficient index, but for large distances this won't matter.

This only affects minmax-multi indexes on timestamp columns, with ranges
containing values sufficiently distant to cause an overflow. That seems
like a fairly rare case in practice.

Backpatch to 14, where minmax-multi indexes were introduced.

Reported-by: Ashutosh Bapat
Reviewed-by: Ashutosh Bapat, Dean Rasheed
Backpatch-through: 14
Discussion: https://postgr.es/m/eef0ea8c-4aaa-8d0d-027f-58b1f35dd170@enterprisedb.com
2023-10-27 18:15:37 +02:00
Alexander Korotkov
83510534d5 Adjust parallel_schedule with event triggers on authenticated login
Event triggers on authenticated login could catch the connection of the
concurrent test.  In order to prevent this side effect we need to run
the event_trigger test alone.

Discussion: https://postgr.es/m/flat/CAMEv5_sS5G7K1PCV2oBx5+1NF1ZneJ6C5Z_xV_SWsZMukdFZiA@mail.gmail.com
Author: Mikhail A. Gribkov
Reviewed-by: Aleksander Alekseev
2023-10-27 11:34:30 +03:00
Amit Kapila
8949b978ff Commit b195e6d482 forgot to update meson.build.
Author: Hayato Kuroda
Discussion: http://postgr.es/m/TYCPR01MB58701BA358D80073ED1BD528F5DCA@TYCPR01MB5870.jpnprd01.prod.outlook.com
2023-10-27 13:19:49 +05:30
Amit Kapila
b195e6d482 Use shorter file names in the upgrade logical slots test.
The longer file names exceeded the Windows path limit on buildfarm animal
fairywren.

Diagnosed-by: Hou Zhijie
Author: Hayato Kuroda
Reviewed-by: Bharath Rupireddy
Discussion: http://postgr.es/m/OS0PR01MB57160DF709ACD02248DB830C94DDA@OS0PR01MB5716.jpnprd01.prod.outlook.com
2023-10-27 11:09:45 +05:30
Alexander Korotkov
2b26a69455 Make UniqueRelInfo a node
d3d55ce571 changed RelOptInfo.unique_for_rels from the list of Relid sets to
the list of UniqueRelInfo's.  But it didn't make UniqueRelInfo a node.
This commit makes UniqueRelInfo a node.  Also this commit revises some
comments related to RelOptInfo.unique_for_rels.

Reported-by: Tom Lane
Discussion: https://postgr.es/m/flat/1189851.1698340331%40sss.pgh.pa.us
2023-10-27 05:45:16 +03:00
Michael Paquier
74604a37f2 Remove buffers_backend and buffers_backend_fsync from pg_stat_checkpointer
Two attributes related to checkpointer statistics are removed in this
commit:
- buffers_backend, that counts the number of buffers written directly by
a backend.
- buffers_backend_fsync, that counts the number of times a backend had
to do fsync() by its own.

These are actually not checkpointer properties but backend properties.
Also, pg_stat_io provides a more accurate and equivalent report of these
numbers, by tracking all the I/O stats related to backends, including
writes and fsyncs, so storing them in pg_stat_checkpointer was
redundant.

Thanks also to Robert Haas and Amit Kapila for their input.

Bump catalog version.

Author: Bharath Rupireddy
Reviewed-by: Bertrand Drouvot, Andres Freund
Discussion: https://postgr.es/m/20230210004604.mcszbscsqs3bc5nx@awork3.anarazel.de
2023-10-27 11:16:39 +09:00
David Rowley
0c882a2988 Optimize various aggregate deserialization functions, take 2
f0efa5aec added initReadOnlyStringInfo to allow a StringInfo to be
initialized from an existing buffer and also relaxed the requirement
that a StringInfo's buffer must be NUL terminated at data[len].  Now
that we have that, there's no need for these aggregate deserial
functions to use appendBinaryStringInfo() as that rather wastefully
palloc'd a new buffer and memcpy'd in the bytea's buffer.  Instead, we can
just use the bytea's buffer and point the StringInfo directly to that
using the new initializer function.

In Amdahl's law, this speeds up the serial portion of parallel
aggregates and makes sum(numeric), avg(numeric), var_pop(numeric),
var_samp(numeric), variance(numeric), stddev_pop(numeric),
stddev_samp(numeric), stddev(numeric), array_agg(anyarray),
string_agg(text) and string_agg(bytea) scale better in parallel queries.

Author: David Rowley
Discussion: https://postgr.es/m/CAApHDvr%3De-YOigriSHHm324a40HPqcUhSp6pWWgjz5WwegR%3DcQ%40mail.gmail.com
2023-10-27 10:41:55 +13:00
Amit Langote
1f06b7fc6e Avoid compiler warning in non-assert builds
After 01575ad788e3, expand_single_inheritance_child()'s parentOID
variable is read only in an Assert, provoking a compiler warning in
non-assert builds.  Fix that by marking the variable with
PG_USED_FOR_ASSERTS_ONLY.

Per report and suggestion from David Rowley

Discussion: https://postgr.es/m/CAApHDvpjA_8Wxu4DCTRVAvPxC9atwMe6N%2ByvrcGsgb7mrfdpJA%40mail.gmail.com
2023-10-26 17:32:38 +09:00
Peter Eisentraut
611806cd72 Add trailing commas to enum definitions
Since C99, there can be a trailing comma after the last value in an
enum definition.  A lot of new code has been introducing this style on
the fly.  Some new patches are now taking an inconsistent approach to
this.  Some add the last comma on the fly if they add a new last
value, some are trying to preserve the existing style in each place,
some are even dropping the last comma if there was one.  We could
nudge this all in a consistent direction if we just add the trailing
commas everywhere once.

I omitted a few places where there was a fixed "last" value that will
always stay last.  I also skipped the header files of libpq and ecpg,
in case people want to use those with older compilers.  There were
also a small number of cases where the enum type wasn't used anywhere
(but the enum values were), which ended up confusing pgindent a bit,
so I left those alone.

Discussion: https://www.postgresql.org/message-id/flat/386f8c45-c8ac-4681-8add-e3b0852c1620%40eisentraut.org
2023-10-26 09:20:54 +02:00
David Rowley
f0efa5aec1 Introduce the concept of read-only StringInfos
There were various places in our codebase which conjured up a StringInfo
by manually assigning the StringInfo fields and setting the data field
to point to some existing buffer.  There wasn't much consistency here as
to what fields like maxlen got set to and in one location we didn't
correctly ensure that the buffer was correctly NUL terminated at len
bytes, as per what was documented as required in stringinfo.h

Here we introduce 2 new functions to initialize StringInfos.  One allows
callers to initialize a StringInfo passing along a buffer that is
already allocated by palloc.  Here the StringInfo code uses this buffer
directly rather than doing any memcpying into a new allocation.  Having
this as a function allows us to verify the buffer is correctly NUL
terminated.  StringInfos initialized this way can be appended to and
reset just like any other normal StringInfo.

The other new initialization function also accepts an existing buffer,
but the given buffer does not need to be a pointer to a palloc'd chunk.
This buffer could be a pointer pointing partway into some palloc'd chunk
or may not even be palloc'd at all.  StringInfos initialized this way
are deemed as "read-only".  This means that it's not possible to
append to them or reset them.

For the latter of the two new initialization functions mentioned above,
we relax the requirement that the data buffer must be NUL terminated.
Relaxing this requirement is convenient in a few places as it can save
us from having to allocate an entire new buffer just to add the NUL
terminator or save us from having to temporarily add a NUL only to have to
put the original char back again later.

Incompatibility note:

Here we also forego adding the NUL in a few places where it does not
seem to be required.  These locations are passing the given StringInfo
into a type's receive function.  It does not seem like any of our
built-in receive functions require this, but perhaps there's some UDT
out there in the wild which does require this.  It is likely worthy of
a mention in the release notes that a UDT's receive function mustn't rely
on the input StringInfo being NUL terminated.

Author: David Rowley
Reviewed-by: Tom Lane
Discussion: https://postgr.es/m/CAApHDvorfO3iBZ%3DxpiZvp3uHtJVLyFaPBSvcAhAq2HPLnaNSwQ%40mail.gmail.com
2023-10-26 16:31:48 +13:00
Amit Langote
01575ad788 Prevent duplicate RTEPermissionInfo for plain-inheritance parents
Currently, expand_single_inheritance_child() doesn't reset
perminfoindex in a plain-inheritance parent's child RTE, because
prior to 387f9ed0a0, the executor would use the first child RTE to
locate the parent's RTEPermissionInfo.  That in turn causes
add_rte_to_flat_rtable() to create an extra RTEPermissionInfo
belonging to the parent's child RTE with the same content as the one
belonging to the parent's original ("root") RTE.

In 387f9ed0a0, we changed things so that the executor can now use the
parent's "root" RTE for locating its RTEPermissionInfo instead of the
child RTE, so the latter's perminfoindex need not be set anymore, so
make it so.

Reported-by: Tom Lane
Discussion: https://postgr.es/m/839708.1698174464@sss.pgh.pa.us
Backpatch-through: 16
2023-10-26 11:53:56 +09:00
Amit Kapila
29d0a77fa6 Migrate logical slots to the new node during an upgrade.
While reading information from the old cluster, a list of logical
slots is fetched. At the later part of upgrading, pg_upgrade revisits the
list and restores slots by executing pg_create_logical_replication_slot()
on the new cluster. Migration of logical replication slots is only
supported when the old cluster is version 17.0 or later.

If the old node has invalid slots or slots with unconsumed WAL records,
the pg_upgrade fails. These checks are needed to prevent data loss.

The significant advantage of this commit is that it makes it easy to
continue logical replication even after upgrading the publisher node.
Previously, pg_upgrade allowed copying publications to a new node. With
this patch, adjusting the connection string to the new publisher will
cause the apply worker on the subscriber to connect to the new publisher
automatically. This enables seamless continuation of logical replication,
even after an upgrade.

Author: Hayato Kuroda, Hou Zhijie
Reviewed-by: Peter Smith, Bharath Rupireddy, Dilip Kumar, Vignesh C, Shlok Kyal
Discussion: http://postgr.es/m/TYAPR01MB58664C81887B3AF2EB6B16E3F5939@TYAPR01MB5866.jpnprd01.prod.outlook.com
Discussion: http://postgr.es/m/CAA4eK1+t7xYcfa0rEQw839=b2MzsfvYDPz3xbD+ZqOdP3zpKYg@mail.gmail.com
2023-10-26 07:06:55 +05:30
Nathan Bossart
fdeb6e6a74 Remove dead code in pg_ctl.c.
Missed in 39969e2a1e.

Author: David Steele
Discussion: https://postgr.es/m/0c742f0c-d663-419d-b5a7-4fe867f5566c%40pgmasters.net
2023-10-25 16:26:59 -05:00
Alexander Korotkov
9ba9c7074f Fix some regression tests for d3d55ce57136
Add missing (cost off) to explain.
2023-10-25 14:53:14 +03:00
Alexander Korotkov
d3d55ce571 Remove useless self-joins
The Self Join Elimination (SJE) feature removes an inner join of a plain table
to itself in the query tree if is proved that the join can be replaced with
a scan without impacting the query result.  Self join and inner relation are
replaced with the outer in query, equivalence classes, and planner info
structures. Also, inner restrictlist moves to the outer one with removing
duplicated clauses. Thus, this optimization reduces the length of the range
table list (this especially makes sense for partitioned relations), reduces
the number of restriction clauses === selectivity estimations, and potentially
can improve total planner prediction for the query.

The SJE proof is based on innerrel_is_unique machinery.

We can remove a self-join when for each outer row:
 1. At most one inner row matches the join clause.
 2. Each matched inner row must be (physically) the same row as the outer one.

In this patch we use the next approach to identify a self-join:
 1. Collect all merge-joinable join quals which look like a.x = b.x
 2. Add to the list above the baseretrictinfo of the inner table.
 3. Check innerrel_is_unique() for the qual list.  If it returns false, skip
    this pair of joining tables.
 4. Check uniqueness, proved by the baserestrictinfo clauses. To prove
    the possibility of self-join elimination inner and outer clauses must have
    an exact match.

The relation replacement procedure is not trivial and it is partly combined
with the one, used to remove useless left joins.  Tests, covering this feature,
were added to join.sql.  Some regression tests changed due to self-join removal
logic.

Discussion: https://postgr.es/m/flat/64486b0b-0404-e39e-322d-0801154901f3%40postgrespro.ru
Author: Andrey Lepikhov, Alexander Kuzmenkov
Reviewed-by: Tom Lane, Robert Haas, Andres Freund, Simon Riggs, Jonathan S. Katz
Reviewed-by: David Rowley, Thomas Munro, Konstantin Knizhnik, Heikki Linnakangas
Reviewed-by: Hywel Carver, Laurenz Albe, Ronan Dunklau, vignesh C, Zhihong Yu
Reviewed-by: Greg Stark, Jaime Casanova, Michał Kłeczek, Alena Rybakina
Reviewed-by: Alexander Korotkov
2023-10-25 12:59:16 +03:00
Daniel Gustafsson
8f0fd47fa3 Use snprintf instead of sprintf in pg_regress.
To avoid static analyzers sounding the alarm, move to using snprintf
instead of sprintf. This was an oversight in 66d6086cbcbfc8dee789a6.

Reported-by: Tom Lane <tgl@sss.pgh.pa.us>
Discussion: https://postgr.es/m/849588.1698179694@sss.pgh.pa.us
2023-10-25 10:53:11 +02:00
David Rowley
673a17e312 Add missing include dir and references to libpq for MSVC build
66d6086cb adjusted pg_regress to require this but forgot to adjust the
Visual Studio build script.
2023-10-25 15:37:37 +13:00
Daniel Gustafsson
66d6086cbc Speed up pg_regress server readiness testing.
Instead of connecting to the server with psql to check if it is ready
for running tests, this changes pg_regress to use PQPing which avoids
performing system() calls which are expensive on some platforms, like
Windows. The frequency of tests is also increased in order to connect
to the server faster.

This patch is part of a larger effort to make testing consume fewer
resources in order to be able to fit more tests into the available
CI system constraints.

Reviewed-by: Andres Freund <andres@anarazel.de>
Discussion: https://postgr.es/m/20230823192239.jxew5s3sjru63lio@awork3.anarazel.de
2023-10-24 21:37:35 +02:00
Tom Lane
387f9ed0a0 Fix problems when a plain-inheritance parent table is excluded.
When an UPDATE/DELETE/MERGE's target table is an old-style
inheritance tree, it's possible for the parent to get excluded
from the plan while some children are not.  (I believe this is
only possible if we can prove that a CHECK ... NO INHERIT
constraint on the parent contradicts the query WHERE clause,
so it's a very unusual case.)  In such a case, ExecInitModifyTable
mistakenly concluded that the first surviving child is the target
table, leading to at least two bugs:

1. The wrong table's statement-level triggers would get fired.

2. In v16 and up, it was possible to fail with "invalid perminfoindex
0 in RTE with relid nnnn" due to the child RTE not having permissions
data included in the query plan.  This was hard to reproduce reliably
because it did not occur unless the update triggered some non-HOT
index updates.

In v14 and up, this is easy to fix by defining ModifyTable.rootRelation
to be the parent RTE in plain inheritance as well as partitioned cases.

While the wrong-triggers bug also appears in older branches, the
relevant code in both the planner and executor is quite a bit
different, so it would take a good deal of effort to develop and
test a suitable patch.  Given the lack of field complaints about the
trigger issue, I'll desist for now.  (Patching v11 for this seems
unwise anyway, given that it will have no more releases after next
month.)

Per bug #18147 from Hans Buschmann.

Amit Langote and Tom Lane

Discussion: https://postgr.es/m/18147-6fc796538913ee88@postgresql.org
2023-10-24 14:48:33 -04:00
Jeff Davis
00d7fb5e2e Assert that buffers are marked dirty before XLogRegisterBuffer().
Enforce the rule from transam/README in XLogRegisterBuffer(), and
update callers to follow the rule.

Hash indexes sometimes register clean pages as a part of the locking
protocol, so provide a REGBUF_NO_CHANGE flag to support that use.

Discussion: https://postgr.es/m/c84114f8-c7f1-5b57-f85a-3adc31e1a904@iki.fi
Reviewed-by: Heikki Linnakangas
2023-10-23 17:17:46 -07:00
Michael Paquier
9972c7de1d Fix typos in wait_event.c
Noticed while working on a different patch.  Introduced in af720b4c50a1.
2023-10-24 08:05:29 +09:00
Robert Haas
5b36e8f078 Change struct tablespaceinfo's oid member from 'char *' to 'Oid'
This shouldn't change behavior except in the unusual case where
there are file in the tablespace directory that have entirely
numeric names but are nevertheless not possible names for a
tablespace directory, either because their names have leading zeroes
that shouldn't be there, or the value is actually zero, or because
the value is too large to represent as an OID.

In those cases, the directory would previously have made it into
the list of tablespaceinfo objects and no longer will. Thus, base
backups will now ignore such directories, instead of treating them
as legitimate tablespace directories. Similarly, if entries for
such tablespaces occur in a tablespace_map file, they will now
be rejected as erroneous, instead of being honored.

This is infrastructure for future work that wants to be able to
know the tablespace of each relation that is part of a backup
*as an OID*. By strengthening the up-front validation, we don't
have to worry about weird cases later, and can more easily avoid
repeated string->integer conversions.

Patch by me, reviewed by David Steele.

Discussion: http://postgr.es/m/CA+TgmoZNVeBzoqDL8xvr-nkaepq815jtDR4nJzPew7=3iEuM1g@mail.gmail.com
2023-10-23 15:17:26 -04:00
Robert Haas
5c47c6546c Refactor parse_filename_for_nontemp_relation to parse more.
Instead of returning the number of characters in the RelFileNumber,
return the RelFileNumber itself. Continue to return the fork number,
as before, and additionally return the segment number.

parse_filename_for_nontemp_relation now rejects a RelFileNumber or
segment number that begins with a leading zero. Before, we accepted
such cases as relation filenames, but if we continued to do so after
this change, the function might return the same values for two
different files (e.g. 1234.5 and 001234.5 or 1234.005) which could be
annoying for callers. Since we don't actually ever generate filenames
with leading zeroes in the names, any such files that we find must
have been created by something other than PostgreSQL, and it is
therefore reasonable to treat them as non-relation files.

Along the way, change unlogged_relation_entry to store a RelFileNumber
rather than an OID. This update should have been made in
851f4cc75cdd8c831f1baa9a7abf8c8248b65890, but it was overlooked.
It's trivial to make the update as part of this commit, perhaps more
trivial than it would have been without it, so do that.

Patch by me, reviewed by David Steele.

Discussion: http://postgr.es/m/CA+TgmoZNVeBzoqDL8xvr-nkaepq815jtDR4nJzPew7=3iEuM1g@mail.gmail.com
2023-10-23 15:08:53 -04:00
Michael Paquier
b6f1cca9ba Remove unnecessary break in pg_logical_replication_slot_advance()
pg_logical_replication_slot_advance() included a break condition to stop
when a targeted LSN is reached, when processing a series of WAL records
with XLogReadRecord().  Since 38a957316d7e, it matched with the check of
its main while loop.  This condition saved from an extra CFI check,
actually pointless, so let's remove this condition and simplify the
code.

In passing, fix an incorrect comment.

Author: Bharath Rupireddy
Reviewed-by: Tom Lane, Gurjeet Singh
Discussion: https://postgr.es/m/CALj2ACWfGDLQ2cy7ZKwxnJqbDkO6Yvqqrqxne5ZN4HYm=PRTGg@mail.gmail.com
2023-10-23 10:20:30 +09:00
Thomas Munro
dab889d60b Fix min_dynamic_shared_memory on Windows.
When min_dynamic_shared_memory is set above 0, we try to find space in a
pre-allocated region of the main shared memory area instead of calling
dsm_impl_XXX() routines to allocate more.  The dsm_pin_segment() and
dsm_unpin_segment() routines had a bug: they called dsm_impl_XXX()
routines even for main region segments.  Nobody noticed before now
because those routines do nothing on Unix, but on Windows they'd fail
while attempting to duplicate an invalid Windows HANDLE.  Add the
missing gating.

Back-patch to 14, where commit 84b1c63a added this feature.  Fixes
pgsql-bugs bug #18165.

Reported-by: Maxime Boyer <maxime.boyer@cra-arc.gc.ca>
Tested-by: Alexander Lakhin <exclusion@gmail.com>
Discussion: https://postgr.es/m/18165-bf4f525cea6e51de%40postgresql.org
2023-10-22 10:04:55 +13:00
Tom Lane
2d870b4aef Allow ALTER SYSTEM to set unrecognized custom GUCs.
Previously, ALTER SYSTEM failed if the target GUC wasn't present in
the session's GUC hashtable.  That is a reasonable behavior for core
(single-part) GUC names, and for custom GUCs for which we have loaded
an extension that's reserved the prefix.  But it's unnecessarily
restrictive otherwise, and it also causes inconsistent behavior:
you can "ALTER SYSTEM SET foo.bar" only if you did "SET foo.bar"
earlier in the session.  That's fairly silly.

Hence, refactor things so that we can execute ALTER SYSTEM even
if the variable doesn't have a GUC hashtable entry, as long as the
name meets the custom-variable naming requirements and does not
have a reserved prefix.  (It's safe to do this even if the
variable belongs to an extension we currently don't have loaded.
A bad value will at worst cause a WARNING when the extension
does get loaded.)

Also, adjust GRANT ON PARAMETER to have the same opinions about
whether to allow an unrecognized GUC name, and to throw the
same errors if not (it previously used a one-size-fits-all
message for several distinguishable conditions).  By default,
only a superuser will be allowed to do ALTER SYSTEM SET on an
unrecognized name, but it's possible to GRANT the ability to
do it.

Patch by me, pursuant to a documentation complaint from
Gavin Panella.  Arguably this is a bug fix, but given the
lack of other complaints I'll refrain from back-patching.

Discussion: https://postgr.es/m/2617358.1697501956@sss.pgh.pa.us
Discussion: https://postgr.es/m/169746329791.169914.16613647309012285391@wrigleys.postgresql.org
2023-10-21 13:35:19 -04:00
Alvaro Herrera
36a14afc07
Make some error strings more generic
It's undesirable to have SQL commands or configuration options in a
translatable error string, so take some of these out.
2023-10-20 22:52:15 +02:00
Tom Lane
2b5154beab Extend ALTER OPERATOR to allow setting more optimization attributes.
Allow the COMMUTATOR, NEGATOR, MERGES, and HASHES attributes to be set
by ALTER OPERATOR.  However, we don't allow COMMUTATOR/NEGATOR to be
changed once set, nor allow the MERGES/HASHES flags to be unset once
set.  Changes like that might invalidate plans already made, and
dealing with the consequences seems like more trouble than it's worth.
The main use-case we foresee for this is to allow addition of missed
properties in extension update scripts, such as extending an existing
operator to support hashing.  So only transitions from not-set to set
states seem very useful.

This patch also causes us to reject some incorrect cases that formerly
resulted in inconsistent catalog state, such as trying to set the
commutator of an operator to be some other operator that already has a
(different) commutator.

While at it, move the InvokeObjectPostCreateHook call for CREATE
OPERATOR to not occur until after we've fixed up commutator or negator
links as needed.  The previous ordering could only be justified by
thinking of the OperatorUpd call as a kind of ALTER OPERATOR step;
but we don't call InvokeObjectPostAlterHook therein.  It seems better
to let the hook see the final state of the operator object.

In the documentation, move the discussion of how to establish
commutator pairs from xoper.sgml to the CREATE OPERATOR ref page.

Tommy Pavlicek, reviewed and editorialized a bit by me

Discussion: https://postgr.es/m/CAEhP-W-vGVzf4udhR5M8Bdv88UYnPrhoSkj3ieR3QNrsGQoqdg@mail.gmail.com
2023-10-20 12:28:46 -04:00
Michael Paquier
dcd4454590 Remove unnecessary dependencies to wal_level=logical in TAP tests
A couple of TAP tests make use of wal_level=logical for nodes that do
not need to do any kind of logical decoding operations, like
subscription nodes on which changes are only applied.  This can be
confusing when reading these tests as setup examples, so let's remove
this configuration where not required (contrary to two-way logical
replication and similar more complex cases).  This simplifies the tests
a bit, making them slightly cheaper with less WAL generated overall.

Author: Hayato Kuroda
Discussion: https://postgr.es/m/TYAPR01MB5866946BCEB747ABE513ACC6F5D5A@TYAPR01MB5866.jpnprd01.prod.outlook.com
2023-10-20 10:09:27 +09:00