mirrors/postgres
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Improve run-time partition pruning to handle any stable expression.

Browse Source 
The initial coding of the run-time-pruning feature only coped with cases
where the partition key(s) are compared to Params.  That is a bit silly;
we can allow it to work with any non-Var-containing stable expression, as
long as we take special care with expressions containing PARAM_EXEC Params.
The code is hardly any longer this way, and it's considerably clearer
(IMO at least).  Per gripe from Pavel Stehule.

David Rowley, whacked around a bit by me

Discussion: https://postgr.es/m/CAFj8pRBjrufA3ocDm8o4LPGNye9Y+pm1b9kCwode4X04CULG3g@mail.gmail.com
This commit is contained in:
Tom Lane 2018-06-10 15:22:25 -04:00
parent c83e202990
commit 73b7f48f78
11 changed files with 461 additions and 271 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

248
src/backend/executor/execPartition.c
View File

@ -48,10 +48,10 @@ static char *ExecBuildSlotPartitionKeyDescription(Relation rel,
bool *isnull,
int maxfieldlen);
static List *adjust_partition_tlist(List *tlist, TupleConversionMap *map);
static void find_subplans_for_params_recurse(PartitionPruneState *prunestate,
PartitionPruningData *pprune,
bool allparams,
Bitmapset **validsubplans);
static void find_matching_subplans_recurse(PartitionPruneState *prunestate,
PartitionPruningData *pprune,
bool initial_prune,
Bitmapset **validsubplans);
/*
@ -1338,25 +1338,23 @@ adjust_partition_tlist(List *tlist, TupleConversionMap *map)
* here are designed to work with any node type which supports an arbitrary
* number of subnodes, e.g. Append, MergeAppend.
*
* Normally this pruning work is performed by the query planner's partition
* pruning code, however, the planner is limited to only being able to prune
* away unneeded partitions using quals which compare the partition key to a
* value which is known to be Const during planning. To allow the same
* pruning to be performed for values which are only determined during
* execution, we must make an additional pruning attempt during execution.
* When pruning involves comparison of a partition key to a constant, it's
* done by the planner. However, if we have a comparison to a non-constant
* but not volatile expression, that presents an opportunity for run-time
* pruning by the executor, allowing irrelevant partitions to be skipped
* dynamically.
*
* Here we support pruning using both external and exec Params. The main
* difference between these that we need to concern ourselves with is the
* time when the values of the Params are known. External Param values are
* known at any time of execution, including executor startup, but exec Param
* values are only known when the executor is running.
* We must distinguish expressions containing PARAM_EXEC Params from
* expressions that don't contain those. Even though a PARAM_EXEC Param is
* considered to be a stable expression, it can change value from one node
* scan to the next during query execution. Stable comparison expressions
* that don't involve such Params allow partition pruning to be done once
* during executor startup. Expressions that do involve such Params require
* us to prune separately for each scan of the parent plan node.
*
* For external Params we may be able to prune away unneeded partitions
* during executor startup. This has the added benefit of not having to
* initialize the unneeded subnodes at all. This is useful as it can save
* quite a bit of effort during executor startup.
* Note that pruning away unneeded subnodes during executor startup has the
* added benefit of not having to initialize the unneeded subnodes at all.
*
* For exec Params, we must delay pruning until the executor is running.
*
* Functions:
*
@ -1369,19 +1367,20 @@ adjust_partition_tlist(List *tlist, TupleConversionMap *map)
* planner's partition prune function into subnode indexes.
*
* ExecFindInitialMatchingSubPlans:
* Returns indexes of matching subnodes utilizing only external Params
* to eliminate subnodes. The function must only be called during
* executor startup for the given node before the subnodes themselves
* are initialized. Subnodes which are found not to match by this
* function must not be included in the node's list of subnodes as this
* function performs a remap of the partition index to subplan index map
* and the newly created map provides indexes only for subnodes which
* remain after calling this function.
* Returns indexes of matching subnodes. Partition pruning is attempted
* without any evaluation of expressions containing PARAM_EXEC Params.
* This function must be called during executor startup for the given
* node before the subnodes themselves are initialized. Subnodes which
* are found not to match by this function must not be included in the
* node's list of subnodes as this function performs a remap of the
* partition index to subplan index map and the newly created map
* provides indexes only for subnodes which remain after calling this
* function.
*
* ExecFindMatchingSubPlans:
* Returns indexes of matching subnodes utilizing all Params to eliminate
* subnodes which can't possibly contain matching tuples. This function
* can only be called while the executor is running.
* Returns indexes of matching subnodes after evaluating all available
* expressions. This function can only be called while the executor is
* running.
*-------------------------------------------------------------------------
*/
@ -1416,8 +1415,9 @@ ExecSetupPartitionPruneState(PlanState *planstate, List *partitionpruneinfo)
*/
prunestate->partprunedata = prunedata;
prunestate->num_partprunedata = list_length(partitionpruneinfo);
prunestate->extparams = NULL;
prunestate->execparams = NULL;
prunestate->do_initial_prune = false; /* may be set below */
prunestate->do_exec_prune = false; /* may be set below */
prunestate->execparamids = NULL;
/*
* Create a sub memory context which we'll use when making calls to the
@ -1444,19 +1444,20 @@ ExecSetupPartitionPruneState(PlanState *planstate, List *partitionpruneinfo)
int partnatts;
int n_steps;
pprune->present_parts = bms_copy(pinfo->present_parts);
pprune->subnode_map = palloc(sizeof(int) * pinfo->nparts);
/*
* We must make a copy of this rather than pointing directly to the
* plan's version as we may end up making modifications to it later.
*/
pprune->subnode_map = palloc(sizeof(int) * pinfo->nparts);
memcpy(pprune->subnode_map, pinfo->subnode_map,
sizeof(int) * pinfo->nparts);
/* We can use the subpart_map verbatim, since we never modify it */
pprune->subpart_map = pinfo->subpart_map;
/* present_parts is also subject to later modification */
pprune->present_parts = bms_copy(pinfo->present_parts);
/*
* Grab some info from the table's relcache; lock was already obtained
* by ExecLockNonLeafAppendTables.
@ -1465,7 +1466,6 @@ ExecSetupPartitionPruneState(PlanState *planstate, List *partitionpruneinfo)
partkey = RelationGetPartitionKey(rel);
partdesc = RelationGetPartitionDesc(rel);
n_steps = list_length(pinfo->pruning_steps);
context->strategy = partkey->strategy;
context->partnatts = partnatts = partkey->partnatts;
@ -1476,10 +1476,11 @@ ExecSetupPartitionPruneState(PlanState *planstate, List *partitionpruneinfo)
context->nparts = pinfo->nparts;
context->boundinfo = partition_bounds_copy(partdesc->boundinfo, partkey);
context->planstate = planstate;
context->safeparams = NULL; /* empty for now */
context->exprstates = palloc0(sizeof(ExprState *) * n_steps * partnatts);
/* Initialize expression states for each expression */
/* Initialize expression state for each expression we need */
n_steps = list_length(pinfo->pruning_steps);
context->exprstates = (ExprState **)
palloc0(sizeof(ExprState *) * n_steps * partnatts);
foreach(lc2, pinfo->pruning_steps)
{
PartitionPruneStepOp *step = (PartitionPruneStepOp *) lfirst(lc2);
@ -1496,13 +1497,14 @@ ExecSetupPartitionPruneState(PlanState *planstate, List *partitionpruneinfo)
foreach(lc3, step->exprs)
{
Expr *expr = (Expr *) lfirst(lc3);
int stateidx;
/* not needed for Consts */
if (!IsA(expr, Const))
{
stateidx = PruneCxtStateIdx(partnatts,
step->step.step_id, keyno);
int stateidx = PruneCxtStateIdx(partnatts,
step->step.step_id,
keyno);
context->exprstates[stateidx] =
ExecInitExpr(expr, context->planstate);
}
@ -1510,32 +1512,29 @@ ExecSetupPartitionPruneState(PlanState *planstate, List *partitionpruneinfo)
}
}
/* Array is not modified at runtime, so just point to plan's copy */
context->exprhasexecparam = pinfo->hasexecparam;
pprune->pruning_steps = pinfo->pruning_steps;
pprune->extparams = bms_copy(pinfo->extparams);
pprune->allparams = bms_union(pinfo->extparams, pinfo->execparams);
pprune->do_initial_prune = pinfo->do_initial_prune;
pprune->do_exec_prune = pinfo->do_exec_prune;
/* Record if pruning would be useful at any level */
prunestate->do_initial_prune |= pinfo->do_initial_prune;
prunestate->do_exec_prune |= pinfo->do_exec_prune;
/*
* Accumulate the paramids which match the partitioned keys of all
* partitioned tables.
* Accumulate the IDs of all PARAM_EXEC Params affecting the
* partitioning decisions at this node.
*/
prunestate->extparams = bms_add_members(prunestate->extparams,
pinfo->extparams);
prunestate->execparams = bms_add_members(prunestate->execparams,
pinfo->execparams);
prunestate->execparamids = bms_add_members(prunestate->execparamids,
pinfo->execparamids);
relation_close(rel, NoLock);
i++;
}
/*
* Cache the union of the paramids of both types. This saves having to
* recalculate it everytime we need to know what they are.
*/
prunestate->allparams = bms_union(prunestate->extparams,
prunestate->execparams);
return prunestate;
}
@ -1543,9 +1542,8 @@ ExecSetupPartitionPruneState(PlanState *planstate, List *partitionpruneinfo)
* ExecFindInitialMatchingSubPlans
* Determine which subset of subplan nodes we need to initialize based
* on the details stored in 'prunestate'. Here we only determine the
* matching partitions using values known during plan startup, which is
* only external Params. Exec Params will be unknown at this time. We
* must delay pruning using exec Params until the actual executor run.
* matching partitions using values known during plan startup, which
* excludes any expressions containing PARAM_EXEC Params.
*
* It is expected that callers of this function do so only once during their
* init plan. The caller must only initialize the subnodes which are returned
@ -1554,8 +1552,6 @@ ExecSetupPartitionPruneState(PlanState *planstate, List *partitionpruneinfo)
* return its matching subnode indexes assuming that the caller discarded
* the original non-matching subnodes.
*
* This function must only be called if 'prunestate' has any extparams.
*
* 'nsubnodes' must be passed as the total number of unpruned subnodes.
*/
Bitmapset *
@ -1565,11 +1561,7 @@ ExecFindInitialMatchingSubPlans(PartitionPruneState *prunestate, int nsubnodes)
MemoryContext oldcontext;
Bitmapset *result = NULL;
/*
* Ensure there's actually external params, or we've not been called
* already.
*/
Assert(!bms_is_empty(prunestate->extparams));
Assert(prunestate->do_initial_prune);
pprune = prunestate->partprunedata;
@ -1579,27 +1571,17 @@ ExecFindInitialMatchingSubPlans(PartitionPruneState *prunestate, int nsubnodes)
*/
oldcontext = MemoryContextSwitchTo(prunestate->prune_context);
/* Determine which subnodes match the external params */
find_subplans_for_params_recurse(prunestate, pprune, false, &result);
/* Perform pruning without using PARAM_EXEC Params */
find_matching_subplans_recurse(prunestate, pprune, true, &result);
MemoryContextSwitchTo(oldcontext);
/* Move to the correct memory context */
/* Copy result out of the temp context before we reset it */
result = bms_copy(result);
MemoryContextReset(prunestate->prune_context);
/*
* Record that partition pruning has been performed for external params.
* These are not required again afterwards, and nullifying them helps
* ensure nothing accidentally calls this function twice on the same
* PartitionPruneState.
*
* (Note we keep prunestate->allparams, because we do use that one
* repeatedly in ExecFindMatchingSubPlans).
*/
bms_free(prunestate->extparams);
prunestate->extparams = NULL;
/* Expression eval may have used space in node's ps_ExprContext too */
ResetExprContext(pprune->context.planstate->ps_ExprContext);
/*
* If any subnodes were pruned, we must re-sequence the subnode indexes so
@ -1669,6 +1651,41 @@ ExecFindInitialMatchingSubPlans(PartitionPruneState *prunestate, int nsubnodes)
}
}
/*
* Now we must determine which sub-partitioned tables still have
* unpruned partitions. The easiest way to do this is to simply loop
* over each PartitionPruningData again checking if there are any
* 'present_parts' in the sub-partitioned table. We needn't bother
* doing this if there are no sub-partitioned tables.
*/
if (prunestate->num_partprunedata > 1)
{
for (i = 0; i < prunestate->num_partprunedata; i++)
{
int nparts;
int j;
pprune = &prunestate->partprunedata[i];
nparts = pprune->context.nparts;
for (j = 0; j < nparts; j++)
{
int subidx = pprune->subpart_map[j];
if (subidx >= 0)
{
PartitionPruningData *subprune;
subprune = &prunestate->partprunedata[subidx];
if (!bms_is_empty(subprune->present_parts))
pprune->present_parts =
bms_add_member(pprune->present_parts, j);
}
}
}
}
pfree(new_subnode_indexes);
}
@ -1678,9 +1695,9 @@ ExecFindInitialMatchingSubPlans(PartitionPruneState *prunestate, int nsubnodes)
/*
* ExecFindMatchingSubPlans
* Determine which subplans match the pruning steps detailed in
* 'pprune' for the current Param values.
* 'pprune' for the current comparison expression values.
*
* Here we utilize both external and exec Params for pruning.
* Here we assume we may evaluate PARAM_EXEC Params.
*/
Bitmapset *
ExecFindMatchingSubPlans(PartitionPruneState *prunestate)
@ -1697,63 +1714,58 @@ ExecFindMatchingSubPlans(PartitionPruneState *prunestate)
*/
oldcontext = MemoryContextSwitchTo(prunestate->prune_context);
find_subplans_for_params_recurse(prunestate, pprune, true, &result);
find_matching_subplans_recurse(prunestate, pprune, false, &result);
MemoryContextSwitchTo(oldcontext);
/* Move to the correct memory context */
/* Copy result out of the temp context before we reset it */
result = bms_copy(result);
MemoryContextReset(prunestate->prune_context);
/* Expression eval may have used space in node's ps_ExprContext too */
ResetExprContext(pprune->context.planstate->ps_ExprContext);
return result;
}
/*
* find_subplans_for_params_recurse
* find_matching_subplans_recurse
* Recursive worker function for ExecFindMatchingSubPlans and
* ExecFindInitialMatchingSubPlans
*
* Adds valid (non-prunable) subplan IDs to *validsubplans
*/
static void
find_subplans_for_params_recurse(PartitionPruneState *prunestate,
PartitionPruningData *pprune,
bool allparams,
Bitmapset **validsubplans)
find_matching_subplans_recurse(PartitionPruneState *prunestate,
PartitionPruningData *pprune,
bool initial_prune,
Bitmapset **validsubplans)
{
PartitionPruneContext *context = &pprune->context;
Bitmapset *partset;
Bitmapset *pruneparams;
int i;
/* Guard against stack overflow due to overly deep partition hierarchy. */
check_stack_depth();
/*
* Use only external params unless we've been asked to also use exec
* params too.
*/
if (allparams)
pruneparams = pprune->allparams;
else
pruneparams = pprune->extparams;
/*
* We only need to determine the matching partitions if there are any
* params matching the partition key at this level. If there are no
* matching params, then we can simply return all subnodes which belong to
* this parent partition. The planner should have already determined
* these to be the minimum possible set. We must still recursively visit
* any subpartitioned tables as we may find their partition keys match
* some Params at their level.
*/
if (!bms_is_empty(pruneparams))
/* Only prune if pruning would be useful at this level. */
if (initial_prune ? pprune->do_initial_prune : pprune->do_exec_prune)
{
context->safeparams = pruneparams;
PartitionPruneContext *context = &pprune->context;
/* Set whether we can evaluate PARAM_EXEC Params or not */
context->evalexecparams = !initial_prune;
partset = get_matching_partitions(context,
pprune->pruning_steps);
}
else
{
/*
* If no pruning is to be done, just include all partitions at this
* level.
*/
partset = pprune->present_parts;
}
/* Translate partset into subnode indexes */
i = -1;
@ -1767,9 +1779,9 @@ find_subplans_for_params_recurse(PartitionPruneState *prunestate,
int partidx = pprune->subpart_map[i];
if (partidx != -1)
find_subplans_for_params_recurse(prunestate,
&prunestate->partprunedata[partidx],
allparams, validsubplans);
find_matching_subplans_recurse(prunestate,
&prunestate->partprunedata[partidx],
initial_prune, validsubplans);
else
{
/*

53
src/backend/executor/nodeAppend.c
View File

@ -133,29 +133,27 @@ ExecInitAppend(Append *node, EState *estate, int eflags)
{
PartitionPruneState *prunestate;
/* We may need an expression context to evaluate partition exprs */
ExecAssignExprContext(estate, &appendstate->ps);
prunestate = ExecSetupPartitionPruneState(&appendstate->ps,
node->part_prune_infos);
/*
* When there are external params matching the partition key we may be
* able to prune away Append subplans now.
*/
if (!bms_is_empty(prunestate->extparams))
/* Perform an initial partition prune, if required. */
if (prunestate->do_initial_prune)
{
/* Determine which subplans match the external params */
/* Determine which subplans survive initial pruning */
validsubplans = ExecFindInitialMatchingSubPlans(prunestate,
list_length(node->appendplans));
/*
* If no subplans match the given parameters then we must handle
* this case in a special way. The problem here is that code in
* explain.c requires an Append to have at least one subplan in
* order for it to properly determine the Vars in that subplan's
* targetlist. We sidestep this issue by just initializing the
* first subplan and setting as_whichplan to NO_MATCHING_SUBPLANS
* to indicate that we don't need to scan any subnodes.
* The case where no subplans survive pruning must be handled
* specially. The problem here is that code in explain.c requires
* an Append to have at least one subplan in order for it to
* properly determine the Vars in that subplan's targetlist. We
* sidestep this issue by just initializing the first subplan and
* setting as_whichplan to NO_MATCHING_SUBPLANS to indicate that
* we don't really need to scan any subnodes.
*/
if (bms_is_empty(validsubplans))
{
@ -175,14 +173,13 @@ ExecInitAppend(Append *node, EState *estate, int eflags)
}
/*
* If there's no exec params then no further pruning can be done, we
* can just set the valid subplans to all remaining subplans.
* If no runtime pruning is required, we can fill as_valid_subplans
* immediately, preventing later calls to ExecFindMatchingSubPlans.
*/
if (bms_is_empty(prunestate->execparams))
if (!prunestate->do_exec_prune)
appendstate->as_valid_subplans = bms_add_range(NULL, 0, nplans - 1);
appendstate->as_prune_state = prunestate;
}
else
{
@ -190,7 +187,7 @@ ExecInitAppend(Append *node, EState *estate, int eflags)
/*
* When run-time partition pruning is not enabled we can just mark all
* subplans as valid, they must also all be initialized.
* subplans as valid; they must also all be initialized.
*/
appendstate->as_valid_subplans = validsubplans =
bms_add_range(NULL, 0, nplans - 1);
@ -341,13 +338,13 @@ ExecReScanAppend(AppendState *node)
int i;
/*
* If any of the parameters being used for partition pruning have changed,
* then we'd better unset the valid subplans so that they are reselected
* for the new parameter values.
* If any PARAM_EXEC Params used in pruning expressions have changed, then
* we'd better unset the valid subplans so that they are reselected for
* the new parameter values.
*/
if (node->as_prune_state &&
bms_overlap(node->ps.chgParam,
node->as_prune_state->execparams))
node->as_prune_state->execparamids))
{
bms_free(node->as_valid_subplans);
node->as_valid_subplans = NULL;
@ -531,9 +528,9 @@ choose_next_subplan_for_leader(AppendState *node)
node->as_whichplan = node->as_nplans - 1;
/*
* If we've yet to determine the valid subplans for these parameters
* then do so now. If run-time pruning is disabled then the valid
* subplans will always be set to all subplans.
* If we've yet to determine the valid subplans then do so now. If
* run-time pruning is disabled then the valid subplans will always be
* set to all subplans.
*/
if (node->as_valid_subplans == NULL)
{
@ -606,9 +603,9 @@ choose_next_subplan_for_worker(AppendState *node)
node->as_pstate->pa_finished[node->as_whichplan] = true;
/*
* If we've yet to determine the valid subplans for these parameters then
* do so now. If run-time pruning is disabled then the valid subplans
* will always be set to all subplans.
* If we've yet to determine the valid subplans then do so now. If
* run-time pruning is disabled then the valid subplans will always be set
* to all subplans.
*/
else if (node->as_valid_subplans == NULL)
{

7
src/backend/nodes/copyfuncs.c
View File

@ -2175,10 +2175,13 @@ _copyPartitionPruneInfo(const PartitionPruneInfo *from)
COPY_NODE_FIELD(pruning_steps);
COPY_BITMAPSET_FIELD(present_parts);
COPY_SCALAR_FIELD(nparts);
COPY_SCALAR_FIELD(nexprs);
COPY_POINTER_FIELD(subnode_map, from->nparts * sizeof(int));
COPY_POINTER_FIELD(subpart_map, from->nparts * sizeof(int));
COPY_BITMAPSET_FIELD(extparams);
COPY_BITMAPSET_FIELD(execparams);
COPY_POINTER_FIELD(hasexecparam, from->nexprs * sizeof(bool));
COPY_SCALAR_FIELD(do_initial_prune);
COPY_SCALAR_FIELD(do_exec_prune);
COPY_BITMAPSET_FIELD(execparamids);
return newnode;
}

10
src/backend/nodes/outfuncs.c
View File

@ -1742,6 +1742,7 @@ _outPartitionPruneInfo(StringInfo str, const PartitionPruneInfo *node)
WRITE_NODE_FIELD(pruning_steps);
WRITE_BITMAPSET_FIELD(present_parts);
WRITE_INT_FIELD(nparts);
WRITE_INT_FIELD(nexprs);
appendStringInfoString(str, " :subnode_map");
for (i = 0; i < node->nparts; i++)
@ -1751,8 +1752,13 @@ _outPartitionPruneInfo(StringInfo str, const PartitionPruneInfo *node)
for (i = 0; i < node->nparts; i++)
appendStringInfo(str, " %d", node->subpart_map[i]);
WRITE_BITMAPSET_FIELD(extparams);
WRITE_BITMAPSET_FIELD(execparams);
appendStringInfoString(str, " :hasexecparam");
for (i = 0; i < node->nexprs; i++)
appendStringInfo(str, " %s", booltostr(node->hasexecparam[i]));
WRITE_BOOL_FIELD(do_initial_prune);
WRITE_BOOL_FIELD(do_exec_prune);
WRITE_BITMAPSET_FIELD(execparamids);
}
/*****************************************************************************

7
src/backend/nodes/readfuncs.c
View File

@ -1363,10 +1363,13 @@ _readPartitionPruneInfo(void)
READ_NODE_FIELD(pruning_steps);
READ_BITMAPSET_FIELD(present_parts);
READ_INT_FIELD(nparts);
READ_INT_FIELD(nexprs);
READ_INT_ARRAY(subnode_map, local_node->nparts);
READ_INT_ARRAY(subpart_map, local_node->nparts);
READ_BITMAPSET_FIELD(extparams);
READ_BITMAPSET_FIELD(execparams);
READ_BOOL_ARRAY(hasexecparam, local_node->nexprs);
READ_BOOL_FIELD(do_initial_prune);
READ_BOOL_FIELD(do_exec_prune);
READ_BITMAPSET_FIELD(execparamids);
READ_DONE();
}

219
src/backend/partitioning/partprune.c
View File

@ -53,6 +53,7 @@
#include "optimizer/planner.h"
#include "optimizer/predtest.h"
#include "optimizer/prep.h"
#include "optimizer/var.h"
#include "partitioning/partprune.h"
#include "partitioning/partbounds.h"
#include "rewrite/rewriteManip.h"
@ -162,7 +163,10 @@ static PruneStepResult *get_matching_list_bounds(PartitionPruneContext *context,
static PruneStepResult *get_matching_range_bounds(PartitionPruneContext *context,
StrategyNumber opstrategy, Datum *values, int nvalues,
FmgrInfo *partsupfunc, Bitmapset *nullkeys);
static bool pull_partkey_params(PartitionPruneInfo *pinfo, List *steps);
static Bitmapset *pull_exec_paramids(Expr *expr);
static bool pull_exec_paramids_walker(Node *node, Bitmapset **context);
static bool analyze_partkey_exprs(PartitionPruneInfo *pinfo, List *steps,
int partnatts);
static PruneStepResult *perform_pruning_base_step(PartitionPruneContext *context,
PartitionPruneStepOp *opstep);
static PruneStepResult *perform_pruning_combine_step(PartitionPruneContext *context,
@ -180,12 +184,12 @@ static bool partkey_datum_from_expr(PartitionPruneContext *context,
* pruning to take place.
*
* Here we generate partition pruning steps for 'prunequal' and also build a
* data stucture which allows mapping of partition indexes into 'subpaths'
* data structure which allows mapping of partition indexes into 'subpaths'
* indexes.
*
* If no Params were found to match the partition key in any of the
* 'partitioned_rels', then we return NIL. In such a case run-time partition
* pruning would be useless.
* If no non-Const expressions are being compared to the partition key in any
* of the 'partitioned_rels', then we return NIL. In such a case run-time
* partition pruning would be useless, since the planner did it already.
*/
List *
make_partition_pruneinfo(PlannerInfo *root, List *partition_rels,
@ -197,7 +201,7 @@ make_partition_pruneinfo(PlannerInfo *root, List *partition_rels,
int *relid_subnode_map;
int *relid_subpart_map;
int i;
bool gotparam = false;
bool doruntimeprune = false;
/*
* Allocate two arrays to store the 1-based indexes of the 'subpaths' and
@ -229,7 +233,7 @@ make_partition_pruneinfo(PlannerInfo *root, List *partition_rels,
relid_subpart_map[rti] = i++;
}
/* We now build a PartitionPruneInfo for each partition_rels */
/* We now build a PartitionPruneInfo for each rel in partition_rels */
foreach(lc, partition_rels)
{
Index rti = lfirst_int(lc);
@ -238,6 +242,7 @@ make_partition_pruneinfo(PlannerInfo *root, List *partition_rels,
RangeTblEntry *rte;
Bitmapset *present_parts;
int nparts = subpart->nparts;
int partnatts = subpart->part_scheme->partnatts;
int *subnode_map;
int *subpart_map;
List *partprunequal;
@ -320,17 +325,11 @@ make_partition_pruneinfo(PlannerInfo *root, List *partition_rels,
pinfo->pruning_steps = pruning_steps;
pinfo->present_parts = present_parts;
pinfo->nparts = nparts;
pinfo->extparams = NULL;
pinfo->execparams = NULL;
pinfo->subnode_map = subnode_map;
pinfo->subpart_map = subpart_map;
/*
* Extract Params matching partition key and record if we got any.
* We'll not bother enabling run-time pruning if no params matched the
* partition key at any level of partitioning.
*/
gotparam |= pull_partkey_params(pinfo, pruning_steps);
/* Determine which pruning types should be enabled at this level */
doruntimeprune |= analyze_partkey_exprs(pinfo, pruning_steps, partnatts);
pinfolist = lappend(pinfolist, pinfo);
}
@ -338,14 +337,10 @@ make_partition_pruneinfo(PlannerInfo *root, List *partition_rels,
pfree(relid_subnode_map);
pfree(relid_subpart_map);
if (gotparam)
if (doruntimeprune)
return pinfolist;
/*
* If no Params were found to match the partition key on any of the
* partitioned relations then there's no point doing any run-time
* partition pruning.
*/
/* No run-time pruning required. */
return NIL;
}
@ -443,10 +438,11 @@ prune_append_rel_partitions(RelOptInfo *rel)
context.nparts = rel->nparts;
context.boundinfo = rel->boundinfo;
/* Not valid when being called from the planner */
/* These are not valid when being called from the planner */
context.planstate = NULL;
context.safeparams = NULL;
context.exprstates = NULL;
context.exprhasexecparam = NULL;
context.evalexecparams = false;
/* Actual pruning happens here. */
partindexes = get_matching_partitions(&context, pruning_steps);
@ -1478,6 +1474,10 @@ match_clause_to_partition_key(RelOptInfo *rel,
if (contain_volatile_functions((Node *) expr))
return PARTCLAUSE_UNSUPPORTED;
/* We can't prune using an expression with Vars. */
if (contain_var_clause((Node *) expr))
return PARTCLAUSE_UNSUPPORTED;
/*
* Determine the input types of the operator we're considering.
*
@ -1624,10 +1624,14 @@ match_clause_to_partition_key(RelOptInfo *rel,
if (!op_strict(saop_op))
return PARTCLAUSE_UNSUPPORTED;
/* Useless if the array has any volatile functions. */
/* We can't use any volatile expressions to prune partitions. */
if (contain_volatile_functions((Node *) rightop))
return PARTCLAUSE_UNSUPPORTED;
/* We can't prune using an expression with Vars. */
if (contain_var_clause((Node *) rightop))
return PARTCLAUSE_UNSUPPORTED;
/*
* In case of NOT IN (..), we get a '<>', which we handle if list
* partitioning is in use and we're able to confirm that it's negator
@ -1655,7 +1659,7 @@ match_clause_to_partition_key(RelOptInfo *rel,
return PARTCLAUSE_UNSUPPORTED;
}
else
return PARTCLAUSE_UNSUPPORTED; /* no useful negator */
return PARTCLAUSE_UNSUPPORTED; /* no useful negator */
}
/*
@ -2683,54 +2687,102 @@ get_matching_range_bounds(PartitionPruneContext *context,
}
/*
* pull_partkey_params
* Loop through each pruning step and record each external and exec
* Params being compared to the partition keys.
* pull_exec_paramids
* Returns a Bitmapset containing the paramids of all Params with
* paramkind = PARAM_EXEC in 'expr'.
*/
static Bitmapset *
pull_exec_paramids(Expr *expr)
{
Bitmapset *result = NULL;
(void) pull_exec_paramids_walker((Node *) expr, &result);
return result;
}
static bool
pull_exec_paramids_walker(Node *node, Bitmapset **context)
{
if (node == NULL)
return false;
if (IsA(node, Param))
{
Param *param = (Param *) node;
if (param->paramkind == PARAM_EXEC)
*context = bms_add_member(*context, param->paramid);
return false;
}
return expression_tree_walker(node, pull_exec_paramids_walker,
(void *) context);
}
/*
* analyze_partkey_exprs
* Loop through all pruning steps and identify which ones require
* executor startup-time or executor run-time pruning.
*
* Returns true if any executor partition pruning should be attempted at this
* level. Also fills fields of *pinfo to record how to process each step.
*/
static bool
pull_partkey_params(PartitionPruneInfo *pinfo, List *steps)
analyze_partkey_exprs(PartitionPruneInfo *pinfo, List *steps, int partnatts)
{
bool doruntimeprune = false;
ListCell *lc;
bool gotone = false;
/*
* Steps require run-time pruning if they contain EXEC_PARAM Params.
* Otherwise, if their expressions aren't simple Consts, they require
* startup-time pruning.
*/
pinfo->nexprs = list_length(steps) * partnatts;
pinfo->hasexecparam = (bool *) palloc0(sizeof(bool) * pinfo->nexprs);
pinfo->do_initial_prune = false;
pinfo->do_exec_prune = false;
pinfo->execparamids = NULL;
foreach(lc, steps)
{
PartitionPruneStepOp *stepop = lfirst(lc);
PartitionPruneStepOp *step = (PartitionPruneStepOp *) lfirst(lc);
ListCell *lc2;
int keyno;
if (!IsA(stepop, PartitionPruneStepOp))
if (!IsA(step, PartitionPruneStepOp))
continue;
foreach(lc2, stepop->exprs)
keyno = 0;
foreach(lc2, step->exprs)
{
Expr *expr = lfirst(lc2);
if (IsA(expr, Param))
if (!IsA(expr, Const))
{
Param *param = (Param *) expr;
Bitmapset *execparamids = pull_exec_paramids(expr);
bool hasexecparams;
int stateidx = PruneCxtStateIdx(partnatts,
step->step.step_id,
keyno);
switch (param->paramkind)
{
case PARAM_EXTERN:
pinfo->extparams = bms_add_member(pinfo->extparams,
param->paramid);
break;
case PARAM_EXEC:
pinfo->execparams = bms_add_member(pinfo->execparams,
param->paramid);
break;
Assert(stateidx < pinfo->nexprs);
hasexecparams = !bms_is_empty(execparamids);
pinfo->hasexecparam[stateidx] = hasexecparams;
pinfo->execparamids = bms_join(pinfo->execparamids,
execparamids);
default:
elog(ERROR, "unrecognized paramkind: %d",
(int) param->paramkind);
break;
}
gotone = true;
if (hasexecparams)
pinfo->do_exec_prune = true;
else
pinfo->do_initial_prune = true;
doruntimeprune = true;
}
keyno++;
}
}
return gotone;
return doruntimeprune;
}
/*
@ -3026,42 +3078,47 @@ match_boolean_partition_clause(Oid partopfamily, Expr *clause, Expr *partkey,
* Evaluate 'expr', whose ExprState is stateidx of the context exprstate
* array; set *value to the resulting Datum. Return true if evaluation was
* possible, otherwise false.
*
* Note that the evaluated result may be in the per-tuple memory context of
* context->planstate->ps_ExprContext, and we may have leaked other memory
* there too. This memory must be recovered by resetting that ExprContext
* after we're done with the pruning operation (see execPartition.c).
*/
static bool
partkey_datum_from_expr(PartitionPruneContext *context,
Expr *expr, int stateidx, Datum *value)
{
switch (nodeTag(expr))
if (IsA(expr, Const))
{
case T_Const:
*value = ((Const *) expr)->constvalue;
*value = ((Const *) expr)->constvalue;
return true;
}
else
{
/*
* When called from the executor we'll have a valid planstate so we
* may be able to evaluate an expression which could not be folded to
* a Const during planning. Since run-time pruning can occur both
* during initialization of the executor or while it's running, we
* must be careful here to evaluate expressions containing PARAM_EXEC
* Params only when told it's OK.
*/
if (context->planstate &&
(context->evalexecparams ||
!context->exprhasexecparam[stateidx]))
{
ExprState *exprstate;
ExprContext *ectx;
bool isNull;
exprstate = context->exprstates[stateidx];
ectx = context->planstate->ps_ExprContext;
*value = ExecEvalExprSwitchContext(exprstate, ectx, &isNull);
if (isNull)
return false;
return true;
case T_Param:
/*
* When being called from the executor we may be able to evaluate
* the Param's value.
*/
if (context->planstate &&
bms_is_member(((Param *) expr)->paramid, context->safeparams))
{
ExprState *exprstate;
ExprContext *ectx;
bool isNull;
exprstate = context->exprstates[stateidx];
ectx = context->planstate->ps_ExprContext;
*value = ExecEvalExprSwitchContext(exprstate, ectx, &isNull);
if (isNull)
return false;
return true;
}
break;
default:
break;
}
}
return false;

38
src/include/executor/execPartition.h
View File

@ -127,15 +127,16 @@ typedef struct PartitionTupleRouting
* subpart_map An array containing the offset into the
* 'partprunedata' array in PartitionPruning, or
* -1 if there is no such element in that array.
* present_parts A Bitmapset of the partition index that we have
* subnodes mapped for.
* present_parts A Bitmapset of the partition indexes that we
* have subnodes mapped for.
* context Contains the context details required to call
* the partition pruning code.
* pruning_steps Contains a list of PartitionPruneStep used to
* pruning_steps List of PartitionPruneSteps used to
* perform the actual pruning.
* extparams Contains paramids of external params found
* matching partition keys in 'pruning_steps'.
* allparams As 'extparams' but also including exec params.
* do_initial_prune true if pruning should be performed during
* executor startup.
* do_exec_prune true if pruning should be performed during
* executor run.
*-----------------------
*/
typedef struct PartitionPruningData
@ -145,15 +146,14 @@ typedef struct PartitionPruningData
Bitmapset *present_parts;
PartitionPruneContext context;
List *pruning_steps;
Bitmapset *extparams;
Bitmapset *allparams;
bool do_initial_prune;
bool do_exec_prune;
} PartitionPruningData;
/*-----------------------
* PartitionPruneState - State object required for executor nodes to perform
* partition pruning elimination of their subnodes. This encapsulates a
* flattened hierarchy of PartitionPruningData structs and also stores all
* paramids which were found to match the partition keys of each partition.
* flattened hierarchy of PartitionPruningData structs.
* This struct can be attached to node types which support arbitrary Lists of
* subnodes containing partitions to allow subnodes to be eliminated due to
* the clauses being unable to match to any tuple that the subnode could
@ -163,24 +163,24 @@ typedef struct PartitionPruningData
* partitioned relation. First element contains the
* details for the target partitioned table.
* num_partprunedata Number of items in 'partprunedata' array.
* do_initial_prune true if pruning should be performed during executor
* startup (at any hierarchy level).
* do_exec_prune true if pruning should be performed during
* executor run (at any hierarchy level).
* prune_context A memory context which can be used to call the query
* planner's partition prune functions.
* extparams All PARAM_EXTERN paramids which were found to match a
* partition key in each of the contained
* PartitionPruningData structs.
* execparams As above but for PARAM_EXEC.
* allparams Union of 'extparams' and 'execparams', saved to avoid
* recalculation.
* execparamids Contains paramids of PARAM_EXEC Params found within
* any of the partprunedata structs.
*-----------------------
*/
typedef struct PartitionPruneState
{
PartitionPruningData *partprunedata;
int num_partprunedata;
bool do_initial_prune;
bool do_exec_prune;
MemoryContext prune_context;
Bitmapset *extparams;
Bitmapset *execparams;
Bitmapset *allparams;
Bitmapset *execparamids;
} PartitionPruneState;
extern PartitionTupleRouting *ExecSetupPartitionTupleRouting(ModifyTableState *mtstate,

12
src/include/nodes/primnodes.h
View File

@ -1597,11 +1597,17 @@ typedef struct PartitionPruneInfo
List *pruning_steps; /* List of PartitionPruneStep */
Bitmapset *present_parts; /* Indexes of all partitions which subnodes
* are present for. */
int nparts; /* The length of the following two arrays */
int nparts; /* Length of subnode_map[] and subpart_map[] */
int nexprs; /* Length of hasexecparam[] */
int *subnode_map; /* subnode index by partition id, or -1 */
int *subpart_map; /* subpart index by partition id, or -1 */
Bitmapset *extparams; /* All external paramids seen in prunesteps */
Bitmapset *execparams; /* All exec paramids seen in prunesteps */
bool *hasexecparam; /* true if corresponding pruning_step contains
* any PARAM_EXEC Params. */
bool do_initial_prune; /* true if pruning should be performed
* during executor startup. */
bool do_exec_prune; /* true if pruning should be performed during
* executor run. */
Bitmapset *execparamids; /* All PARAM_EXEC Param IDs in pruning_steps */
} PartitionPruneInfo;
#endif /* PRIMNODES_H */

20
src/include/partitioning/partprune.h
View File

@ -40,23 +40,27 @@ typedef struct PartitionPruneContext
PartitionBoundInfo boundinfo;
/*
* Can be set when the context is used from the executor to allow params
* found matching the partition key to be evaluated.
* This will be set when the context is used from the executor, to allow
* Params to be evaluated.
*/
PlanState *planstate;
/*
* Parameters that are safe to be used for partition pruning. execparams
* are not safe to use until the executor is running.
*/
Bitmapset *safeparams;
/*
* Array of ExprStates, indexed as per PruneCtxStateIdx; one for each
* partkey in each pruning step. Allocated if planstate is non-NULL,
* otherwise NULL.
*/
ExprState **exprstates;
/*
* Similar array of flags, each true if corresponding 'exprstate'
* expression contains any PARAM_EXEC Params. (Can be NULL if planstate
* is NULL.)
*/
bool *exprhasexecparam;
/* true if it's safe to evaluate PARAM_EXEC Params */
bool evalexecparams;
} PartitionPruneContext;
#define PruneCxtStateIdx(partnatts, step_id, keyno) \

90
src/test/regress/expected/partition_prune.out
View File

@ -1726,8 +1726,8 @@ explain (analyze, costs off, summary off, timing off) execute ab_q1 (2, 4);
Filter: ((a >= $1) AND (a <= $2) AND (b < 3))
(10 rows)
-- Ensure a mix of external and exec params work together at different
-- levels of partitioning.
-- Ensure a mix of PARAM_EXTERN and PARAM_EXEC Params work together at
-- different levels of partitioning.
prepare ab_q2 (int, int) as
select a from ab where a between $1 and $2 and b < (select 3);
execute ab_q2 (1, 8);
@ -1770,7 +1770,7 @@ explain (analyze, costs off, summary off, timing off) execute ab_q2 (2, 2);
Filter: ((a >= $1) AND (a <= $2) AND (b < $0))
(10 rows)
-- As above, but with swap the exec param to the first partition level
-- As above, but swap the PARAM_EXEC Param to the first partition level
prepare ab_q3 (int, int) as
select a from ab where b between $1 and $2 and a < (select 3);
execute ab_q3 (1, 8);
@ -1835,6 +1835,54 @@ fetch backward all from cur;
(2 rows)
commit;
begin;
-- Test run-time pruning using stable functions
create function list_part_fn(int) returns int as $$ begin return $1; end;$$ language plpgsql stable;
-- Ensure pruning works using a stable function containing no Vars
explain (analyze, costs off, summary off, timing off) select * from list_part where a = list_part_fn(1);
QUERY PLAN
------------------------------------------------------
Append (actual rows=1 loops=1)
Subplans Removed: 3
-> Seq Scan on list_part1 (actual rows=1 loops=1)
Filter: (a = list_part_fn(1))
(4 rows)
-- Ensure pruning does not take place when the function has a Var parameter
explain (analyze, costs off, summary off, timing off) select * from list_part where a = list_part_fn(a);
QUERY PLAN
------------------------------------------------------
Append (actual rows=4 loops=1)
-> Seq Scan on list_part1 (actual rows=1 loops=1)
Filter: (a = list_part_fn(a))
-> Seq Scan on list_part2 (actual rows=1 loops=1)
Filter: (a = list_part_fn(a))
-> Seq Scan on list_part3 (actual rows=1 loops=1)
Filter: (a = list_part_fn(a))
-> Seq Scan on list_part4 (actual rows=1 loops=1)
Filter: (a = list_part_fn(a))
(9 rows)
-- Ensure pruning does not take place when the expression contains a Var.
explain (analyze, costs off, summary off, timing off) select * from list_part where a = list_part_fn(1) + a;
QUERY PLAN
------------------------------------------------------
Append (actual rows=0 loops=1)
-> Seq Scan on list_part1 (actual rows=0 loops=1)
Filter: (a = (list_part_fn(1) + a))
Rows Removed by Filter: 1
-> Seq Scan on list_part2 (actual rows=0 loops=1)
Filter: (a = (list_part_fn(1) + a))
Rows Removed by Filter: 1
-> Seq Scan on list_part3 (actual rows=0 loops=1)
Filter: (a = (list_part_fn(1) + a))
Rows Removed by Filter: 1
-> Seq Scan on list_part4 (actual rows=0 loops=1)
Filter: (a = (list_part_fn(1) + a))
Rows Removed by Filter: 1
(13 rows)
rollback;
drop table list_part;
-- Parallel append
-- Suppress the number of loops each parallel node runs for. This is because
@ -2007,7 +2055,7 @@ select explain_parallel_append('execute ab_q5 (33, 44, 55)');
Filter: ((b < 4) AND (a = ANY (ARRAY[$1, $2, $3])))
(9 rows)
-- Test Parallel Append with exec params
-- Test Parallel Append with PARAM_EXEC Params
select explain_parallel_append('select count(*) from ab where (a = (select 1) or a = (select 3)) and b = 2');
explain_parallel_append
-------------------------------------------------------------------------
@ -2079,6 +2127,40 @@ select explain_parallel_append('select avg(ab.a) from ab inner join lprt_a a on
Index Cond: (a = a.a)
(27 rows)
-- Ensure the same partitions are pruned when we make the nested loop
-- parameter an Expr rather than a plain Param.
select explain_parallel_append('select avg(ab.a) from ab inner join lprt_a a on ab.a = a.a + 0 where a.a in(0, 0, 1)');
explain_parallel_append
---------------------------------------------------------------------------------------------------
Finalize Aggregate (actual rows=1 loops=1)
-> Gather (actual rows=2 loops=1)
Workers Planned: 1
Workers Launched: 1
-> Partial Aggregate (actual rows=1 loops=2)
-> Nested Loop (actual rows=0 loops=2)
-> Parallel Seq Scan on lprt_a a (actual rows=51 loops=N)
Filter: (a = ANY ('{0,0,1}'::integer[]))
-> Append (actual rows=0 loops=102)
-> Index Scan using ab_a1_b1_a_idx on ab_a1_b1 (actual rows=0 loops=2)
Index Cond: (a = (a.a + 0))
-> Index Scan using ab_a1_b2_a_idx on ab_a1_b2 (actual rows=0 loops=2)
Index Cond: (a = (a.a + 0))
-> Index Scan using ab_a1_b3_a_idx on ab_a1_b3 (actual rows=0 loops=2)
Index Cond: (a = (a.a + 0))
-> Index Scan using ab_a2_b1_a_idx on ab_a2_b1 (never executed)
Index Cond: (a = (a.a + 0))
-> Index Scan using ab_a2_b2_a_idx on ab_a2_b2 (never executed)
Index Cond: (a = (a.a + 0))
-> Index Scan using ab_a2_b3_a_idx on ab_a2_b3 (never executed)
Index Cond: (a = (a.a + 0))
-> Index Scan using ab_a3_b1_a_idx on ab_a3_b1 (never executed)
Index Cond: (a = (a.a + 0))
-> Index Scan using ab_a3_b2_a_idx on ab_a3_b2 (never executed)
Index Cond: (a = (a.a + 0))
-> Index Scan using ab_a3_b3_a_idx on ab_a3_b3 (never executed)
Index Cond: (a = (a.a + 0))
(27 rows)
insert into lprt_a values(3),(3);
select explain_parallel_append('select avg(ab.a) from ab inner join lprt_a a on ab.a = a.a where a.a in(1, 0, 3)');
explain_parallel_append

28
src/test/regress/sql/partition_prune.sql
View File

@ -348,8 +348,8 @@ execute ab_q1 (1, 8);
explain (analyze, costs off, summary off, timing off) execute ab_q1 (2, 2);
explain (analyze, costs off, summary off, timing off) execute ab_q1 (2, 4);
-- Ensure a mix of external and exec params work together at different
-- levels of partitioning.
-- Ensure a mix of PARAM_EXTERN and PARAM_EXEC Params work together at
-- different levels of partitioning.
prepare ab_q2 (int, int) as
select a from ab where a between $1 and $2 and b < (select 3);
@ -361,7 +361,7 @@ execute ab_q2 (1, 8);
explain (analyze, costs off, summary off, timing off) execute ab_q2 (2, 2);
-- As above, but with swap the exec param to the first partition level
-- As above, but swap the PARAM_EXEC Param to the first partition level
prepare ab_q3 (int, int) as
select a from ab where b between $1 and $2 and a < (select 3);
@ -396,6 +396,22 @@ fetch backward all from cur;
commit;
begin;
-- Test run-time pruning using stable functions
create function list_part_fn(int) returns int as $$ begin return $1; end;$$ language plpgsql stable;
-- Ensure pruning works using a stable function containing no Vars
explain (analyze, costs off, summary off, timing off) select * from list_part where a = list_part_fn(1);
-- Ensure pruning does not take place when the function has a Var parameter
explain (analyze, costs off, summary off, timing off) select * from list_part where a = list_part_fn(a);
-- Ensure pruning does not take place when the expression contains a Var.
explain (analyze, costs off, summary off, timing off) select * from list_part where a = list_part_fn(1) + a;
rollback;
drop table list_part;
-- Parallel append
@ -458,7 +474,7 @@ select explain_parallel_append('execute ab_q5 (2, 3, 3)');
-- We'll still get a single subplan in this case, but it should not be scanned.
select explain_parallel_append('execute ab_q5 (33, 44, 55)');
-- Test Parallel Append with exec params
-- Test Parallel Append with PARAM_EXEC Params
select explain_parallel_append('select count(*) from ab where (a = (select 1) or a = (select 3)) and b = 2');
-- Test pruning during parallel nested loop query
@ -486,6 +502,10 @@ set enable_mergejoin = 0;
select explain_parallel_append('select avg(ab.a) from ab inner join lprt_a a on ab.a = a.a where a.a in(0, 0, 1)');
-- Ensure the same partitions are pruned when we make the nested loop
-- parameter an Expr rather than a plain Param.
select explain_parallel_append('select avg(ab.a) from ab inner join lprt_a a on ab.a = a.a + 0 where a.a in(0, 0, 1)');
insert into lprt_a values(3),(3);
select explain_parallel_append('select avg(ab.a) from ab inner join lprt_a a on ab.a = a.a where a.a in(1, 0, 3)');