Re: How to retain lesser paths at add_path()?
Kohei KaiGai <[email protected]> Tue, 14 Jan 2020 00:46:02 +0900
| Newsgroups | gmane.comp.db.postgresql.devel.general |
|---|---|
| Message-ID | <CAOP8fzYcCZDpc8tzO4qn-ggXf9K5PYhD7JnQS+UfaEgn4hrvtg@mail.gmail.com> |
The v2 patch is attached. This adds two dedicated lists on the RelOptInfo to preserve lesser paths if extension required to retain the path-node to be removed in usual manner. These lesser paths are kept in the separated list, so it never expand the length of pathlist and partial_pathlist. That was the arguable point in the discussion at the last October. The new hook is called just before the path-node removal operation, and gives extension a chance for extra decision. If extension considers the path-node to be removed can be used in the upper path construction stage, they can return 'true' as a signal to preserve this lesser path-node. In case when same kind of path-node already exists in the preserved_pathlist and the supplied lesser path-node is cheaper than the old one, extension can remove the worse one arbitrarily to keep the length of preserved_pathlist. (E.g, PG-Strom may need one GpuJoin path-node either pathlist or preserved- pathlist for further opportunity of combined usage with GpuPreAgg path-node. It just needs "the best GpuJoin path-node" somewhere, not two or more.) Because PostgreSQL core has no information which preserved path-node can be removed, extensions that uses path_removal_decision_hook() has responsibility to keep the length of preserved_(partial_)pathlist reasonable. BTW, add_path() now removes the lesser path-node by pfree(), not only detach from the path-list. (IndexPath is an exception) Does it really make sense? It only releases the path-node itself, so may not release entire objects. So, efficiency of memory usage is limited. And ForeignScan / CustomScan may references the path-node to be removed. It seems to me here is no guarantee lesser path-nodes except for IndexPath nodes are safe to release. Best regards, 2020年1月11日(土) 21:27 Tomas Vondra <[email protected]>: > > On Sat, Jan 11, 2020 at 05:07:11PM +0900, Kohei KaiGai wrote: > >Hi, > > > >The proposition I posted at 10th-Oct proposed to have a separate list to retain > >lesser paths not to expand the path_list length, but here are no comments by > >others at that time. > >Indeed, the latest patch has not been updated yet. > >Please wait for a few days. I'll refresh the patch again. > > > > OK, thanks for the update. I've marked the patch as "waiting on author". > > > regards > > -- > Tomas Vondra http://www.2ndQuadrant.com > PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services -- HeteroDB, Inc / The PG-Strom Project KaiGai Kohei <[email protected]>
pgsql13-path_removal_decision_hook.v2.patch
(application/octet-stream, 5.3 KB)
src/backend/optimizer/util/pathnode.c | 65 +++++++++++++++++++++++++++++++----
src/include/nodes/pathnodes.h | 6 ++++
src/include/optimizer/pathnode.h | 9 +++++
3 files changed, 73 insertions(+), 7 deletions(-)
diff --git a/src/backend/optimizer/util/pathnode.c b/src/backend/optimizer/util/pathnode.c
index e6d08aede5..59d27a5885 100644
--- a/src/backend/optimizer/util/pathnode.c
+++ b/src/backend/optimizer/util/pathnode.c
@@ -57,6 +57,7 @@ static List *reparameterize_pathlist_by_child(PlannerInfo *root,
List *pathlist,
RelOptInfo *child_rel);
+path_removal_decision_hook_type path_removal_decision_hook = NULL;
/*****************************************************************************
* MISC. PATH UTILITIES
@@ -586,12 +587,27 @@ add_path(RelOptInfo *parent_rel, Path *new_path)
{
parent_rel->pathlist = foreach_delete_current(parent_rel->pathlist,
p1);
-
- /*
- * Delete the data pointed-to by the deleted cell, if possible
- */
- if (!IsA(old_path, IndexPath))
- pfree(old_path);
+ if (path_removal_decision_hook &&
+ path_removal_decision_hook(parent_rel,
+ new_path,
+ old_path,
+ false))
+ {
+ /*
+ * Old path is moved to the preserved_pathlist for future
+ * usage, not remove right now.
+ */
+ parent_rel->preserved_pathlist =
+ lappend(parent_rel->preserved_pathlist, old_path);
+ }
+ else
+ {
+ /*
+ * Delete the data pointed-to by the deleted cell, if possible
+ */
+ if (!IsA(old_path, IndexPath))
+ pfree(old_path);
+ }
}
else
{
@@ -615,6 +631,19 @@ add_path(RelOptInfo *parent_rel, Path *new_path)
parent_rel->pathlist =
list_insert_nth(parent_rel->pathlist, insert_at, new_path);
}
+ else if (path_removal_decision_hook &&
+ path_removal_decision_hook(parent_rel,
+ new_path,
+ NULL,
+ false))
+ {
+ /*
+ * New but rejected path is moved to the preserved_pathlist for
+ * future usage, not remove right now.
+ */
+ parent_rel->preserved_pathlist =
+ lappend(parent_rel->preserved_pathlist, new_path);
+ }
else
{
/* Reject and recycle the new path */
@@ -822,7 +851,20 @@ add_partial_path(RelOptInfo *parent_rel, Path *new_path)
{
parent_rel->partial_pathlist =
foreach_delete_current(parent_rel->partial_pathlist, p1);
- pfree(old_path);
+ if (path_removal_decision_hook &&
+ path_removal_decision_hook(parent_rel,
+ new_path,
+ old_path,
+ true))
+ {
+ /* Old path is preserved for future usage */
+ parent_rel->preserved_partial_pathlist =
+ lappend(parent_rel->preserved_partial_pathlist, old_path);
+ }
+ else
+ {
+ pfree(old_path);
+ }
}
else
{
@@ -846,6 +888,15 @@ add_partial_path(RelOptInfo *parent_rel, Path *new_path)
parent_rel->partial_pathlist =
list_insert_nth(parent_rel->partial_pathlist, insert_at, new_path);
}
+ else if (path_removal_decision_hook &&
+ path_removal_decision_hook(parent_rel,
+ new_path,
+ NULL,
+ true))
+ {
+ parent_rel->preserved_partial_pathlist =
+ lappend(parent_rel->preserved_partial_pathlist, new_path);
+ }
else
{
/* Reject and recycle the new path */
diff --git a/src/include/nodes/pathnodes.h b/src/include/nodes/pathnodes.h
index 3d3be197e0..b31491350c 100644
--- a/src/include/nodes/pathnodes.h
+++ b/src/include/nodes/pathnodes.h
@@ -483,6 +483,10 @@ typedef struct PartitionSchemeData *PartitionScheme;
* (no duplicates) output from relation; NULL if not yet requested
* cheapest_parameterized_paths - best paths for their parameterizations;
* always includes cheapest_total_path, even if that's unparameterized
+ * preserved_pathlist - List of lesser Path nodes; they are not used in
+ * the Path consideration in usual, but extension may want to pick up
+ * in case when a special combination of Path nodes can provide more
+ * efficient execution plan.
* direct_lateral_relids - rels this rel has direct LATERAL references to
* lateral_relids - required outer rels for LATERAL, as a Relids set
* (includes both direct and indirect lateral references)
@@ -657,6 +661,8 @@ typedef struct RelOptInfo
List *pathlist; /* Path structures */
List *ppilist; /* ParamPathInfos used in pathlist */
List *partial_pathlist; /* partial Paths */
+ List *preserved_pathlist; /* preserved 'lesser' Paths */
+ List *preserved_partial_pathlist; /* preserved partial Paths */
struct Path *cheapest_startup_path;
struct Path *cheapest_total_path;
struct Path *cheapest_unique_path;
diff --git a/src/include/optimizer/pathnode.h b/src/include/optimizer/pathnode.h
index e450fe112a..a595920a48 100644
--- a/src/include/optimizer/pathnode.h
+++ b/src/include/optimizer/pathnode.h
@@ -17,6 +17,15 @@
#include "nodes/bitmapset.h"
#include "nodes/pathnodes.h"
+/*
+ * Plugins can provide extra decision whether the Path-node should be
+ * retained at the preserved_(partial_)pathlist.
+ */
+typedef bool (*path_removal_decision_hook_type)(RelOptInfo *parent_rel,
+ Path *new_path,
+ Path *old_path,
+ bool is_partial_pathlist);
+extern PGDLLIMPORT path_removal_decision_hook_type path_removal_decision_hook;
/*
* prototypes for pathnode.c