Re: How to retain lesser paths at add_path()?

Kohei KaiGai <[email protected]> Tue, 14 Jan 2020 00:46:02 +0900
Newsgroups gmane.comp.db.postgresql.devel.general
Message-ID <CAOP8fzYcCZDpc8tzO4qn-ggXf9K5PYhD7JnQS+UfaEgn4hrvtg@mail.gmail.com>
The v2 patch is attached.

This adds two dedicated lists on the RelOptInfo to preserve lesser paths
if extension required to retain the path-node to be removed in usual manner.
These lesser paths are kept in the separated list, so it never expand the length
of pathlist and partial_pathlist. That was the arguable point in the discussion
at the last October.

The new hook is called just before the path-node removal operation, and
gives extension a chance for extra decision.
If extension considers the path-node to be removed can be used in the upper
path construction stage, they can return 'true' as a signal to preserve this
lesser path-node.
In case when same kind of path-node already exists in the preserved_pathlist
and the supplied lesser path-node is cheaper than the old one, extension can
remove the worse one arbitrarily to keep the length of preserved_pathlist.
(E.g, PG-Strom may need one GpuJoin path-node either pathlist or preserved-
pathlist for further opportunity of combined usage with GpuPreAgg path-node.
It just needs "the best GpuJoin path-node" somewhere, not two or more.)

Because PostgreSQL core has no information which preserved path-node can
be removed, extensions that uses path_removal_decision_hook() has responsibility
to keep the length of preserved_(partial_)pathlist reasonable.


BTW, add_path() now removes the lesser path-node by pfree(), not only detach
from the path-list. (IndexPath is an exception)
Does it really make sense? It only releases the path-node itself, so may not
release entire objects. So, efficiency of memory usage is limited. And
ForeignScan
/ CustomScan may references the path-node to be removed. It seems to me here
is no guarantee lesser path-nodes except for IndexPath nodes are safe
to release.

Best regards,

2020年1月11日(土) 21:27 Tomas Vondra <[email protected]>:
>
> On Sat, Jan 11, 2020 at 05:07:11PM +0900, Kohei KaiGai wrote:
> >Hi,
> >
> >The proposition I posted at 10th-Oct proposed to have a separate list to retain
> >lesser paths not to expand the path_list length, but here are no comments by
> >others at that time.
> >Indeed, the latest patch has not been updated yet.
> >Please wait for a few days. I'll refresh the patch again.
> >
>
> OK, thanks for the update. I've marked the patch as "waiting on author".
>
>
> regards
>
> --
> Tomas Vondra                  http://www.2ndQuadrant.com
> PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services



-- 
HeteroDB, Inc / The PG-Strom Project
KaiGai Kohei <[email protected]>
pgsql13-path_removal_decision_hook.v2.patch (application/octet-stream, 5.3 KB)
 src/backend/optimizer/util/pathnode.c | 65 +++++++++++++++++++++++++++++++----
 src/include/nodes/pathnodes.h         |  6 ++++
 src/include/optimizer/pathnode.h      |  9 +++++
 3 files changed, 73 insertions(+), 7 deletions(-)

diff --git a/src/backend/optimizer/util/pathnode.c b/src/backend/optimizer/util/pathnode.c
index e6d08aede5..59d27a5885 100644
--- a/src/backend/optimizer/util/pathnode.c
+++ b/src/backend/optimizer/util/pathnode.c
@@ -57,6 +57,7 @@ static List *reparameterize_pathlist_by_child(PlannerInfo *root,
 											  List *pathlist,
 											  RelOptInfo *child_rel);
 
+path_removal_decision_hook_type path_removal_decision_hook = NULL;
 
 /*****************************************************************************
  *		MISC. PATH UTILITIES
@@ -586,12 +587,27 @@ add_path(RelOptInfo *parent_rel, Path *new_path)
 		{
 			parent_rel->pathlist = foreach_delete_current(parent_rel->pathlist,
 														  p1);
-
-			/*
-			 * Delete the data pointed-to by the deleted cell, if possible
-			 */
-			if (!IsA(old_path, IndexPath))
-				pfree(old_path);
+			if (path_removal_decision_hook &&
+				path_removal_decision_hook(parent_rel,
+										   new_path,
+										   old_path,
+										   false))
+			{
+				/*
+				 * Old path is moved to the preserved_pathlist for future
+				 * usage, not remove right now.
+				 */
+				parent_rel->preserved_pathlist =
+					lappend(parent_rel->preserved_pathlist, old_path);
+			}
+			else
+			{
+				/*
+				 * Delete the data pointed-to by the deleted cell, if possible
+				 */
+				if (!IsA(old_path, IndexPath))
+					pfree(old_path);
+			}
 		}
 		else
 		{
@@ -615,6 +631,19 @@ add_path(RelOptInfo *parent_rel, Path *new_path)
 		parent_rel->pathlist =
 			list_insert_nth(parent_rel->pathlist, insert_at, new_path);
 	}
+	else if (path_removal_decision_hook &&
+			 path_removal_decision_hook(parent_rel,
+										new_path,
+										NULL,
+										false))
+	{
+		/*
+		 * New but rejected path is moved to the preserved_pathlist for
+		 * future usage, not remove right now.
+		 */
+		parent_rel->preserved_pathlist =
+			lappend(parent_rel->preserved_pathlist, new_path);
+	}
 	else
 	{
 		/* Reject and recycle the new path */
@@ -822,7 +851,20 @@ add_partial_path(RelOptInfo *parent_rel, Path *new_path)
 		{
 			parent_rel->partial_pathlist =
 				foreach_delete_current(parent_rel->partial_pathlist, p1);
-			pfree(old_path);
+			if (path_removal_decision_hook &&
+				path_removal_decision_hook(parent_rel,
+										   new_path,
+										   old_path,
+										   true))
+			{
+				/* Old path is preserved for future usage */
+				parent_rel->preserved_partial_pathlist =
+					lappend(parent_rel->preserved_partial_pathlist, old_path);
+			}
+			else
+			{
+				pfree(old_path);
+			}
 		}
 		else
 		{
@@ -846,6 +888,15 @@ add_partial_path(RelOptInfo *parent_rel, Path *new_path)
 		parent_rel->partial_pathlist =
 			list_insert_nth(parent_rel->partial_pathlist, insert_at, new_path);
 	}
+	else if (path_removal_decision_hook &&
+			 path_removal_decision_hook(parent_rel,
+										new_path,
+										NULL,
+										true))
+	{
+		parent_rel->preserved_partial_pathlist =
+			lappend(parent_rel->preserved_partial_pathlist, new_path);
+	}
 	else
 	{
 		/* Reject and recycle the new path */
diff --git a/src/include/nodes/pathnodes.h b/src/include/nodes/pathnodes.h
index 3d3be197e0..b31491350c 100644
--- a/src/include/nodes/pathnodes.h
+++ b/src/include/nodes/pathnodes.h
@@ -483,6 +483,10 @@ typedef struct PartitionSchemeData *PartitionScheme;
  *			(no duplicates) output from relation; NULL if not yet requested
  *		cheapest_parameterized_paths - best paths for their parameterizations;
  *			always includes cheapest_total_path, even if that's unparameterized
+ *		preserved_pathlist - List of lesser Path nodes; they are not used in
+ *			the Path consideration in usual, but extension may want to pick up
+ *			in case when a special combination of Path nodes can provide more
+ *			efficient execution plan.
  *		direct_lateral_relids - rels this rel has direct LATERAL references to
  *		lateral_relids - required outer rels for LATERAL, as a Relids set
  *			(includes both direct and indirect lateral references)
@@ -657,6 +661,8 @@ typedef struct RelOptInfo
 	List	   *pathlist;		/* Path structures */
 	List	   *ppilist;		/* ParamPathInfos used in pathlist */
 	List	   *partial_pathlist;	/* partial Paths */
+	List	   *preserved_pathlist;	/* preserved 'lesser' Paths */
+	List	   *preserved_partial_pathlist; /* preserved partial Paths */
 	struct Path *cheapest_startup_path;
 	struct Path *cheapest_total_path;
 	struct Path *cheapest_unique_path;
diff --git a/src/include/optimizer/pathnode.h b/src/include/optimizer/pathnode.h
index e450fe112a..a595920a48 100644
--- a/src/include/optimizer/pathnode.h
+++ b/src/include/optimizer/pathnode.h
@@ -17,6 +17,15 @@
 #include "nodes/bitmapset.h"
 #include "nodes/pathnodes.h"
 
+/*
+ * Plugins can provide extra decision whether the Path-node should be
+ * retained at the preserved_(partial_)pathlist.
+ */
+typedef bool (*path_removal_decision_hook_type)(RelOptInfo *parent_rel,
+												Path *new_path,
+												Path *old_path,
+												bool is_partial_pathlist);
+extern PGDLLIMPORT path_removal_decision_hook_type path_removal_decision_hook;
 
 /*
  * prototypes for pathnode.c