Debian Codegraph @

doc/src/sgml/custom-scan.sgml

raw
1 <!-- doc/src/sgml/custom-scan.sgml -->
2
3 <chapter id="custom-scan">
4 <title>Writing a Custom Scan Provider</title>
5
6 <indexterm zone="custom-scan">
7 <primary>custom scan provider</primary>
8 <secondary>handler for</secondary>
9 </indexterm>
10
11 <para>
12 <productname>PostgreSQL</productname> supports a set of experimental facilities which
13 are intended to allow extension modules to add new scan types to the system.
14 Unlike a <link linkend="fdwhandler">foreign data wrapper</link>, which is only
15 responsible for knowing how to scan its own foreign tables, a custom scan
16 provider can provide an alternative method of scanning any relation in the
17 system. Typically, the motivation for writing a custom scan provider will
18 be to allow the use of some optimization not supported by the core
19 system, such as caching or some form of hardware acceleration. This chapter
20 outlines how to write a new custom scan provider.
21 </para>
22
23 <para>
24 Implementing a new type of custom scan is a three-step process. First,
25 during planning, it is necessary to generate access paths representing a
26 scan using the proposed strategy. Second, if one of those access paths
27 is selected by the planner as the optimal strategy for scanning a
28 particular relation, the access path must be converted to a plan.
29 Finally, it must be possible to execute the plan and generate the same
30 results that would have been generated for any other access path targeting
31 the same relation.
32 </para>
33
34 <sect1 id="custom-scan-path">
35 <title>Creating Custom Scan Paths</title>
36
37 <para>
38 A custom scan provider will typically add paths for a base relation by
39 setting the following hook, which is called after the core code has
40 generated all the access paths it can for the relation (except for
41 Gather and Gather Merge paths, which are made after this call so that
42 they can use partial paths added by the hook):
43 <programlisting>
44 typedef void (*set_rel_pathlist_hook_type) (PlannerInfo *root,
45 RelOptInfo *rel,
46 Index rti,
47 RangeTblEntry *rte);
48 extern PGDLLIMPORT set_rel_pathlist_hook_type set_rel_pathlist_hook;
49 </programlisting>
50 </para>
51
52 <para>
53 Although this hook function can be used to examine, modify, or remove
54 paths generated by the core system, a custom scan provider will typically
55 confine itself to generating <structname>CustomPath</structname> objects and adding
56 them to <literal>rel</literal> using <function>add_path</function>, or
57 <function>add_partial_path</function> if they are partial paths. The
58 custom scan provider is responsible for initializing the
59 <structname>CustomPath</structname> object, which is declared like this:
60 <programlisting>
61 typedef struct CustomPath
62 {
63 Path path;
64 uint32 flags;
65 List *custom_paths;
66 List *custom_restrictinfo;
67 List *custom_private;
68 const CustomPathMethods *methods;
69 } CustomPath;
70 </programlisting>
71 </para>
72
73 <para>
74 <structfield>path</structfield> must be initialized as for any other path, including
75 the row-count estimate, start and total cost, and sort ordering provided
76 by this path. <structfield>flags</structfield> is a bit mask, which
77 specifies whether the scan provider can support certain optional
78 capabilities. <structfield>flags</structfield> should include
79 <literal>CUSTOMPATH_SUPPORT_BACKWARD_SCAN</literal> if the custom path can support
80 a backward scan, <literal>CUSTOMPATH_SUPPORT_MARK_RESTORE</literal> if it
81 can support mark and restore,
82 and <literal>CUSTOMPATH_SUPPORT_PROJECTION</literal> if it can perform
83 projections. (If <literal>CUSTOMPATH_SUPPORT_PROJECTION</literal> is not
84 set, the scan node will only be asked to produce Vars of the scanned
85 relation; while if that flag is set, the scan node must be able to
86 evaluate scalar expressions over these Vars.)
87 An optional <structfield>custom_paths</structfield> is a list of <structname>Path</structname>
88 nodes used by this custom-path node; these will be transformed into
89 <structname>Plan</structname> nodes by planner.
90 As described below, custom paths can be created for join relations as
91 well. In such a case, <structfield>custom_restrictinfo</structfield>
92 should be used to store the set of join clauses to apply to the join the
93 custom path replaces. Otherwise it should be NIL.
94 <structfield>custom_private</structfield> can be used to store the custom path's
95 private data. Private data should be stored in a form that can be handled
96 by <literal>nodeToString</literal>, so that debugging routines that attempt to
97 print the custom path will work as designed. <structfield>methods</structfield> must
98 point to a (usually statically allocated) object implementing the required
99 custom path methods, which are further detailed below.
100 </para>
101
102 <para>
103 A custom scan provider can also provide join paths. Just as for base
104 relations, such a path must produce the same output as would normally be
105 produced by the join it replaces. To do this, the join provider should
106 set the following hook, and then within the hook function,
107 create <structname>CustomPath</structname> path(s) for the join relation.
108 <programlisting>
109 typedef void (*set_join_pathlist_hook_type) (PlannerInfo *root,
110 RelOptInfo *joinrel,
111 RelOptInfo *outerrel,
112 RelOptInfo *innerrel,
113 JoinType jointype,
114 JoinPathExtraData *extra);
115 extern PGDLLIMPORT set_join_pathlist_hook_type set_join_pathlist_hook;
116 </programlisting>
117
118 This hook will be invoked repeatedly for the same join relation, with
119 different combinations of inner and outer relations; it is the
120 responsibility of the hook to minimize duplicated work.
121 </para>
122
123 <para>
124 Note also that the set of join clauses to apply to the join,
125 which is passed as <literal>extra-&gt;restrictlist</literal>, varies
126 depending on the combination of inner and outer relations. A
127 <structname>CustomPath</structname> path generated for the
128 <literal>joinrel</literal> must contain the set of join clauses it uses,
129 which will be used by the planner to convert the
130 <structname>CustomPath</structname> path into a plan, if it is selected
131 by the planner as the best path for the <literal>joinrel</literal>.
132 </para>
133
134 <sect2 id="custom-scan-path-callbacks">
135 <title>Custom Scan Path Callbacks</title>
136
137 <para>
138 <programlisting>
139 Plan *(*PlanCustomPath) (PlannerInfo *root,
140 RelOptInfo *rel,
141 CustomPath *best_path,
142 List *tlist,
143 List *clauses,
144 List *custom_plans);
145 </programlisting>
146 Convert a custom path to a finished plan. The return value will generally
147 be a <literal>CustomScan</literal> object, which the callback must allocate and
148 initialize. See <xref linkend="custom-scan-plan"/> for more details.
149 </para>
150
151 <para>
152 <programlisting>
153 List *(*ReparameterizeCustomPathByChild) (PlannerInfo *root,
154 List *custom_private,
155 RelOptInfo *child_rel);
156 </programlisting>
157 This callback is called while converting a path parameterized by the
158 top-most parent of the given child relation <literal>child_rel</literal>
159 to be parameterized by the child relation. The callback is used to
160 reparameterize any paths or translate any expression nodes saved in the
161 given <literal>custom_private</literal> member of a
162 <structname>CustomPath</structname>. The callback may use
163 <literal>reparameterize_path_by_child</literal>,
164 <literal>adjust_appendrel_attrs</literal> or
165 <literal>adjust_appendrel_attrs_multilevel</literal> as required.
166 </para>
167 </sect2>
168 </sect1>
169
170 <sect1 id="custom-scan-plan">
171 <title>Creating Custom Scan Plans</title>
172
173 <para>
174 A custom scan is represented in a finished plan tree using the following
175 structure:
176 <programlisting>
177 typedef struct CustomScan
178 {
179 Scan scan;
180 uint32 flags;
181 List *custom_plans;
182 List *custom_exprs;
183 List *custom_private;
184 List *custom_scan_tlist;
185 Bitmapset *custom_relids;
186 const CustomScanMethods *methods;
187 } CustomScan;
188 </programlisting>
189 </para>
190
191 <para>
192 <structfield>scan</structfield> must be initialized as for any other scan, including
193 estimated costs, target lists, qualifications, and so on.
194 <structfield>flags</structfield> is a bit mask with the same meaning as in
195 <structname>CustomPath</structname>.
196 <structfield>custom_plans</structfield> can be used to store child
197 <structname>Plan</structname> nodes.
198 <structfield>custom_exprs</structfield> should be used to
199 store expression trees that will need to be fixed up by
200 <filename>setrefs.c</filename> and <filename>subselect.c</filename>, while
201 <structfield>custom_private</structfield> should be used to store other private data
202 that is only used by the custom scan provider itself.
203 <structfield>custom_scan_tlist</structfield> can be NIL when scanning a base
204 relation, indicating that the custom scan returns scan tuples that match
205 the base relation's row type. Otherwise it is a target list describing
206 the actual scan tuples. <structfield>custom_scan_tlist</structfield> must be
207 provided for joins, and could be provided for scans if the custom scan
208 provider can compute some non-Var expressions.
209 <structfield>custom_relids</structfield> is set by the core code to the set of
210 relations (range table indexes) that this scan node handles; except when
211 this scan is replacing a join, it will have only one member.
212 <structfield>methods</structfield> must point to a (usually statically allocated)
213 object implementing the required custom scan methods, which are further
214 detailed below.
215 </para>
216
217 <para>
218 When a <structname>CustomScan</structname> scans a single relation,
219 <structfield>scan.scanrelid</structfield> must be the range table index of the table
220 to be scanned. When it replaces a join, <structfield>scan.scanrelid</structfield>
221 should be zero.
222 </para>
223
224 <para>
225 Plan trees must be able to be duplicated using <function>copyObject</function>,
226 so all the data stored within the <quote>custom</quote> fields must consist of
227 nodes that that function can handle. Furthermore, custom scan providers
228 cannot substitute a larger structure that embeds
229 a <structname>CustomScan</structname> for the structure itself, as would be possible
230 for a <structname>CustomPath</structname> or <structname>CustomScanState</structname>.
231 </para>
232
233 <sect2 id="custom-scan-plan-callbacks">
234 <title>Custom Scan Plan Callbacks</title>
235 <para>
236 <programlisting>
237 Node *(*CreateCustomScanState) (CustomScan *cscan);
238 </programlisting>
239 Allocate a <structname>CustomScanState</structname> for this
240 <structname>CustomScan</structname>. The actual allocation will often be larger than
241 required for an ordinary <structname>CustomScanState</structname>, because many
242 providers will wish to embed that as the first field of a larger structure.
243 The value returned must have the node tag and <structfield>methods</structfield>
244 set appropriately, but other fields should be left as zeroes at this
245 stage; after <function>ExecInitCustomScan</function> performs basic initialization,
246 the <function>BeginCustomScan</function> callback will be invoked to give the
247 custom scan provider a chance to do whatever else is needed.
248 </para>
249 </sect2>
250 </sect1>
251
252 <sect1 id="custom-scan-execution">
253 <title>Executing Custom Scans</title>
254
255 <para>
256 When a <structfield>CustomScan</structfield> is executed, its execution state is
257 represented by a <structfield>CustomScanState</structfield>, which is declared as
258 follows:
259 <programlisting>
260 typedef struct CustomScanState
261 {
262 ScanState ss;
263 uint32 flags;
264 const CustomExecMethods *methods;
265 } CustomScanState;
266 </programlisting>
267 </para>
268
269 <para>
270 <structfield>ss</structfield> is initialized as for any other scan state,
271 except that if the scan is for a join rather than a base relation,
272 <literal>ss.ss_currentRelation</literal> is left NULL.
273 <structfield>flags</structfield> is a bit mask with the same meaning as in
274 <structname>CustomPath</structname> and <structname>CustomScan</structname>.
275 <structfield>methods</structfield> must point to a (usually statically allocated)
276 object implementing the required custom scan state methods, which are
277 further detailed below. Typically, a <structname>CustomScanState</structname>, which
278 need not support <function>copyObject</function>, will actually be a larger
279 structure embedding the above as its first member.
280 </para>
281
282 <sect2 id="custom-scan-execution-callbacks">
283 <title>Custom Scan Execution Callbacks</title>
284
285 <para>
286 <programlisting>
287 void (*BeginCustomScan) (CustomScanState *node,
288 EState *estate,
289 int eflags);
290 </programlisting>
291 Complete initialization of the supplied <structname>CustomScanState</structname>.
292 Standard fields have been initialized by <function>ExecInitCustomScan</function>,
293 but any private fields should be initialized here.
294 </para>
295
296 <para>
297 <programlisting>
298 TupleTableSlot *(*ExecCustomScan) (CustomScanState *node);
299 </programlisting>
300 Fetch the next scan tuple. If any tuples remain, it should fill
301 <literal>ps_ResultTupleSlot</literal> with the next tuple in the current scan
302 direction, and then return the tuple slot. If not,
303 <literal>NULL</literal> or an empty slot should be returned.
304 </para>
305
306 <para>
307 <programlisting>
308 void (*EndCustomScan) (CustomScanState *node);
309 </programlisting>
310 Clean up any private data associated with the <literal>CustomScanState</literal>.
311 This method is required, but it does not need to do anything if there is
312 no associated data or it will be cleaned up automatically.
313 </para>
314
315 <para>
316 <programlisting>
317 void (*ReScanCustomScan) (CustomScanState *node);
318 </programlisting>
319 Rewind the current scan to the beginning and prepare to rescan the
320 relation.
321 </para>
322
323 <para>
324 <programlisting>
325 void (*MarkPosCustomScan) (CustomScanState *node);
326 </programlisting>
327 Save the current scan position so that it can subsequently be restored
328 by the <function>RestrPosCustomScan</function> callback. This callback is
329 optional, and need only be supplied if the
330 <literal>CUSTOMPATH_SUPPORT_MARK_RESTORE</literal> flag is set.
331 </para>
332
333 <para>
334 <programlisting>
335 void (*RestrPosCustomScan) (CustomScanState *node);
336 </programlisting>
337 Restore the previous scan position as saved by the
338 <function>MarkPosCustomScan</function> callback. This callback is optional,
339 and need only be supplied if the
340 <literal>CUSTOMPATH_SUPPORT_MARK_RESTORE</literal> flag is set.
341 </para>
342
343 <para>
344 <programlisting>
345 Size (*EstimateDSMCustomScan) (CustomScanState *node,
346 ParallelContext *pcxt);
347 </programlisting>
348 Estimate the amount of dynamic shared memory that will be required
349 for parallel operation. This may be higher than the amount that will
350 actually be used, but it must not be lower. The return value is in bytes.
351 This callback is optional, and need only be supplied if this custom
352 scan provider supports parallel execution.
353 </para>
354
355 <para>
356 <programlisting>
357 void (*InitializeDSMCustomScan) (CustomScanState *node,
358 ParallelContext *pcxt,
359 void *coordinate);
360 </programlisting>
361 Initialize the dynamic shared memory that will be required for parallel
362 operation. <literal>coordinate</literal> points to a shared memory area of
363 size equal to the return value of <function>EstimateDSMCustomScan</function>.
364 This callback is optional, and need only be supplied if this custom
365 scan provider supports parallel execution.
366 </para>
367
368 <para>
369 <programlisting>
370 void (*ReInitializeDSMCustomScan) (CustomScanState *node,
371 ParallelContext *pcxt,
372 void *coordinate);
373 </programlisting>
374 Re-initialize the dynamic shared memory required for parallel operation
375 when the custom-scan plan node is about to be re-scanned.
376 This callback is optional, and need only be supplied if this custom
377 scan provider supports parallel execution.
378 Recommended practice is that this callback reset only shared state,
379 while the <function>ReScanCustomScan</function> callback resets only local
380 state. Currently, this callback will be called
381 before <function>ReScanCustomScan</function>, but it's best not to rely on
382 that ordering.
383 </para>
384
385 <para>
386 <programlisting>
387 void (*InitializeWorkerCustomScan) (CustomScanState *node,
388 shm_toc *toc,
389 void *coordinate);
390 </programlisting>
391 Initialize a parallel worker's local state based on the shared state
392 set up by the leader during <function>InitializeDSMCustomScan</function>.
393 This callback is optional, and need only be supplied if this custom
394 scan provider supports parallel execution.
395 </para>
396
397 <para>
398 <programlisting>
399 void (*ShutdownCustomScan) (CustomScanState *node);
400 </programlisting>
401 Release resources when it is anticipated the node will not be executed
402 to completion. This is not called in all cases; sometimes,
403 <literal>EndCustomScan</literal> may be called without this function having
404 been called first. Since the DSM segment used by parallel query is
405 destroyed just after this callback is invoked, custom scan providers that
406 wish to take some action before the DSM segment goes away should implement
407 this method.
408 </para>
409
410 <para>
411 <programlisting>
412 void (*ExplainCustomScan) (CustomScanState *node,
413 List *ancestors,
414 ExplainState *es);
415 </programlisting>
416 Output additional information for <command>EXPLAIN</command> of a custom-scan
417 plan node. This callback is optional. Common data stored in the
418 <structname>ScanState</structname>, such as the target list and scan relation, will
419 be shown even without this callback, but the callback allows the display
420 of additional, private state.
421 </para>
422 </sect2>
423 </sect1>
424 </chapter>