Debian Codegraph @

jenkins_jobs/yaml_objects.py

raw
1 # Licensed under the Apache License, Version 2.0 (the "License");
2 # you may not use this file except in compliance with the License.
3 # You may obtain a copy of the License at
4 #
5 # http://www.apache.org/licenses/LICENSE-2.0
6 #
7 # Unless required by applicable law or agreed to in writing, software
8 # distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
9 # WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
10 # License for the specific language governing permissions and limitations
11 # under the License.
12
13 # Provides local yaml parsing classes and extends yaml module.
14
15 """Custom application specific yamls tags are supported to provide
16 enhancements when reading yaml configuration.
17
18 Action Tags
19 ^^^^^^^^^^^
20
21 These allow manipulation of data being stored in one layout in the source
22 yaml for convenience and/or clarity, to another format to be processed by
23 the targeted module instead of requiring all modules in JJB being capable
24 of supporting multiple input formats.
25
26 The tag ``!join:`` will treat the first element of the following list as
27 the delimiter to use, when joining the remaining elements into a string
28 and returning a single string to be consumed by the specified module option.
29
30 This allows users to maintain elements of data in a list structure for ease
31 of review/maintenance, and have the yaml parser convert it to a string for
32 consumption as any argument for modules. The main expected use case is to
33 allow for generic plugin data such as shell properties to be populated from
34 a list construct which the yaml parser converts to a single string, instead
35 of trying to support this within the module code which would require a
36 templating engine similar to Jinja.
37
38 Generic Example:
39
40 .. literalinclude:: /../../tests/loader/fixtures/joinlists.yaml
41
42
43 Environment Inject:
44
45 .. literalinclude:: /../../tests/yamlparser/job_fixtures/string_join.yaml
46
47
48 While this mechanism can also be used items where delimiters are supported by
49 the module, that should be considered a bug that the existing code doesn't
50 handle being provided a list and delimiter to perform the correct conversion
51 for you. Should you discover a module that takes arguments with delimiters and
52 the existing JJB codebase does not handle accepting lists, then this can be
53 used as a temporary solution in place of using very long strings:
54
55 Extended Params Example:
56
57 .. literalinclude::
58 /../../tests/parameters/fixtures/extended-choice-param-full.yaml
59
60
61 Inclusion Tags
62 ^^^^^^^^^^^^^^
63
64 These allow inclusion of arbitrary files as a method of having blocks of data
65 managed separately to the yaml job configurations. A specific usage of this is
66 inlining scripts contained in separate files, although such tags may also be
67 used to simplify usage of macros or job templates.
68
69 The tag ``!include:`` will treat the following string as file which should be
70 parsed as yaml configuration data.
71
72 Example:
73
74 .. literalinclude:: /../../tests/loader/fixtures/include001.yaml
75
76 contents of include001.yaml.inc:
77
78 .. literalinclude:: /../../tests/loader/fixtures/include001.yaml.inc
79
80
81 The tag ``!include-raw-expand:`` will treat the given string or list of strings
82 as filenames to be opened as one or more data blob, which should be read into
83 the calling yaml construct without any further parsing. Any data in a file
84 included through this tag, will be treated as string data.
85
86 It will expand variables inside the file. If your file contains curly braces,
87 you should double them. Or, you can use tag ``!include-raw-verbatim:``, which
88 does not substitute variables.
89
90 Examples:
91
92 .. literalinclude::
93 /../../tests/loader/fixtures/include-raw-verbatim-template.yaml
94
95 contents of include-raw-hello-world.sh:
96
97 .. literalinclude::
98 /../../tests/loader/fixtures/include-raw-hello-world.sh
99
100 contents of include-raw-vars.sh:
101
102 .. literalinclude::
103 /../../tests/loader/fixtures/include-raw-vars.sh
104
105 Using a list of files:
106
107 .. literalinclude::
108 /../../tests/loader/fixtures/include-raw-verbatim-multi-template.yaml
109
110
111 For all the multi file includes, the files are simply appended using a newline
112 character.
113
114
115 You can also use variables in included file paths.
116
117 Example:
118
119 .. literalinclude:: /../../tests/yamlparser/job_fixtures/lazy-load-jobs001.yaml
120
121 with variable substitution inside included files:
122
123 .. literalinclude:: /../../tests/yamlparser/job_fixtures/lazy-load-with-variables.yaml
124
125 using a list of files:
126
127 .. literalinclude::
128 /../../tests/yamlparser/job_fixtures/lazy-load-jobs-multi001.yaml
129
130
131 The tag ``!include-jinja2:`` will treat the given string or list of strings as
132 filenames to be opened as Jinja2 templates, which should be rendered to a
133 string and included in the calling YAML construct. (This is analogous to the
134 templating that will happen with ``!include-raw``.)
135
136 Examples:
137
138 .. literalinclude:: /../../tests/yamlparser/job_fixtures/jinja01.yaml
139
140 contents of jinja01.yaml.inc:
141
142 .. literalinclude:: /../../tests/yamlparser/job_fixtures/jinja01.yaml.inc
143
144
145 The tag ``!j2:`` takes a string and treats it as a Jinja2 template. It will be
146 rendered (with the variables in that context) and included in the calling YAML
147 construct.
148
149 Examples:
150
151 .. literalinclude:: /../../tests/yamlparser/job_fixtures/jinja-string01.yaml
152
153 The tag ``!j2-yaml:`` is similar to the ``!j2:`` tag, just that it loads the
154 Jinja-rendered string as YAML and embeds it in the calling YAML construct. This
155 provides a very flexible and convenient way of generating pieces of YAML
156 structures. One of use cases is defining complex YAML structures with much
157 simpler configuration, without any duplication.
158
159 Examples:
160
161 .. literalinclude:: /../../tests/yamlparser/job_fixtures/jinja-yaml01.yaml
162
163 Another use case is controlling lists dynamically, like conditionally adding
164 list elements based on project configuration.
165
166 Examples:
167
168 .. literalinclude:: /../../tests/yamlparser/job_fixtures/jinja-yaml02.yaml
169
170 """
171
172 import abc
173 import importlib
174 import logging
175 import traceback
176 import sys
177 from pathlib import Path
178
179 import jinja2
180 import jinja2.meta
181 import yaml
182
183 from .errors import Context, JenkinsJobsException
184 from .loc_loader import LocList
185 from .position import Pos
186 from .formatter import CustomFormatter, enum_str_format_required_params
187
188 if sys.version_info >= (3, 8):
189 from functools import cached_property
190 else:
191 from .cached_property import cached_property
192
193 logger = logging.getLogger(__name__)
194
195
196 class BaseYamlObject(metaclass=abc.ABCMeta):
197 @staticmethod
198 def path_list_from_node(loader, node):
199 if isinstance(node, yaml.ScalarNode):
200 return LocList(
201 [loader.construct_yaml_str(node)],
202 value_pos=[loader.pos_from_node(node)],
203 )
204 elif isinstance(node, yaml.SequenceNode):
205 return LocList(
206 loader.construct_sequence(node),
207 value_pos=[loader.pos_from_node(n) for n in node.value],
208 )
209 else:
210 raise JenkinsJobsException(
211 f"Expected either a sequence or scalar node, but found {node.id}",
212 pos=loader.pos_from_node(node),
213 )
214
215 @classmethod
216 def from_yaml(cls, loader, node):
217 value = loader.construct_yaml_str(node)
218 return cls(loader.jjb_config, loader, loader.pos_from_node(node), value)
219
220 def __init__(self, jjb_config, loader, pos):
221 self._search_path = jjb_config.yamlparser["include_path"].copy()
222 if loader.source_dir:
223 # Loaded from a file, find includes beside it too.
224 self._search_path.append(loader.source_dir)
225 self._filter_modules = jjb_config.yamlparser["filter_modules"].copy()
226 self._loader = loader
227 self._pos = pos
228 allow_empty = jjb_config.yamlparser["allow_empty_variables"]
229 self._formatter = CustomFormatter(allow_empty)
230
231 @abc.abstractmethod
232 def expand(self, expander, params):
233 """Expand object and substitute template parameters"""
234 pass
235
236 def _find_file(self, rel_path, pos):
237 search_path = self._search_path
238 if "." not in search_path:
239 search_path.append(".")
240 dir_list = [Path(d).expanduser() for d in self._search_path]
241 for dir in dir_list:
242 candidate = dir.joinpath(rel_path)
243 if candidate.is_file():
244 logger.debug("Including file %r from path %r", str(rel_path), str(dir))
245 return candidate
246 dir_list_str = ",".join(str(d) for d in dir_list)
247 raise JenkinsJobsException(
248 f"File {rel_path} does not exist in any of include directories: {dir_list_str}",
249 pos=pos,
250 )
251
252 def _expand_path_list(self, path_list, *args):
253 for idx, path in enumerate(path_list):
254 yield self._expand_path(path, path_list.value_pos[idx], *args)
255
256
257 class J2BaseYamlObject(BaseYamlObject):
258 def __init__(self, jjb_config, loader, pos):
259 super().__init__(jjb_config, loader, pos)
260 self._filters = {}
261 for module_name in self._filter_modules:
262 module = importlib.import_module(module_name)
263 self._filters.update(module.FILTERS)
264 self._jinja2_env = jinja2.Environment(
265 loader=jinja2.FileSystemLoader(self._search_path),
266 undefined=jinja2.StrictUndefined,
267 )
268 self._jinja2_env.filters.update(self._filters)
269
270 def _render_template(self, pos, template_text, template, params):
271 try:
272 return template.render(params)
273 except jinja2.UndefinedError as x:
274 # Jinja2 adds fake traceback entry with template line number.
275 tb = traceback.extract_tb(x.__traceback__)
276 line_ofs = tb[-1].lineno - 1 # traceback lineno starts with 1.
277 lines = template_text.splitlines()
278 start_ofs = pos.body.index(lines[0])
279 # Examples for pre_pad: '!j2: \n<indent spaces>', '!j2: '.
280 pre_pad = pos.body[:start_ofs]
281 # Shift position to reflect template position inside yaml file:
282 if "\n" in pre_pad:
283 pos = pos.with_offset(line_ofs=1)
284 column_ofs = 0
285 else:
286 column_ofs = start_ofs
287 # Move position to error inside template:
288 pos = pos.with_offset(line_ofs, column_ofs)
289 pos = pos.with_contents_start()
290 if len(template_text) > 40:
291 text = template_text[:40] + "..."
292 else:
293 text = template_text
294 context = Context(f"While formatting jinja2 template {text!r}", self._pos)
295 raise JenkinsJobsException(str(x), pos=pos, ctx=[context])
296
297
298 class J2Template(J2BaseYamlObject):
299 def __init__(self, jjb_config, loader, pos, template_text):
300 super().__init__(jjb_config, loader, pos)
301 self._template_text = template_text
302 self._template = self._jinja2_env.from_string(template_text)
303
304 def _params_from_referenced_templates(self, template_text):
305 """
306 Find recursively undeclared jinja2 variables from any
307 (nested) included template(s)
308 """
309 required_params = set()
310 ast = self._jinja2_env.parse(template_text)
311 for rt in jinja2.meta.find_referenced_templates(ast):
312 # recursive call to find params also from nested includes
313 template_text = Path(self._find_file(rt, 0)).read_text()
314 required_params.update(
315 self._params_from_referenced_templates(template_text)
316 )
317 required_params.update(jinja2.meta.find_undeclared_variables(ast))
318 return required_params
319
320 @cached_property
321 def required_params(self):
322 return self._params_from_referenced_templates(self._template_text)
323
324 def _render(self, params):
325 return self._render_template(
326 self._pos, self._template_text, self._template, params
327 )
328
329
330 class J2String(J2Template):
331 yaml_tag = "!j2:"
332
333 def expand(self, expander, params):
334 return self._render(params)
335
336
337 class J2Yaml(J2Template):
338 yaml_tag = "!j2-yaml:"
339
340 def expand(self, expander, params):
341 text = self._render(params)
342 data = self._loader.load(
343 text, source_path="<expanded j2-yaml>", source_dir=self._loader.source_dir
344 )
345 try:
346 return expander.expand(data, params)
347 except JenkinsJobsException as x:
348 raise x.with_context("In expanded !j2-yaml:", self._pos)
349
350
351 class IncludeJinja2(J2BaseYamlObject):
352 yaml_tag = "!include-jinja2:"
353
354 @classmethod
355 def from_yaml(cls, loader, node):
356 path_list = cls.path_list_from_node(loader, node)
357 return cls(loader.jjb_config, loader, loader.pos_from_node(node), path_list)
358
359 def __init__(self, jjb_config, loader, pos, path_list):
360 super().__init__(jjb_config, loader, pos)
361 self._path_list = path_list
362
363 @property
364 def required_params(self):
365 return []
366
367 def expand(self, expander, params):
368 return "\n".join(self._expand_path_list(self._path_list, expander, params))
369
370 def _expand_path(self, path_template, pos, expander, params):
371 rel_path = self._formatter.format(path_template, **params)
372 full_path = self._find_file(rel_path, pos)
373 template_text = full_path.read_text()
374 template = self._jinja2_env.from_string(template_text)
375 pos = Pos.from_file(full_path, template_text)
376 try:
377 return self._render_template(pos, template_text, template, params)
378 except JenkinsJobsException as x:
379 raise x.with_context(f"In included file {str(full_path)!r}", pos=self._pos)
380
381
382 class IncludeBaseObject(BaseYamlObject):
383 @classmethod
384 def from_yaml(cls, loader, node):
385 path_list = cls.path_list_from_node(loader, node)
386 return cls(loader.jjb_config, loader, loader.pos_from_node(node), path_list)
387
388 def __init__(self, jjb_config, loader, pos, path_list):
389 super().__init__(jjb_config, loader, pos)
390 self._path_list = path_list
391
392 @property
393 def required_params(self):
394 for idx, path in enumerate(self._path_list):
395 yield from enum_str_format_required_params(
396 path, pos=self._path_list.value_pos[idx]
397 )
398
399
400 class YamlInclude(IncludeBaseObject):
401 yaml_tag = "!include:"
402
403 def expand(self, expander, params):
404 yaml_list = list(self._expand_path_list(self._path_list, expander, params))
405 if len(yaml_list) == 1:
406 return yaml_list[0]
407 else:
408 return "\n".join(yaml_list)
409
410 def _expand_path(self, path_template, pos, expander, params):
411 rel_path = self._formatter.format(path_template, **params)
412 full_path = self._find_file(rel_path, pos)
413 data = self._loader.load_path(full_path)
414 try:
415 return expander.expand(data, params)
416 except JenkinsJobsException as x:
417 raise x.with_context(f"In included file {str(full_path)!r}", pos=self._pos)
418
419
420 class IncludeRawBase(IncludeBaseObject):
421 def expand(self, expander, params):
422 return "\n".join(self._expand_path_list(self._path_list, params))
423
424
425 class IncludeRawExpand(IncludeRawBase):
426 yaml_tag = "!include-raw-expand:"
427
428 def _expand_path(self, rel_path_template, pos, params):
429 rel_path = self._formatter.format(rel_path_template, **params)
430 full_path = self._find_file(rel_path, pos)
431 template = full_path.read_text()
432 try:
433 return self._formatter.format(template, **params)
434 except JenkinsJobsException as x:
435 raise x.with_context(f"In included file {str(full_path)!r}", pos=self._pos)
436
437
438 class IncludeRawVerbatim(IncludeRawBase):
439 yaml_tag = "!include-raw-verbatim:"
440
441 def _expand_path(self, rel_path_template, pos, params):
442 rel_path = self._formatter.format(rel_path_template, **params)
443 full_path = self._find_file(rel_path, pos)
444 return full_path.read_text()
445
446
447 class YamlListJoin:
448 yaml_tag = "!join:"
449
450 @classmethod
451 def from_yaml(cls, loader, node):
452 value = loader.construct_sequence(node, deep=True)
453 if len(value) != 2:
454 raise JenkinsJobsException(
455 "Join value should contain 2 elements: delimiter and string list,"
456 f" but contains {len(value)} elements: {value!r}",
457 pos=loader.pos_from_node(node),
458 )
459 delimiter, seq = value
460 return delimiter.join(seq)