Debian Codegraph @

README.markdown

raw
1 Puppet FileMapper
2 =================
3
4 Synopsis
5 --------
6
7 Map files to resources and back with this handy dandy mixin!
8
9 Documentation is available at [http://adrienthebo.github.com/puppet-filemapper/](http://adrienthebo.github.com/puppet-filemapper/)
10
11 Travis Test status: [![Build Status](https://travis-ci.org/adrienthebo/puppet-filemapper.png)](https://travis-ci.org/adrienthebo/puppet-filemapper)
12
13 Description
14 -----------
15
16 Things that are harder than they should be:
17
18 * Acquiring a pet monkey
19 * Getting anywhere in Los Angeles
20 * Understanding the ParsedFile provider
21 * Writing Puppet providers that directly manipulate files
22
23 The solution for this is to completely bypass parsing in any sort of base
24 provider, and delegate the role of parsing and generating to including classes.
25
26 You figure out how to parse and write the file, and this will do the rest.
27
28 Synopsis of implementation requirements
29 ---------------------------------------
30
31 Providers using the Filemapper extension need to implement the following
32 methods.
33
34 ### `self.target_files`
35
36 This should return an array of filenames specifying which files should be
37 prefetched.
38
39 ### `self.parse_file(filename, file_contents)`
40
41 This should take two values, a string containing the file name, and a string
42 containing the contents of the file. It should return an array of hashes,
43 where each hash represents {property => value} pairs.
44
45 ### `select_file`
46
47 This is a provider instance method. It should return a string containing the
48 filename that the provider should be flushed to.
49
50 ### `self.format_file(filename, providers)`
51
52 This should take two values, a string containing the file name to be flushed,
53 and an array of providers that should be flushed to this file. It should return
54 a string containing the contents of the file to be written.
55
56 Synopsis of optional implementation hooks
57 -----------------------------------------
58
59 ### `self.pre_flush_hook(filename)` and `self.post_flush_hook(filename)`
60
61 These methods can be implemented to add behavior right before and right after
62 filesystem operations. Both methods take a single argument, a string
63 containing the name of the file to be flushed.
64
65 If `self.pre_flush_hook` raises an exception, the flush will not occur and the
66 provider will be marked as failed and will refuse to perform any more flushes.
67 If some sort of critical error occurred, this can force the provider to error
68 out before it starts stomping on files.
69
70 `self.post_flush_hook` is guaranteed to run after any filesystem operations
71 occur. This can be used for recovery if something goes wrong during the flush.
72 If this method raises an exception, the provider will be marked as failed and
73 will refuse to perform any more flushes.
74
75 Removing empty files
76 --------------------
77
78 If a file is empty, it's often reasonable to just delete it. The Filemapper
79 mixin implements `attr_accessor :unlink_empty_files`. If that value is set to
80 true, then if `self.format_file` returns the empty string then the file will be
81 deleted from the file system.
82
83 How it works
84 ------------
85
86 [transaction]: http://somethingsinistral.net/blog/reading-puppet-the-transaction/
87
88 The Filemapper extension takes advantage of hooks within the
89 [Transaction][transaction] to reduce the number of reads and writes needed to
90 perform operations.
91
92 ### prefetching
93
94 When a catalog is being applied, providers can define the `prefetch` method to
95 load all resources before runtime. The Filemapper extension uses this method to
96 preemptively read all files that the provider requires, and generates and stores
97 the state of the requested resources. This means that if you have a few thousand
98 resources in 20 files, you only need to do 20 reads for the entire Puppet run.
99
100 ### post-evaluation flushing
101
102 When resources are normally evaluated, each time a property is synchronized it's
103 expected that an action will be run right then. The Filemapper extension instead
104 records all the requested changes and defers operating on them. When the
105 resource is finished, it will be flushed, at which time all of the requested
106 changes will be applied in one pass. Given a resource with 10 properties, all of
107 which are out of sync, the file will be written only once. If no properties are
108 out of sync, the file will be untouched.
109
110 To ensure that the system state matches what Puppet thinks is going on, any file
111 that has changed resources will be re-written after each resource is flushed.
112 That means that if you have 20 resources out of sync, that file will have to be
113 written 20 times. While it's technically possible to write the file in a single
114 pass, this means that some resources will be applied either early or late, which
115 utterly smashes POLA.
116
117 ### Use on the command line
118
119 The Filemapper extension implements the `instances` method, which means that you
120 can use the `puppet resource` command to interact with the associated provider
121 without having to perform a full blown Puppet run.
122
123 ### Selecting files to load
124
125 In order to provide prefetching and `puppet resource` in a clean manner, the
126 Filemapper extension has to have a full list of what files to read. Implementing
127 classes need to implement the `target_files` method which returns a list of
128 files to read. The implementation is entirely up to the implementing class; it
129 can return a single file every time, such as "/etc/inittab", or it can generate
130 that information on the fly, by returning `Dir["/etc/sysconfig/network/ifcfg-*"]`.
131 Basically, files that will be used as a source of data can be as complex or
132 simple as you need.
133
134 ### Writing back files
135
136 In a similar vein, resources can be written back to files in whatever method you
137 need. Implementing classes need to implement the *instance method* `#select_file`
138 so that when that resource is changed, the correct file is modified.
139
140 ### Parsing
141
142 When parsing a file, the implementing class needs to implement the `parse_file`
143 method. It will get the name of the file being parsed as well as the contents.
144 It can parse this file in whatever manner needed, and should return an array of
145 any provider instances generated. If the file only contains a single provider
146 instance, then just wrap that instance in an array.
147
148 ### Writing
149
150 Whenever a file is marked as dirty, that is a resource associated with that file
151 has changed, the `format_file` method will be called. The implementing class
152 needs to implement a method that takes the filename and an array of provider
153 instances associated with that file, and return a string. The method needs to
154 determine how that file should be written to disk and then return the contents.
155 This can be as complex as needed.
156
157 Under no conditions should implementing classes modify any files directly. No,
158 seriously, don't do it. The Filemapper extension uses the built in methods for
159 modifying files, which will back up changed files to the filebucket. This is for
160 your own safety, so if you bypass this then you are on your own.
161
162 ### Storing state outside of resources
163
164 It's more or less expected that there will be no state outside of the provider
165 instances, but there are plenty of cases where this could be the case. For
166 instance, if one wanted to preserve the comments in a file but didn't directly
167 associate them with resource attributes, the `parse_file` method can store data
168 in an instance variable, such as `@comments = my_list_of_comments`. When
169 formatting the file, the implementing class can read the `@comments` variable
170 and re-add that data to the content that will be written back.
171
172 Basically, you can store whatever data you need in these methods and pass things
173 around to maintain more complex state.
174
175 Using this sort of operation of reading outside state, you can theoretically
176 have multiple Filemapper extensions that work on shared files. By communicating
177 the state between them, you can manage multiple different resources in one file.
178 **HOWEVER**, this will require careful communication, so don't take this sort of
179 thing lightly. However, I don't thing that anything else in Puppet can provide
180 this sort of behavior. YMMV.
181
182 ### Why a mixin?
183
184 While the ParsedFile provider is supposed to be inherited, this class is a mixin
185 and needs to be included. This is done because the Filemapper extension only
186 *adds* behavior, and isn't really an object or entity in its own right. This way
187 you can use the Filemapper extension while inheriting from something like the
188 Puppet::Provider::Package provider.
189
190 The Backstory
191 -------------
192
193 Managing Unix-ish systems generally means dealing with one of two things:
194
195 1. Processes - starting them, stopping them, monitoring them, etc.
196 1. Files - Creating them, editing, deleting them, specifying permissions, etc.
197
198 Puppet has pretty good support in the provider layer for running commands, but
199 the file manipulation layer has been lacking. The long-standing approach for
200 manipulating files has been to select one of the following, and hope for the best.
201
202 ### Shipping flat files to the client
203
204 Using the `File` resource to ship flat files is a really common solution, and
205 it's very easy. It also has the finesse of a brick thrown through a window.
206 There is very little customizability here, aside from the array notation for
207 [specifying the `source` field](http://docs.puppetlabs.com/references/latest/type.html#file).
208
209 ### Using ERB templates to customize files
210
211 The File resource can also take a content field, to which you can pass the
212 output of a template. This allows more sophistication, but not much. It also
213 adds more of a burden to your master; template rendering happens on the master
214 and if you're doing really crazy number crunching then this pain will be
215 centralized.
216
217 ### Using Augeas
218
219 Augeas is a very powerful tool that allows you to manipulate files, and the
220 `Augeas` type allows you to harness this inside of Puppet. However, it has a
221 rather byzantine syntax, and is dependent on lenses being available.
222
223 ### Sed
224
225 I personally love sed, but sed a file configuration management tool is not.
226
227 ### Using the ParsedFile provider
228
229 [parsedfile]: https://github.com/puppetlabs/puppet/blob/2.7.19/lib/puppet/provider/parsedfile.rb "Puppet 2.7.19 - ParsedFile provider"
230
231 Puppet has a provider extension called the [ParsedFile provider][parsedfile]
232 that's used to manipulate text like crontabs and so forth. It also uses a number
233 of advanced features in puppet, which makes it quite powerful. However, it's
234 incredibly complex, tightly coupled with the FileParsing utility language, has
235 tons of obscure and undocumented hooks that are the only way to do complex
236 operations, and is entirely record based which makes it unsuitable for managing
237 files that have complex structure. While it has basic support for managing
238 multiple files, *basic* is the indicative word.
239
240 - - -
241
242 The Filemapper extension has been designed as a lower level alternative
243 to the ParsedFile.
244
245 Examples
246 --------
247
248 [puppet-network]: https://github.com/adrienthebo/puppet-network
249
250 The Filemapper extension was largely extracted out of the [puppet-network][puppet-network]
251 module. That code base should display the weird edge cases that this extension
252 handles.