| 1 |
Puppet FileMapper |
| 2 |
================= |
| 3 |
|
| 4 |
Synopsis |
| 5 |
-------- |
| 6 |
|
| 7 |
Map files to resources and back with this handy dandy mixin! |
| 8 |
|
| 9 |
Documentation is available at [http://adrienthebo.github.com/puppet-filemapper/](http://adrienthebo.github.com/puppet-filemapper/) |
| 10 |
|
| 11 |
Travis Test status: [](https://travis-ci.org/adrienthebo/puppet-filemapper) |
| 12 |
|
| 13 |
Description |
| 14 |
----------- |
| 15 |
|
| 16 |
Things that are harder than they should be: |
| 17 |
|
| 18 |
* Acquiring a pet monkey |
| 19 |
* Getting anywhere in Los Angeles |
| 20 |
* Understanding the ParsedFile provider |
| 21 |
* Writing Puppet providers that directly manipulate files |
| 22 |
|
| 23 |
The solution for this is to completely bypass parsing in any sort of base |
| 24 |
provider, and delegate the role of parsing and generating to including classes. |
| 25 |
|
| 26 |
You figure out how to parse and write the file, and this will do the rest. |
| 27 |
|
| 28 |
Synopsis of implementation requirements |
| 29 |
--------------------------------------- |
| 30 |
|
| 31 |
Providers using the Filemapper extension need to implement the following |
| 32 |
methods. |
| 33 |
|
| 34 |
### `self.target_files` |
| 35 |
|
| 36 |
This should return an array of filenames specifying which files should be |
| 37 |
prefetched. |
| 38 |
|
| 39 |
### `self.parse_file(filename, file_contents)` |
| 40 |
|
| 41 |
This should take two values, a string containing the file name, and a string |
| 42 |
containing the contents of the file. It should return an array of hashes, |
| 43 |
where each hash represents {property => value} pairs. |
| 44 |
|
| 45 |
### `select_file` |
| 46 |
|
| 47 |
This is a provider instance method. It should return a string containing the |
| 48 |
filename that the provider should be flushed to. |
| 49 |
|
| 50 |
### `self.format_file(filename, providers)` |
| 51 |
|
| 52 |
This should take two values, a string containing the file name to be flushed, |
| 53 |
and an array of providers that should be flushed to this file. It should return |
| 54 |
a string containing the contents of the file to be written. |
| 55 |
|
| 56 |
Synopsis of optional implementation hooks |
| 57 |
----------------------------------------- |
| 58 |
|
| 59 |
### `self.pre_flush_hook(filename)` and `self.post_flush_hook(filename)` |
| 60 |
|
| 61 |
These methods can be implemented to add behavior right before and right after |
| 62 |
filesystem operations. Both methods take a single argument, a string |
| 63 |
containing the name of the file to be flushed. |
| 64 |
|
| 65 |
If `self.pre_flush_hook` raises an exception, the flush will not occur and the |
| 66 |
provider will be marked as failed and will refuse to perform any more flushes. |
| 67 |
If some sort of critical error occurred, this can force the provider to error |
| 68 |
out before it starts stomping on files. |
| 69 |
|
| 70 |
`self.post_flush_hook` is guaranteed to run after any filesystem operations |
| 71 |
occur. This can be used for recovery if something goes wrong during the flush. |
| 72 |
If this method raises an exception, the provider will be marked as failed and |
| 73 |
will refuse to perform any more flushes. |
| 74 |
|
| 75 |
Removing empty files |
| 76 |
-------------------- |
| 77 |
|
| 78 |
If a file is empty, it's often reasonable to just delete it. The Filemapper |
| 79 |
mixin implements `attr_accessor :unlink_empty_files`. If that value is set to |
| 80 |
true, then if `self.format_file` returns the empty string then the file will be |
| 81 |
deleted from the file system. |
| 82 |
|
| 83 |
How it works |
| 84 |
------------ |
| 85 |
|
| 86 |
[transaction]: http://somethingsinistral.net/blog/reading-puppet-the-transaction/ |
| 87 |
|
| 88 |
The Filemapper extension takes advantage of hooks within the |
| 89 |
[Transaction][transaction] to reduce the number of reads and writes needed to |
| 90 |
perform operations. |
| 91 |
|
| 92 |
### prefetching |
| 93 |
|
| 94 |
When a catalog is being applied, providers can define the `prefetch` method to |
| 95 |
load all resources before runtime. The Filemapper extension uses this method to |
| 96 |
preemptively read all files that the provider requires, and generates and stores |
| 97 |
the state of the requested resources. This means that if you have a few thousand |
| 98 |
resources in 20 files, you only need to do 20 reads for the entire Puppet run. |
| 99 |
|
| 100 |
### post-evaluation flushing |
| 101 |
|
| 102 |
When resources are normally evaluated, each time a property is synchronized it's |
| 103 |
expected that an action will be run right then. The Filemapper extension instead |
| 104 |
records all the requested changes and defers operating on them. When the |
| 105 |
resource is finished, it will be flushed, at which time all of the requested |
| 106 |
changes will be applied in one pass. Given a resource with 10 properties, all of |
| 107 |
which are out of sync, the file will be written only once. If no properties are |
| 108 |
out of sync, the file will be untouched. |
| 109 |
|
| 110 |
To ensure that the system state matches what Puppet thinks is going on, any file |
| 111 |
that has changed resources will be re-written after each resource is flushed. |
| 112 |
That means that if you have 20 resources out of sync, that file will have to be |
| 113 |
written 20 times. While it's technically possible to write the file in a single |
| 114 |
pass, this means that some resources will be applied either early or late, which |
| 115 |
utterly smashes POLA. |
| 116 |
|
| 117 |
### Use on the command line |
| 118 |
|
| 119 |
The Filemapper extension implements the `instances` method, which means that you |
| 120 |
can use the `puppet resource` command to interact with the associated provider |
| 121 |
without having to perform a full blown Puppet run. |
| 122 |
|
| 123 |
### Selecting files to load |
| 124 |
|
| 125 |
In order to provide prefetching and `puppet resource` in a clean manner, the |
| 126 |
Filemapper extension has to have a full list of what files to read. Implementing |
| 127 |
classes need to implement the `target_files` method which returns a list of |
| 128 |
files to read. The implementation is entirely up to the implementing class; it |
| 129 |
can return a single file every time, such as "/etc/inittab", or it can generate |
| 130 |
that information on the fly, by returning `Dir["/etc/sysconfig/network/ifcfg-*"]`. |
| 131 |
Basically, files that will be used as a source of data can be as complex or |
| 132 |
simple as you need. |
| 133 |
|
| 134 |
### Writing back files |
| 135 |
|
| 136 |
In a similar vein, resources can be written back to files in whatever method you |
| 137 |
need. Implementing classes need to implement the *instance method* `#select_file` |
| 138 |
so that when that resource is changed, the correct file is modified. |
| 139 |
|
| 140 |
### Parsing |
| 141 |
|
| 142 |
When parsing a file, the implementing class needs to implement the `parse_file` |
| 143 |
method. It will get the name of the file being parsed as well as the contents. |
| 144 |
It can parse this file in whatever manner needed, and should return an array of |
| 145 |
any provider instances generated. If the file only contains a single provider |
| 146 |
instance, then just wrap that instance in an array. |
| 147 |
|
| 148 |
### Writing |
| 149 |
|
| 150 |
Whenever a file is marked as dirty, that is a resource associated with that file |
| 151 |
has changed, the `format_file` method will be called. The implementing class |
| 152 |
needs to implement a method that takes the filename and an array of provider |
| 153 |
instances associated with that file, and return a string. The method needs to |
| 154 |
determine how that file should be written to disk and then return the contents. |
| 155 |
This can be as complex as needed. |
| 156 |
|
| 157 |
Under no conditions should implementing classes modify any files directly. No, |
| 158 |
seriously, don't do it. The Filemapper extension uses the built in methods for |
| 159 |
modifying files, which will back up changed files to the filebucket. This is for |
| 160 |
your own safety, so if you bypass this then you are on your own. |
| 161 |
|
| 162 |
### Storing state outside of resources |
| 163 |
|
| 164 |
It's more or less expected that there will be no state outside of the provider |
| 165 |
instances, but there are plenty of cases where this could be the case. For |
| 166 |
instance, if one wanted to preserve the comments in a file but didn't directly |
| 167 |
associate them with resource attributes, the `parse_file` method can store data |
| 168 |
in an instance variable, such as `@comments = my_list_of_comments`. When |
| 169 |
formatting the file, the implementing class can read the `@comments` variable |
| 170 |
and re-add that data to the content that will be written back. |
| 171 |
|
| 172 |
Basically, you can store whatever data you need in these methods and pass things |
| 173 |
around to maintain more complex state. |
| 174 |
|
| 175 |
Using this sort of operation of reading outside state, you can theoretically |
| 176 |
have multiple Filemapper extensions that work on shared files. By communicating |
| 177 |
the state between them, you can manage multiple different resources in one file. |
| 178 |
**HOWEVER**, this will require careful communication, so don't take this sort of |
| 179 |
thing lightly. However, I don't thing that anything else in Puppet can provide |
| 180 |
this sort of behavior. YMMV. |
| 181 |
|
| 182 |
### Why a mixin? |
| 183 |
|
| 184 |
While the ParsedFile provider is supposed to be inherited, this class is a mixin |
| 185 |
and needs to be included. This is done because the Filemapper extension only |
| 186 |
*adds* behavior, and isn't really an object or entity in its own right. This way |
| 187 |
you can use the Filemapper extension while inheriting from something like the |
| 188 |
Puppet::Provider::Package provider. |
| 189 |
|
| 190 |
The Backstory |
| 191 |
------------- |
| 192 |
|
| 193 |
Managing Unix-ish systems generally means dealing with one of two things: |
| 194 |
|
| 195 |
1. Processes - starting them, stopping them, monitoring them, etc. |
| 196 |
1. Files - Creating them, editing, deleting them, specifying permissions, etc. |
| 197 |
|
| 198 |
Puppet has pretty good support in the provider layer for running commands, but |
| 199 |
the file manipulation layer has been lacking. The long-standing approach for |
| 200 |
manipulating files has been to select one of the following, and hope for the best. |
| 201 |
|
| 202 |
### Shipping flat files to the client |
| 203 |
|
| 204 |
Using the `File` resource to ship flat files is a really common solution, and |
| 205 |
it's very easy. It also has the finesse of a brick thrown through a window. |
| 206 |
There is very little customizability here, aside from the array notation for |
| 207 |
[specifying the `source` field](http://docs.puppetlabs.com/references/latest/type.html#file). |
| 208 |
|
| 209 |
### Using ERB templates to customize files |
| 210 |
|
| 211 |
The File resource can also take a content field, to which you can pass the |
| 212 |
output of a template. This allows more sophistication, but not much. It also |
| 213 |
adds more of a burden to your master; template rendering happens on the master |
| 214 |
and if you're doing really crazy number crunching then this pain will be |
| 215 |
centralized. |
| 216 |
|
| 217 |
### Using Augeas |
| 218 |
|
| 219 |
Augeas is a very powerful tool that allows you to manipulate files, and the |
| 220 |
`Augeas` type allows you to harness this inside of Puppet. However, it has a |
| 221 |
rather byzantine syntax, and is dependent on lenses being available. |
| 222 |
|
| 223 |
### Sed |
| 224 |
|
| 225 |
I personally love sed, but sed a file configuration management tool is not. |
| 226 |
|
| 227 |
### Using the ParsedFile provider |
| 228 |
|
| 229 |
[parsedfile]: https://github.com/puppetlabs/puppet/blob/2.7.19/lib/puppet/provider/parsedfile.rb "Puppet 2.7.19 - ParsedFile provider" |
| 230 |
|
| 231 |
Puppet has a provider extension called the [ParsedFile provider][parsedfile] |
| 232 |
that's used to manipulate text like crontabs and so forth. It also uses a number |
| 233 |
of advanced features in puppet, which makes it quite powerful. However, it's |
| 234 |
incredibly complex, tightly coupled with the FileParsing utility language, has |
| 235 |
tons of obscure and undocumented hooks that are the only way to do complex |
| 236 |
operations, and is entirely record based which makes it unsuitable for managing |
| 237 |
files that have complex structure. While it has basic support for managing |
| 238 |
multiple files, *basic* is the indicative word. |
| 239 |
|
| 240 |
- - - |
| 241 |
|
| 242 |
The Filemapper extension has been designed as a lower level alternative |
| 243 |
to the ParsedFile. |
| 244 |
|
| 245 |
Examples |
| 246 |
-------- |
| 247 |
|
| 248 |
[puppet-network]: https://github.com/adrienthebo/puppet-network |
| 249 |
|
| 250 |
The Filemapper extension was largely extracted out of the [puppet-network][puppet-network] |
| 251 |
module. That code base should display the weird edge cases that this extension |
| 252 |
handles. |