| 1 |
NAME |
| 2 |
Makefile::DOM - Simple DOM parser for Makefiles |
| 3 |
|
| 4 |
VERSION |
| 5 |
This document describes Makefile::DOM 0.008 released on 18 November |
| 6 |
2014. |
| 7 |
|
| 8 |
DESCRIPTION |
| •9 |
This library can serve as an advanced lexer for (GNU) makefiles. It |
| 10 |
parses makefiles as "documents" and the parsing is lossless. The results |
| 11 |
are data structures similar to DOM trees. The DOM trees hold every |
| 12 |
single bit of the information in the original input files, including |
| 13 |
white spaces, blank lines and makefile comments. That means it's |
| 14 |
possible to reproduce the original makefiles from the DOM trees. In |
| 15 |
addition, each node of the DOM trees is modifiable and so is the whole |
| 16 |
tree, just like the PPI module used for Perl source parsing and the |
| 17 |
HTML::TreeBuilder module used for parsing HTML source. |
| 18 |
|
| 19 |
If you're looking for a true GNU make parser that generates an AST, |
| 20 |
please see Makefile::Parser::GmakeDB instead. |
| 21 |
|
| 22 |
The interface of "Makefile::DOM" mimics the API design of PPI. In fact, |
| 23 |
I've directly stolen the source code and POD documentation of PPI::Node, |
| 24 |
PPI::Element, and PPI::Dumper, with the full permission from the author |
| 25 |
of PPI, Adam Kennedy. |
| 26 |
|
| 27 |
"Makefile::DOM" tries to be independent of specific makefile's syntax. |
| 28 |
The same set of DOM node types is supposed to get shared by different |
| 29 |
makefile DOM generators. For example, MDOM::Document::Gmake parses GNU |
| 30 |
makefiles and returns an instance of MDOM::Document, i.e., the root of |
| 31 |
the DOM tree while the NMAKE makefile lexer in the future, |
| 32 |
"MDOM::Document::Nmake", also returns instances of the MDOM::Document |
| 33 |
class. Later, I'll also consider adding support for dmake and bsdmake. |
| 34 |
|
| 35 |
Structure of the DOM |
| 36 |
Makefile DOM (MDOM) is a structured set of a series of data types. They |
| 37 |
provide a flexible document model conformed to the makefile syntax. |
| 38 |
Below is a complete list of the 19 MDOM classes in the current |
| 39 |
implementation where the indentation indicates the class inheritance |
| 40 |
relationships. |
| 41 |
|
| 42 |
MDOM::Element |
| 43 |
MDOM::Node |
| 44 |
MDOM::Unknown |
| 45 |
MDOM::Assignment |
| 46 |
MDOM::Command |
| 47 |
MDOM::Directive |
| 48 |
MDOM::Document |
| 49 |
MDOM::Document::Gmake |
| 50 |
MDOM::Rule |
| 51 |
MDOM::Rule::Simple |
| 52 |
MDOM::Rule::StaticPattern |
| 53 |
MDOM::Token |
| 54 |
MDOM::Token::Bare |
| 55 |
MDOM::Token::Comment |
| 56 |
MDOM::Token::Continuation |
| 57 |
MDOM::Token::Interpolation |
| 58 |
MDOM::Token::Modifier |
| 59 |
MDOM::Token::Separator |
| 60 |
MDOM::Token::Whitespace |
| 61 |
|
| 62 |
It's not hard to see that all of the MDOM classes inherit from the |
| 63 |
MDOM::Element class. MDOM::Token and MDOM::Node are its direct children. |
| 64 |
The former represents a string token which is atomic from the |
| 65 |
perspective of the lexer while the latter represents a structured node, |
| 66 |
which usually has one or more children, and serves as the container for |
| 67 |
other DOM::Element objects. |
| 68 |
|
| 69 |
Next we'll show a few examples to demonstrate how to map DOM trees to |
| 70 |
particular makefiles. |
| 71 |
|
| 72 |
Case 1 |
| 73 |
Consider the following simple "hello, world" makefile: |
| 74 |
|
| 75 |
all : ; echo "hello, world" |
| 76 |
|
| 77 |
We can use the MDOM::Dumper class provided by Makefile::DOM to dump |
| 78 |
out the internal structure of its corresponding MDOM tree: |
| 79 |
|
| 80 |
MDOM::Document::Gmake |
| 81 |
MDOM::Rule::Simple |
| 82 |
MDOM::Token::Bare 'all' |
| 83 |
MDOM::Token::Whitespace ' ' |
| 84 |
MDOM::Token::Separator ':' |
| 85 |
MDOM::Token::Whitespace ' ' |
| 86 |
MDOM::Command |
| 87 |
MDOM::Token::Separator ';' |
| 88 |
MDOM::Token::Whitespace ' ' |
| 89 |
MDOM::Token::Bare 'echo "hello, world"' |
| 90 |
MDOM::Token::Whitespace '\n' |
| 91 |
|
| 92 |
In this example, speparators ":" and ";" are all instances of the |
| 93 |
MDOM::Token::Separator class while spaces and new line characters |
| 94 |
are all represented as MDOM::Token::Whitespace. The other two leaf |
| 95 |
nodes, "all" and "echo "hello, world"" both belong to |
| 96 |
MDOM::Token::Bare. |
| 97 |
|
| 98 |
It's worth mentioning that, the space characters in the rule command |
| 99 |
"echo "hello, world"" were not represented as |
| 100 |
MDOM::Token::Whitespace. That's because in makefiles, the spaces in |
| 101 |
commands do not make any sense to "make" in syntax; those spaces are |
| 102 |
usually sent to shell programs verbatim. Therefore, the DOM parser |
| 103 |
does not try to recognize those spaces specifially so as to reduce |
| 104 |
memory use and the number of nodes. However, leading spaces and |
| 105 |
trailing new lines will still be recognized as |
| 106 |
MDOM::Token::Whitespace. |
| 107 |
|
| 108 |
On a higher level, it's a MDOM::Rule::Simple instance holding |
| 109 |
several "Token" and one MDOM::Command. On the highest level, it's |
| 110 |
the root node of the whole DOM tree, i.e., an instance of |
| 111 |
MDOM::Document::Gmake. |
| 112 |
|
| 113 |
Case 2 |
| 114 |
Below is a relatively complex example: |
| 115 |
|
| 116 |
a: foo.c bar.h $(baz) # hello! |
| 117 |
@echo ... |
| 118 |
|
| 119 |
It's corresponding DOM structure is |
| 120 |
|
| 121 |
MDOM::Document::Gmake |
| 122 |
MDOM::Rule::Simple |
| 123 |
MDOM::Token::Bare 'a' |
| 124 |
MDOM::Token::Separator ':' |
| 125 |
MDOM::Token::Whitespace ' ' |
| 126 |
MDOM::Token::Bare 'foo.c' |
| 127 |
MDOM::Token::Whitespace ' ' |
| 128 |
MDOM::Token::Bare 'bar.h' |
| 129 |
MDOM::Token::Whitespace '\t' |
| 130 |
MDOM::Token::Interpolation '$(baz)' |
| 131 |
MDOM::Token::Whitespace ' ' |
| 132 |
MDOM::Token::Comment '# hello!' |
| 133 |
MDOM::Token::Whitespace '\n' |
| 134 |
MDOM::Command |
| 135 |
MDOM::Token::Separator '\t' |
| 136 |
MDOM::Token::Modifier '@' |
| 137 |
MDOM::Token::Bare 'echo ...' |
| 138 |
MDOM::Token::Whitespace '\n' |
| 139 |
|
| 140 |
Compared to the previous example, here appears several new node |
| 141 |
types. |
| 142 |
|
| 143 |
The variable interpolation "$(baz)" on the first line of the |
| 144 |
original makefile corresponds to a MDOM::Token::Interpolation node |
| 145 |
in its MDOM tree. Similarly, the comment "# hello" corresponds to a |
| 146 |
MDOM::Token::Comment node. |
| 147 |
|
| 148 |
On the second line, the rule command indented by a tab character is |
| 149 |
still represented by a MDOM::Command object. Its first child node |
| 150 |
(or its first element) is also an MDOM::Token::Seperator instance |
| 151 |
corresponding to that tab. The command modifier "@" follows the |
| 152 |
"Separator" immediately, which is of type MDOM::Token::Modifier. |
| 153 |
|
| 154 |
Case 3 |
| 155 |
Now let's study a sample makefile with various global structures: |
| 156 |
|
| 157 |
a: b |
| 158 |
foo = bar |
| 159 |
# hello! |
| 160 |
|
| 161 |
Here on the top level, there are three language structures: one rule |
| 162 |
""a: b"", one assignment statement "foo = bar", and one comment "# |
| 163 |
hello!". |
| 164 |
|
| 165 |
Its MDOM tree is shown below: |
| 166 |
|
| 167 |
MDOM::Document::Gmake |
| 168 |
MDOM::Rule::Simple |
| 169 |
MDOM::Token::Bare 'a' |
| 170 |
MDOM::Token::Separator ':' |
| 171 |
MDOM::Token::Whitespace ' ' |
| 172 |
MDOM::Token::Bare 'b' |
| 173 |
MDOM::Token::Whitespace '\n' |
| 174 |
MDOM::Assignment |
| 175 |
MDOM::Token::Bare 'foo' |
| 176 |
MDOM::Token::Whitespace ' ' |
| 177 |
MDOM::Token::Separator '=' |
| 178 |
MDOM::Token::Whitespace ' ' |
| 179 |
MDOM::Token::Bare 'bar' |
| 180 |
MDOM::Token::Whitespace '\n' |
| 181 |
MDOM::Token::Whitespace '\t' |
| 182 |
MDOM::Token::Comment '# hello!' |
| 183 |
MDOM::Token::Whitespace '\n' |
| 184 |
|
| 185 |
We can see that below the root node MDOM::Document::Gmake, there are |
| 186 |
MDOM::Rule::Simple, MDOM::Assignment, and MDOM::Comment three |
| 187 |
elements, as well as two MDOM::Token::Whitespace objects. |
| 188 |
|
| 189 |
It can be observed from the examples above that the MDOM representation |
| 190 |
for the makefile's lexical elements is rather loose. It only provides |
| 191 |
very limited structural representation instead of making a bad guess. |
| 192 |
|
| 193 |
OPERATIONS FOR MDOM TREES |
| 194 |
Generating an MDOM tree from a GNU makefile only requires two lines of |
| 195 |
Perl code: |
| 196 |
|
| 197 |
use MDOM::Document::Gmake; |
| 198 |
my $dom = MDOM::Document::Gmake->new('Makefile'); |
| 199 |
|
| 200 |
If the makefile source code being parsed is already stored in a Perl |
| 201 |
variable, say, $var, then we can construct an MDOM via the following |
| 202 |
code: |
| 203 |
|
| 204 |
my $dom = MDOM::Document::Gmake->new(\$var); |
| 205 |
|
| 206 |
Now $dom becomes the reference to the root of the MDOM tree and its type |
| 207 |
is now MDOM::Document::Gmake, which is also an instance of the |
| 208 |
MDOM::Node class. |
| 209 |
|
| 210 |
Just as mentioned above, "MDOM::Node" is the container for other |
| 211 |
MDOM::Element instances. So we can retrieve some element node's value |
| 212 |
via its "child" method: |
| 213 |
|
| 214 |
$node = $dom->child(3); |
| 215 |
# or $node = $dom->elements(0); |
| 216 |
|
| 217 |
And we may also use the "elements" method to obtain the values of all |
| 218 |
the nodes: |
| 219 |
|
| 220 |
@elems = $dom->elements; |
| 221 |
|
| 222 |
For every MDOM node, its corresponding makefile source can be generated |
| 223 |
by invoking its "content" method. |
| 224 |
|
| 225 |
BUGS AND TODO |
| 226 |
The current implementation of the MDOM::Document::Gmake lexer is based |
| 227 |
on a hand-written state machie. Although the efficiency of the engine is |
| 228 |
not bad, the code is rather complicated and messy, which hurts both |
| 229 |
extensibility and maintanabilty. So it's expected to rewrite the parser |
| 230 |
using some grammatical tools like the Perl 6 regex engine |
| 231 |
Pugs::Compiler::Rule or a yacc-style one like Parse::Yapp. |
| 232 |
|
| 233 |
SOURCE REPOSITORY |
| 234 |
You can always get the latest source code of this module from its GitHub |
| 235 |
repository: |
| 236 |
|
| 237 |
<http://github.com/agentzh/makefile-dom-pm> |
| 238 |
|
| 239 |
If you want a commit bit, please let me know. |
| 240 |
|
| 241 |
AUTHOR |
| 242 |
Yichun "agentzh" Zhang (章亦春) <[email protected]> |
| 243 |
|
| 244 |
COPYRIGHT |
| 245 |
Copyright 2006-2014 by Yichun "agentzh" Zhang (章亦春). |
| 246 |
|
| 247 |
This library is free software; you can redistribute it and/or modify it |
| 248 |
under the same terms as Perl itself. |
| 249 |
|
| 250 |
SEE ALSO |
| 251 |
MDOM::Document, MDOM::Document::Gmake, PPI, Makefile::Parser::GmakeDB, |
| 252 |
makesimple. |
| 253 |
|