| 1 |
.\" -*- mode: troff; coding: utf-8 -*- |
| 2 |
.\" Automatically generated by Pod::Man 5.0102 (Pod::Simple 3.45) |
| 3 |
.\" |
| 4 |
.\" Standard preamble: |
| 5 |
.\" ======================================================================== |
| 6 |
.de Sp \" Vertical space (when we can't use .PP) |
| 7 |
.if t .sp .5v |
| 8 |
.if n .sp |
| 9 |
.. |
| 10 |
.de Vb \" Begin verbatim text |
| 11 |
.ft CW |
| 12 |
.nf |
| 13 |
.ne \\$1 |
| 14 |
.. |
| 15 |
.de Ve \" End verbatim text |
| 16 |
.ft R |
| 17 |
.fi |
| 18 |
.. |
| 19 |
.\" \*(C` and \*(C' are quotes in nroff, nothing in troff, for use with C<>. |
| 20 |
.ie n \{\ |
| 21 |
. ds C` "" |
| 22 |
. ds C' "" |
| 23 |
'br\} |
| 24 |
.el\{\ |
| 25 |
. ds C` |
| 26 |
. ds C' |
| 27 |
'br\} |
| 28 |
.\" |
| 29 |
.\" Escape single quotes in literal strings from groff's Unicode transform. |
| 30 |
.ie \n(.g .ds Aq \(aq |
| 31 |
.el .ds Aq ' |
| 32 |
.\" |
| 33 |
.\" If the F register is >0, we'll generate index entries on stderr for |
| 34 |
.\" titles (.TH), headers (.SH), subsections (.SS), items (.Ip), and index |
| 35 |
.\" entries marked with X<> in POD. Of course, you'll have to process the |
| 36 |
.\" output yourself in some meaningful fashion. |
| 37 |
.\" |
| 38 |
.\" Avoid warning from groff about undefined register 'F'. |
| 39 |
.de IX |
| 40 |
.. |
| 41 |
.nr rF 0 |
| 42 |
.if \n(.g .if rF .nr rF 1 |
| 43 |
.if (\n(rF:(\n(.g==0)) \{\ |
| 44 |
. if \nF \{\ |
| 45 |
. de IX |
| 46 |
. tm Index:\\$1\t\\n%\t"\\$2" |
| 47 |
.. |
| 48 |
. if !\nF==2 \{\ |
| 49 |
. nr % 0 |
| 50 |
. nr F 2 |
| 51 |
. \} |
| 52 |
. \} |
| 53 |
.\} |
| 54 |
.rr rF |
| 55 |
.\" ======================================================================== |
| 56 |
.\" |
| 57 |
.IX Title "PERLPACKTUT 1" |
| 58 |
.TH PERLPACKTUT 1 2025-11-16 "perl v5.40.1" "Perl Programmers Reference Guide" |
| 59 |
.\" For nroff, turn off justification. Always turn off hyphenation; it makes |
| 60 |
.\" way too many mistakes in technical documents. |
| 61 |
.if n .ad l |
| 62 |
.nh |
| 63 |
.SH NAME |
| 64 |
perlpacktut \- tutorial on "pack" and "unpack" |
| 65 |
.SH DESCRIPTION |
| 66 |
.IX Header "DESCRIPTION" |
| 67 |
\&\f(CW\*(C`pack\*(C'\fR and \f(CW\*(C`unpack\*(C'\fR are two functions for transforming data according |
| 68 |
to a user-defined template, between the guarded way Perl stores values |
| 69 |
and some well-defined representation as might be required in the |
| 70 |
environment of a Perl program. Unfortunately, they're also two of |
| 71 |
the most misunderstood and most often overlooked functions that Perl |
| 72 |
provides. This tutorial will demystify them for you. |
| 73 |
.SH "The Basic Principle" |
| 74 |
.IX Header "The Basic Principle" |
| 75 |
Most programming languages don't shelter the memory where variables are |
| 76 |
stored. In C, for instance, you can take the address of some variable, |
| 77 |
and the \f(CW\*(C`sizeof\*(C'\fR operator tells you how many bytes are allocated to |
| 78 |
the variable. Using the address and the size, you may access the storage |
| 79 |
to your heart's content. |
| 80 |
.PP |
| 81 |
In Perl, you just can't access memory at random, but the structural and |
| 82 |
representational conversion provided by \f(CW\*(C`pack\*(C'\fR and \f(CW\*(C`unpack\*(C'\fR is an |
| 83 |
excellent alternative. The \f(CW\*(C`pack\*(C'\fR function converts values to a byte |
| 84 |
sequence containing representations according to a given specification, |
| 85 |
the so-called "template" argument. \f(CW\*(C`unpack\*(C'\fR is the reverse process, |
| 86 |
deriving some values from the contents of a string of bytes. (Be cautioned, |
| 87 |
however, that not all that has been packed together can be neatly unpacked \- |
| 88 |
a very common experience as seasoned travellers are likely to confirm.) |
| 89 |
.PP |
| 90 |
Why, you may ask, would you need a chunk of memory containing some values |
| 91 |
in binary representation? One good reason is input and output accessing |
| 92 |
some file, a device, or a network connection, whereby this binary |
| 93 |
representation is either forced on you or will give you some benefit |
| 94 |
in processing. Another cause is passing data to some system call that |
| 95 |
is not available as a Perl function: \f(CW\*(C`syscall\*(C'\fR requires you to provide |
| 96 |
parameters stored in the way it happens in a C program. Even text processing |
| 97 |
(as shown in the next section) may be simplified with judicious usage |
| 98 |
of these two functions. |
| 99 |
.PP |
| 100 |
To see how (un)packing works, we'll start with a simple template |
| 101 |
code where the conversion is in low gear: between the contents of a byte |
| 102 |
sequence and a string of hexadecimal digits. Let's use \f(CW\*(C`unpack\*(C'\fR, since |
| 103 |
this is likely to remind you of a dump program, or some desperate last |
| 104 |
message unfortunate programs are wont to throw at you before they expire |
| 105 |
into the wild blue yonder. Assuming that the variable \f(CW$mem\fR holds a |
| 106 |
sequence of bytes that we'd like to inspect without assuming anything |
| 107 |
about its meaning, we can write |
| 108 |
.PP |
| 109 |
.Vb 2 |
| 110 |
\& my( $hex ) = unpack( \*(AqH*\*(Aq, $mem ); |
| 111 |
\& print "$hex\en"; |
| 112 |
.Ve |
| 113 |
.PP |
| 114 |
whereupon we might see something like this, with each pair of hex digits |
| 115 |
corresponding to a byte: |
| 116 |
.PP |
| 117 |
.Vb 1 |
| 118 |
\& 41204d414e204120504c414e20412043414e414c2050414e414d41 |
| 119 |
.Ve |
| 120 |
.PP |
| 121 |
What was in this chunk of memory? Numbers, characters, or a mixture of |
| 122 |
both? Assuming that we're on a computer where ASCII (or some similar) |
| 123 |
encoding is used: hexadecimal values in the range \f(CW0x40\fR \- \f(CW0x5A\fR |
| 124 |
indicate an uppercase letter, and \f(CW0x20\fR encodes a space. So we might |
| 125 |
assume it is a piece of text, which some are able to read like a tabloid; |
| 126 |
but others will have to get hold of an ASCII table and relive that |
| 127 |
firstgrader feeling. Not caring too much about which way to read this, |
| 128 |
we note that \f(CW\*(C`unpack\*(C'\fR with the template code \f(CW\*(C`H\*(C'\fR converts the contents |
| 129 |
of a sequence of bytes into the customary hexadecimal notation. Since |
| 130 |
"a sequence of" is a pretty vague indication of quantity, \f(CW\*(C`H\*(C'\fR has been |
| 131 |
defined to convert just a single hexadecimal digit unless it is followed |
| 132 |
by a repeat count. An asterisk for the repeat count means to use whatever |
| 133 |
remains. |
| 134 |
.PP |
| 135 |
The inverse operation \- packing byte contents from a string of hexadecimal |
| 136 |
digits \- is just as easily written. For instance: |
| 137 |
.PP |
| 138 |
.Vb 2 |
| 139 |
\& my $s = pack( \*(AqH2\*(Aq x 10, 30..39 ); |
| 140 |
\& print "$s\en"; |
| 141 |
.Ve |
| 142 |
.PP |
| 143 |
Since we feed a list of ten 2\-digit hexadecimal strings to \f(CW\*(C`pack\*(C'\fR, the |
| 144 |
pack template should contain ten pack codes. If this is run on a computer |
| 145 |
with ASCII character coding, it will print \f(CW0123456789\fR. |
| 146 |
.SH "Packing Text" |
| 147 |
.IX Header "Packing Text" |
| 148 |
Let's suppose you've got to read in a data file like this: |
| 149 |
.PP |
| 150 |
.Vb 4 |
| 151 |
\& Date |Description | Income|Expenditure |
| 152 |
\& 01/24/2001 Zed\*(Aqs Camel Emporium 1147.99 |
| 153 |
\& 01/28/2001 Flea spray 24.99 |
| 154 |
\& 01/29/2001 Camel rides to tourists 235.00 |
| 155 |
.Ve |
| 156 |
.PP |
| 157 |
How do we do it? You might think first to use \f(CW\*(C`split\*(C'\fR; however, since |
| 158 |
\&\f(CW\*(C`split\*(C'\fR collapses blank fields, you'll never know whether a record was |
| 159 |
income or expenditure. Oops. Well, you could always use \f(CW\*(C`substr\*(C'\fR: |
| 160 |
.PP |
| 161 |
.Vb 7 |
| 162 |
\& while (<>) { |
| 163 |
\& my $date = substr($_, 0, 11); |
| 164 |
\& my $desc = substr($_, 12, 27); |
| 165 |
\& my $income = substr($_, 40, 7); |
| 166 |
\& my $expend = substr($_, 52, 7); |
| 167 |
\& ... |
| 168 |
\& } |
| 169 |
.Ve |
| 170 |
.PP |
| 171 |
It's not really a barrel of laughs, is it? In fact, it's worse than it |
| 172 |
may seem; the eagle-eyed may notice that the first field should only be |
| 173 |
10 characters wide, and the error has propagated right through the other |
| 174 |
numbers \- which we've had to count by hand. So it's error-prone as well |
| 175 |
as horribly unfriendly. |
| 176 |
.PP |
| 177 |
Or maybe we could use regular expressions: |
| 178 |
.PP |
| 179 |
.Vb 5 |
| 180 |
\& while (<>) { |
| 181 |
\& my($date, $desc, $income, $expend) = |
| 182 |
\& m|(\ed\ed/\ed\ed/\ed{4}) (.{27}) (.{7})(.*)|; |
| 183 |
\& ... |
| 184 |
\& } |
| 185 |
.Ve |
| 186 |
.PP |
| 187 |
Urgh. Well, it's a bit better, but \- well, would you want to maintain |
| 188 |
that? |
| 189 |
.PP |
| 190 |
Hey, isn't Perl supposed to make this sort of thing easy? Well, it does, |
| 191 |
if you use the right tools. \f(CW\*(C`pack\*(C'\fR and \f(CW\*(C`unpack\*(C'\fR are designed to help |
| 192 |
you out when dealing with fixed-width data like the above. Let's have a |
| 193 |
look at a solution with \f(CW\*(C`unpack\*(C'\fR: |
| 194 |
.PP |
| 195 |
.Vb 4 |
| 196 |
\& while (<>) { |
| 197 |
\& my($date, $desc, $income, $expend) = unpack("A10xA27xA7A*", $_); |
| 198 |
\& ... |
| 199 |
\& } |
| 200 |
.Ve |
| 201 |
.PP |
| 202 |
That looks a bit nicer; but we've got to take apart that weird template. |
| 203 |
Where did I pull that out of? |
| 204 |
.PP |
| 205 |
OK, let's have a look at some of our data again; in fact, we'll include |
| 206 |
the headers, and a handy ruler so we can keep track of where we are. |
| 207 |
.PP |
| 208 |
.Vb 5 |
| 209 |
\& 1 2 3 4 5 |
| 210 |
\& 1234567890123456789012345678901234567890123456789012345678 |
| 211 |
\& Date |Description | Income|Expenditure |
| 212 |
\& 01/28/2001 Flea spray 24.99 |
| 213 |
\& 01/29/2001 Camel rides to tourists 235.00 |
| 214 |
.Ve |
| 215 |
.PP |
| 216 |
From this, we can see that the date column stretches from column 1 to |
| 217 |
column 10 \- ten characters wide. The \f(CW\*(C`pack\*(C'\fR\-ese for "character" is |
| 218 |
\&\f(CW\*(C`A\*(C'\fR, and ten of them are \f(CW\*(C`A10\*(C'\fR. So if we just wanted to extract the |
| 219 |
dates, we could say this: |
| 220 |
.PP |
| 221 |
.Vb 1 |
| 222 |
\& my($date) = unpack("A10", $_); |
| 223 |
.Ve |
| 224 |
.PP |
| 225 |
OK, what's next? Between the date and the description is a blank column; |
| 226 |
we want to skip over that. The \f(CW\*(C`x\*(C'\fR template means "skip forward", so we |
| 227 |
want one of those. Next, we have another batch of characters, from 12 to |
| 228 |
38. That's 27 more characters, hence \f(CW\*(C`A27\*(C'\fR. (Don't make the fencepost |
| 229 |
error \- there are 27 characters between 12 and 38, not 26. Count 'em!) |
| 230 |
.PP |
| 231 |
Now we skip another character and pick up the next 7 characters: |
| 232 |
.PP |
| 233 |
.Vb 1 |
| 234 |
\& my($date,$description,$income) = unpack("A10xA27xA7", $_); |
| 235 |
.Ve |
| 236 |
.PP |
| 237 |
Now comes the clever bit. Lines in our ledger which are just income and |
| 238 |
not expenditure might end at column 46. Hence, we don't want to tell our |
| 239 |
\&\f(CW\*(C`unpack\*(C'\fR pattern that we \fBneed\fR to find another 12 characters; we'll |
| 240 |
just say "if there's anything left, take it". As you might guess from |
| 241 |
regular expressions, that's what the \f(CW\*(C`*\*(C'\fR means: "use everything |
| 242 |
remaining". |
| 243 |
.IP \(bu 3 |
| 244 |
Be warned, though, that unlike regular expressions, if the \f(CW\*(C`unpack\*(C'\fR |
| 245 |
template doesn't match the incoming data, Perl will scream and die. |
| 246 |
.PP |
| 247 |
Hence, putting it all together: |
| 248 |
.PP |
| 249 |
.Vb 2 |
| 250 |
\& my ($date, $description, $income, $expend) = |
| 251 |
\& unpack("A10xA27xA7xA*", $_); |
| 252 |
.Ve |
| 253 |
.PP |
| 254 |
Now, that's our data parsed. I suppose what we might want to do now is |
| 255 |
total up our income and expenditure, and add another line to the end of |
| 256 |
our ledger \- in the same format \- saying how much we've brought in and |
| 257 |
how much we've spent: |
| 258 |
.PP |
| 259 |
.Vb 6 |
| 260 |
\& while (<>) { |
| 261 |
\& my ($date, $desc, $income, $expend) = |
| 262 |
\& unpack("A10xA27xA7xA*", $_); |
| 263 |
\& $tot_income += $income; |
| 264 |
\& $tot_expend += $expend; |
| 265 |
\& } |
| 266 |
\& |
| 267 |
\& $tot_income = sprintf("%.2f", $tot_income); # Get them into |
| 268 |
\& $tot_expend = sprintf("%.2f", $tot_expend); # "financial" format |
| 269 |
\& |
| 270 |
\& $date = POSIX::strftime("%m/%d/%Y", localtime); |
| 271 |
\& |
| 272 |
\& # OK, let\*(Aqs go: |
| 273 |
\& |
| 274 |
\& print pack("A10xA27xA7xA*", $date, "Totals", |
| 275 |
\& $tot_income, $tot_expend); |
| 276 |
.Ve |
| 277 |
.PP |
| 278 |
Oh, hmm. That didn't quite work. Let's see what happened: |
| 279 |
.PP |
| 280 |
.Vb 4 |
| 281 |
\& 01/24/2001 Zed\*(Aqs Camel Emporium 1147.99 |
| 282 |
\& 01/28/2001 Flea spray 24.99 |
| 283 |
\& 01/29/2001 Camel rides to tourists 1235.00 |
| 284 |
\& 03/23/2001Totals 1235.001172.98 |
| 285 |
.Ve |
| 286 |
.PP |
| 287 |
OK, it's a start, but what happened to the spaces? We put \f(CW\*(C`x\*(C'\fR, didn't |
| 288 |
we? Shouldn't it skip forward? Let's look at what "pack" in perlfunc says: |
| 289 |
.PP |
| 290 |
.Vb 1 |
| 291 |
\& x A null byte. |
| 292 |
.Ve |
| 293 |
.PP |
| 294 |
Urgh. No wonder. There's a big difference between "a null byte", |
| 295 |
character zero, and "a space", character 32. Perl's put something |
| 296 |
between the date and the description \- but unfortunately, we can't see |
| 297 |
it! |
| 298 |
.PP |
| 299 |
What we actually need to do is expand the width of the fields. The \f(CW\*(C`A\*(C'\fR |
| 300 |
format pads any non-existent characters with spaces, so we can use the |
| 301 |
additional spaces to line up our fields, like this: |
| 302 |
.PP |
| 303 |
.Vb 2 |
| 304 |
\& print pack("A11 A28 A8 A*", $date, "Totals", |
| 305 |
\& $tot_income, $tot_expend); |
| 306 |
.Ve |
| 307 |
.PP |
| 308 |
(Note that you can put spaces in the template to make it more readable, |
| 309 |
but they don't translate to spaces in the output.) Here's what we got |
| 310 |
this time: |
| 311 |
.PP |
| 312 |
.Vb 4 |
| 313 |
\& 01/24/2001 Zed\*(Aqs Camel Emporium 1147.99 |
| 314 |
\& 01/28/2001 Flea spray 24.99 |
| 315 |
\& 01/29/2001 Camel rides to tourists 1235.00 |
| 316 |
\& 03/23/2001 Totals 1235.00 1172.98 |
| 317 |
.Ve |
| 318 |
.PP |
| 319 |
That's a bit better, but we still have that last column which needs to |
| 320 |
be moved further over. There's an easy way to fix this up: |
| 321 |
unfortunately, we can't get \f(CW\*(C`pack\*(C'\fR to right-justify our fields, but we |
| 322 |
can get \f(CW\*(C`sprintf\*(C'\fR to do it: |
| 323 |
.PP |
| 324 |
.Vb 5 |
| 325 |
\& $tot_income = sprintf("%.2f", $tot_income); |
| 326 |
\& $tot_expend = sprintf("%12.2f", $tot_expend); |
| 327 |
\& $date = POSIX::strftime("%m/%d/%Y", localtime); |
| 328 |
\& print pack("A11 A28 A8 A*", $date, "Totals", |
| 329 |
\& $tot_income, $tot_expend); |
| 330 |
.Ve |
| 331 |
.PP |
| 332 |
This time we get the right answer: |
| 333 |
.PP |
| 334 |
.Vb 3 |
| 335 |
\& 01/28/2001 Flea spray 24.99 |
| 336 |
\& 01/29/2001 Camel rides to tourists 1235.00 |
| 337 |
\& 03/23/2001 Totals 1235.00 1172.98 |
| 338 |
.Ve |
| 339 |
.PP |
| 340 |
So that's how we consume and produce fixed-width data. Let's recap what |
| 341 |
we've seen of \f(CW\*(C`pack\*(C'\fR and \f(CW\*(C`unpack\*(C'\fR so far: |
| 342 |
.IP \(bu 3 |
| 343 |
Use \f(CW\*(C`pack\*(C'\fR to go from several pieces of data to one fixed-width |
| 344 |
version; use \f(CW\*(C`unpack\*(C'\fR to turn a fixed-width-format string into several |
| 345 |
pieces of data. |
| 346 |
.IP \(bu 3 |
| 347 |
The pack format \f(CW\*(C`A\*(C'\fR means "any character"; if you're \f(CW\*(C`pack\*(C'\fRing and |
| 348 |
you've run out of things to pack, \f(CW\*(C`pack\*(C'\fR will fill the rest up with |
| 349 |
spaces. |
| 350 |
.IP \(bu 3 |
| 351 |
\&\f(CW\*(C`x\*(C'\fR means "skip a byte" when \f(CW\*(C`unpack\*(C'\fRing; when \f(CW\*(C`pack\*(C'\fRing, it means |
| 352 |
"introduce a null byte" \- that's probably not what you mean if you're |
| 353 |
dealing with plain text. |
| 354 |
.IP \(bu 3 |
| 355 |
You can follow the formats with numbers to say how many characters |
| 356 |
should be affected by that format: \f(CW\*(C`A12\*(C'\fR means "take 12 characters"; |
| 357 |
\&\f(CW\*(C`x6\*(C'\fR means "skip 6 bytes" or "character 0, 6 times". |
| 358 |
.IP \(bu 3 |
| 359 |
Instead of a number, you can use \f(CW\*(C`*\*(C'\fR to mean "consume everything else |
| 360 |
left". |
| 361 |
.Sp |
| 362 |
\&\fBWarning\fR: when packing multiple pieces of data, \f(CW\*(C`*\*(C'\fR only means |
| 363 |
"consume all of the current piece of data". That's to say |
| 364 |
.Sp |
| 365 |
.Vb 1 |
| 366 |
\& pack("A*A*", $one, $two) |
| 367 |
.Ve |
| 368 |
.Sp |
| 369 |
packs all of \f(CW$one\fR into the first \f(CW\*(C`A*\*(C'\fR and then all of \f(CW$two\fR into |
| 370 |
the second. This is a general principle: each format character |
| 371 |
corresponds to one piece of data to be \f(CW\*(C`pack\*(C'\fRed. |
| 372 |
.SH "Packing Numbers" |
| 373 |
.IX Header "Packing Numbers" |
| 374 |
So much for textual data. Let's get onto the meaty stuff that \f(CW\*(C`pack\*(C'\fR |
| 375 |
and \f(CW\*(C`unpack\*(C'\fR are best at: handling binary formats for numbers. There is, |
| 376 |
of course, not just one binary format \- life would be too simple \- but |
| 377 |
Perl will do all the finicky labor for you. |
| 378 |
.SS Integers |
| 379 |
.IX Subsection "Integers" |
| 380 |
Packing and unpacking numbers implies conversion to and from some |
| 381 |
\&\fIspecific\fR binary representation. Leaving floating point numbers |
| 382 |
aside for the moment, the salient properties of any such representation |
| 383 |
are: |
| 384 |
.IP \(bu 4 |
| 385 |
the number of bytes used for storing the integer, |
| 386 |
.IP \(bu 4 |
| 387 |
whether the contents are interpreted as a signed or unsigned number, |
| 388 |
.IP \(bu 4 |
| 389 |
the byte ordering: whether the first byte is the least or most |
| 390 |
significant byte (or: little-endian or big-endian, respectively). |
| 391 |
.PP |
| 392 |
So, for instance, to pack 20302 to a signed 16 bit integer in your |
| 393 |
computer's representation you write |
| 394 |
.PP |
| 395 |
.Vb 1 |
| 396 |
\& my $ps = pack( \*(Aqs\*(Aq, 20302 ); |
| 397 |
.Ve |
| 398 |
.PP |
| 399 |
Again, the result is a string, now containing 2 bytes. If you print |
| 400 |
this string (which is, generally, not recommended) you might see |
| 401 |
\&\f(CW\*(C`ON\*(C'\fR or \f(CW\*(C`NO\*(C'\fR (depending on your system's byte ordering) \- or something |
| 402 |
entirely different if your computer doesn't use ASCII character encoding. |
| 403 |
Unpacking \f(CW$ps\fR with the same template returns the original integer value: |
| 404 |
.PP |
| 405 |
.Vb 1 |
| 406 |
\& my( $s ) = unpack( \*(Aqs\*(Aq, $ps ); |
| 407 |
.Ve |
| 408 |
.PP |
| 409 |
This is true for all numeric template codes. But don't expect miracles: |
| 410 |
if the packed value exceeds the allotted byte capacity, high order bits |
| 411 |
are silently discarded, and unpack certainly won't be able to pull them |
| 412 |
back out of some magic hat. And, when you pack using a signed template |
| 413 |
code such as \f(CW\*(C`s\*(C'\fR, an excess value may result in the sign bit |
| 414 |
getting set, and unpacking this will smartly return a negative value. |
| 415 |
.PP |
| 416 |
16 bits won't get you too far with integers, but there is \f(CW\*(C`l\*(C'\fR and \f(CW\*(C`L\*(C'\fR |
| 417 |
for signed and unsigned 32\-bit integers. And if this is not enough and |
| 418 |
your system supports 64 bit integers you can push the limits much closer |
| 419 |
to infinity with pack codes \f(CW\*(C`q\*(C'\fR and \f(CW\*(C`Q\*(C'\fR. A notable exception is provided |
| 420 |
by pack codes \f(CW\*(C`i\*(C'\fR and \f(CW\*(C`I\*(C'\fR for signed and unsigned integers of the |
| 421 |
"local custom" variety: Such an integer will take up as many bytes as |
| 422 |
a local C compiler returns for \f(CWsizeof(int)\fR, but it'll use \fIat least\fR |
| 423 |
32 bits. |
| 424 |
.PP |
| 425 |
Each of the integer pack codes \f(CW\*(C`sSlLqQ\*(C'\fR results in a fixed number of bytes, |
| 426 |
no matter where you execute your program. This may be useful for some |
| 427 |
applications, but it does not provide for a portable way to pass data |
| 428 |
structures between Perl and C programs (bound to happen when you call |
| 429 |
XS extensions or the Perl function \f(CW\*(C`syscall\*(C'\fR), or when you read or |
| 430 |
write binary files. What you'll need in this case are template codes that |
| 431 |
depend on what your local C compiler compiles when you code \f(CW\*(C`short\*(C'\fR or |
| 432 |
\&\f(CW\*(C`unsigned long\*(C'\fR, for instance. These codes and their corresponding |
| 433 |
byte lengths are shown in the table below. Since the C standard leaves |
| 434 |
much leeway with respect to the relative sizes of these data types, actual |
| 435 |
values may vary, and that's why the values are given as expressions in |
| 436 |
C and Perl. (If you'd like to use values from \f(CW%Config\fR in your program |
| 437 |
you have to import it with \f(CW\*(C`use Config\*(C'\fR.) |
| 438 |
.PP |
| 439 |
.Vb 5 |
| 440 |
\& signed unsigned byte length in C byte length in Perl |
| 441 |
\& s! S! sizeof(short) $Config{shortsize} |
| 442 |
\& i! I! sizeof(int) $Config{intsize} |
| 443 |
\& l! L! sizeof(long) $Config{longsize} |
| 444 |
\& q! Q! sizeof(long long) $Config{longlongsize} |
| 445 |
.Ve |
| 446 |
.PP |
| 447 |
The \f(CW\*(C`i!\*(C'\fR and \f(CW\*(C`I!\*(C'\fR codes aren't different from \f(CW\*(C`i\*(C'\fR and \f(CW\*(C`I\*(C'\fR; they are |
| 448 |
tolerated for completeness' sake. |
| 449 |
.SS "Unpacking a Stack Frame" |
| 450 |
.IX Subsection "Unpacking a Stack Frame" |
| 451 |
Requesting a particular byte ordering may be necessary when you work with |
| 452 |
binary data coming from some specific architecture whereas your program could |
| 453 |
run on a totally different system. As an example, assume you have 24 bytes |
| 454 |
containing a stack frame as it happens on an Intel 8086: |
| 455 |
.PP |
| 456 |
.Vb 11 |
| 457 |
\& +\-\-\-\-\-\-\-\-\-+ +\-\-\-\-+\-\-\-\-+ +\-\-\-\-\-\-\-\-\-+ |
| 458 |
\& TOS: | IP | TOS+4:| FL | FH | FLAGS TOS+14:| SI | |
| 459 |
\& +\-\-\-\-\-\-\-\-\-+ +\-\-\-\-+\-\-\-\-+ +\-\-\-\-\-\-\-\-\-+ |
| 460 |
\& | CS | | AL | AH | AX | DI | |
| 461 |
\& +\-\-\-\-\-\-\-\-\-+ +\-\-\-\-+\-\-\-\-+ +\-\-\-\-\-\-\-\-\-+ |
| 462 |
\& | BL | BH | BX | BP | |
| 463 |
\& +\-\-\-\-+\-\-\-\-+ +\-\-\-\-\-\-\-\-\-+ |
| 464 |
\& | CL | CH | CX | DS | |
| 465 |
\& +\-\-\-\-+\-\-\-\-+ +\-\-\-\-\-\-\-\-\-+ |
| 466 |
\& | DL | DH | DX | ES | |
| 467 |
\& +\-\-\-\-+\-\-\-\-+ +\-\-\-\-\-\-\-\-\-+ |
| 468 |
.Ve |
| 469 |
.PP |
| 470 |
First, we note that this time-honored 16\-bit CPU uses little-endian order, |
| 471 |
and that's why the low order byte is stored at the lower address. To |
| 472 |
unpack such a (unsigned) short we'll have to use code \f(CW\*(C`v\*(C'\fR. A repeat |
| 473 |
count unpacks all 12 shorts: |
| 474 |
.PP |
| 475 |
.Vb 2 |
| 476 |
\& my( $ip, $cs, $flags, $ax, $bx, $cx, $dx, $si, $di, $bp, $ds, $es ) = |
| 477 |
\& unpack( \*(Aqv12\*(Aq, $frame ); |
| 478 |
.Ve |
| 479 |
.PP |
| 480 |
Alternatively, we could have used \f(CW\*(C`C\*(C'\fR to unpack the individually |
| 481 |
accessible byte registers FL, FH, AL, AH, etc.: |
| 482 |
.PP |
| 483 |
.Vb 2 |
| 484 |
\& my( $fl, $fh, $al, $ah, $bl, $bh, $cl, $ch, $dl, $dh ) = |
| 485 |
\& unpack( \*(AqC10\*(Aq, substr( $frame, 4, 10 ) ); |
| 486 |
.Ve |
| 487 |
.PP |
| 488 |
It would be nice if we could do this in one fell swoop: unpack a short, |
| 489 |
back up a little, and then unpack 2 bytes. Since Perl \fIis\fR nice, it |
| 490 |
proffers the template code \f(CW\*(C`X\*(C'\fR to back up one byte. Putting this all |
| 491 |
together, we may now write: |
| 492 |
.PP |
| 493 |
.Vb 5 |
| 494 |
\& my( $ip, $cs, |
| 495 |
\& $flags,$fl,$fh, |
| 496 |
\& $ax,$al,$ah, $bx,$bl,$bh, $cx,$cl,$ch, $dx,$dl,$dh, |
| 497 |
\& $si, $di, $bp, $ds, $es ) = |
| 498 |
\& unpack( \*(Aqv2\*(Aq . (\*(AqvXXCC\*(Aq x 5) . \*(Aqv5\*(Aq, $frame ); |
| 499 |
.Ve |
| 500 |
.PP |
| 501 |
(The clumsy construction of the template can be avoided \- just read on!) |
| 502 |
.PP |
| 503 |
We've taken some pains to construct the template so that it matches |
| 504 |
the contents of our frame buffer. Otherwise we'd either get undefined values, |
| 505 |
or \f(CW\*(C`unpack\*(C'\fR could not unpack all. If \f(CW\*(C`pack\*(C'\fR runs out of items, it will |
| 506 |
supply null strings (which are coerced into zeroes whenever the pack code |
| 507 |
says so). |
| 508 |
.SS "How to Eat an Egg on a Net" |
| 509 |
.IX Subsection "How to Eat an Egg on a Net" |
| 510 |
The pack code for big-endian (high order byte at the lowest address) is |
| 511 |
\&\f(CW\*(C`n\*(C'\fR for 16 bit and \f(CW\*(C`N\*(C'\fR for 32 bit integers. You use these codes |
| 512 |
if you know that your data comes from a compliant architecture, but, |
| 513 |
surprisingly enough, you should also use these pack codes if you |
| 514 |
exchange binary data, across the network, with some system that you |
| 515 |
know next to nothing about. The simple reason is that this |
| 516 |
order has been chosen as the \fInetwork order\fR, and all standard-fearing |
| 517 |
programs ought to follow this convention. (This is, of course, a stern |
| 518 |
backing for one of the Lilliputian parties and may well influence the |
| 519 |
political development there.) So, if the protocol expects you to send |
| 520 |
a message by sending the length first, followed by just so many bytes, |
| 521 |
you could write: |
| 522 |
.PP |
| 523 |
.Vb 1 |
| 524 |
\& my $buf = pack( \*(AqN\*(Aq, length( $msg ) ) . $msg; |
| 525 |
.Ve |
| 526 |
.PP |
| 527 |
or even: |
| 528 |
.PP |
| 529 |
.Vb 1 |
| 530 |
\& my $buf = pack( \*(AqNA*\*(Aq, length( $msg ), $msg ); |
| 531 |
.Ve |
| 532 |
.PP |
| 533 |
and pass \f(CW$buf\fR to your send routine. Some protocols demand that the |
| 534 |
count should include the length of the count itself: then just add 4 |
| 535 |
to the data length. (But make sure to read "Lengths and Widths" before |
| 536 |
you really code this!) |
| 537 |
.SS "Byte-order modifiers" |
| 538 |
.IX Subsection "Byte-order modifiers" |
| 539 |
In the previous sections we've learned how to use \f(CW\*(C`n\*(C'\fR, \f(CW\*(C`N\*(C'\fR, \f(CW\*(C`v\*(C'\fR and |
| 540 |
\&\f(CW\*(C`V\*(C'\fR to pack and unpack integers with big\- or little-endian byte-order. |
| 541 |
While this is nice, it's still rather limited because it leaves out all |
| 542 |
kinds of signed integers as well as 64\-bit integers. For example, if you |
| 543 |
wanted to unpack a sequence of signed big-endian 16\-bit integers in a |
| 544 |
platform-independent way, you would have to write: |
| 545 |
.PP |
| 546 |
.Vb 1 |
| 547 |
\& my @data = unpack \*(Aqs*\*(Aq, pack \*(AqS*\*(Aq, unpack \*(Aqn*\*(Aq, $buf; |
| 548 |
.Ve |
| 549 |
.PP |
| 550 |
This is ugly. As of Perl 5.9.2, there's a much nicer way to express your |
| 551 |
desire for a certain byte-order: the \f(CW\*(C`>\*(C'\fR and \f(CW\*(C`<\*(C'\fR modifiers. |
| 552 |
\&\f(CW\*(C`>\*(C'\fR is the big-endian modifier, while \f(CW\*(C`<\*(C'\fR is the little-endian |
| 553 |
modifier. Using them, we could rewrite the above code as: |
| 554 |
.PP |
| 555 |
.Vb 1 |
| 556 |
\& my @data = unpack \*(Aqs>*\*(Aq, $buf; |
| 557 |
.Ve |
| 558 |
.PP |
| 559 |
As you can see, the "big end" of the arrow touches the \f(CW\*(C`s\*(C'\fR, which is a |
| 560 |
nice way to remember that \f(CW\*(C`>\*(C'\fR is the big-endian modifier. The same |
| 561 |
obviously works for \f(CW\*(C`<\*(C'\fR, where the "little end" touches the code. |
| 562 |
.PP |
| 563 |
You will probably find these modifiers even more useful if you have |
| 564 |
to deal with big\- or little-endian C structures. Be sure to read |
| 565 |
"Packing and Unpacking C Structures" for more on that. |
| 566 |
.SS "Floating point Numbers" |
| 567 |
.IX Subsection "Floating point Numbers" |
| 568 |
For packing floating point numbers you have the choice between the |
| 569 |
pack codes \f(CW\*(C`f\*(C'\fR, \f(CW\*(C`d\*(C'\fR, \f(CW\*(C`F\*(C'\fR and \f(CW\*(C`D\*(C'\fR. \f(CW\*(C`f\*(C'\fR and \f(CW\*(C`d\*(C'\fR pack into (or unpack |
| 570 |
from) single-precision or double-precision representation as it is provided |
| 571 |
by your system. If your systems supports it, \f(CW\*(C`D\*(C'\fR can be used to pack and |
| 572 |
unpack (\f(CW\*(C`long double\*(C'\fR) values, which can offer even more resolution |
| 573 |
than \f(CW\*(C`f\*(C'\fR or \f(CW\*(C`d\*(C'\fR. \fBNote that there are different long double formats.\fR |
| 574 |
.PP |
| 575 |
\&\f(CW\*(C`F\*(C'\fR packs an \f(CW\*(C`NV\*(C'\fR, which is the floating point type used by Perl |
| 576 |
internally. |
| 577 |
.PP |
| 578 |
There is no such thing as a network representation for reals, so if |
| 579 |
you want to send your real numbers across computer boundaries, you'd |
| 580 |
better stick to text representation, possibly using the hexadecimal |
| 581 |
float format (avoiding the decimal conversion loss), unless you're |
| 582 |
absolutely sure what's on the other end of the line. For the even more |
| 583 |
adventuresome, you can use the byte-order modifiers from the previous |
| 584 |
section also on floating point codes. |
| 585 |
.SH "Exotic Templates" |
| 586 |
.IX Header "Exotic Templates" |
| 587 |
.SS "Bit Strings" |
| 588 |
.IX Subsection "Bit Strings" |
| 589 |
Bits are the atoms in the memory world. Access to individual bits may |
| 590 |
have to be used either as a last resort or because it is the most |
| 591 |
convenient way to handle your data. Bit string (un)packing converts |
| 592 |
between strings containing a series of \f(CW0\fR and \f(CW1\fR characters and |
| 593 |
a sequence of bytes each containing a group of 8 bits. This is almost |
| 594 |
as simple as it sounds, except that there are two ways the contents of |
| 595 |
a byte may be written as a bit string. Let's have a look at an annotated |
| 596 |
byte: |
| 597 |
.PP |
| 598 |
.Vb 5 |
| 599 |
\& 7 6 5 4 3 2 1 0 |
| 600 |
\& +\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-+ |
| 601 |
\& | 1 0 0 0 1 1 0 0 | |
| 602 |
\& +\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-+ |
| 603 |
\& MSB LSB |
| 604 |
.Ve |
| 605 |
.PP |
| 606 |
It's egg-eating all over again: Some think that as a bit string this should |
| 607 |
be written "10001100" i.e. beginning with the most significant bit, others |
| 608 |
insist on "00110001". Well, Perl isn't biased, so that's why we have two bit |
| 609 |
string codes: |
| 610 |
.PP |
| 611 |
.Vb 2 |
| 612 |
\& $byte = pack( \*(AqB8\*(Aq, \*(Aq10001100\*(Aq ); # start with MSB |
| 613 |
\& $byte = pack( \*(Aqb8\*(Aq, \*(Aq00110001\*(Aq ); # start with LSB |
| 614 |
.Ve |
| 615 |
.PP |
| 616 |
It is not possible to pack or unpack bit fields \- just integral bytes. |
| 617 |
\&\f(CW\*(C`pack\*(C'\fR always starts at the next byte boundary and "rounds up" to the |
| 618 |
next multiple of 8 by adding zero bits as required. (If you do want bit |
| 619 |
fields, there is "vec" in perlfunc. Or you could implement bit field |
| 620 |
handling at the character string level, using split, substr, and |
| 621 |
concatenation on unpacked bit strings.) |
| 622 |
.PP |
| 623 |
To illustrate unpacking for bit strings, we'll decompose a simple |
| 624 |
status register (a "\-" stands for a "reserved" bit): |
| 625 |
.PP |
| 626 |
.Vb 4 |
| 627 |
\& +\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-+\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-+ |
| 628 |
\& | S Z \- A \- P \- C | \- \- \- \- O D I T | |
| 629 |
\& +\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-+\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-\-+ |
| 630 |
\& MSB LSB MSB LSB |
| 631 |
.Ve |
| 632 |
.PP |
| 633 |
Converting these two bytes to a string can be done with the unpack |
| 634 |
template \f(CW\*(Aqb16\*(Aq\fR. To obtain the individual bit values from the bit |
| 635 |
string we use \f(CW\*(C`split\*(C'\fR with the "empty" separator pattern which dissects |
| 636 |
into individual characters. Bit values from the "reserved" positions are |
| 637 |
simply assigned to \f(CW\*(C`undef\*(C'\fR, a convenient notation for "I don't care where |
| 638 |
this goes". |
| 639 |
.PP |
| 640 |
.Vb 3 |
| 641 |
\& ($carry, undef, $parity, undef, $auxcarry, undef, $zero, $sign, |
| 642 |
\& $trace, $interrupt, $direction, $overflow) = |
| 643 |
\& split( //, unpack( \*(Aqb16\*(Aq, $status ) ); |
| 644 |
.Ve |
| 645 |
.PP |
| 646 |
We could have used an unpack template \f(CW\*(Aqb12\*(Aq\fR just as well, since the |
| 647 |
last 4 bits can be ignored anyway. |
| 648 |
.SS Uuencoding |
| 649 |
.IX Subsection "Uuencoding" |
| 650 |
Another odd-man-out in the template alphabet is \f(CW\*(C`u\*(C'\fR, which packs a |
| 651 |
"uuencoded string". ("uu" is short for Unix-to-Unix.) Chances are that |
| 652 |
you won't ever need this encoding technique which was invented to overcome |
| 653 |
the shortcomings of old-fashioned transmission mediums that do not support |
| 654 |
other than simple ASCII data. The essential recipe is simple: Take three |
| 655 |
bytes, or 24 bits. Split them into 4 six-packs, adding a space (0x20) to |
| 656 |
each. Repeat until all of the data is blended. Fold groups of 4 bytes into |
| 657 |
lines no longer than 60 and garnish them in front with the original byte count |
| 658 |
(incremented by 0x20) and a \f(CW"\en"\fR at the end. \- The \f(CW\*(C`pack\*(C'\fR chef will |
| 659 |
prepare this for you, a la minute, when you select pack code \f(CW\*(C`u\*(C'\fR on the menu: |
| 660 |
.PP |
| 661 |
.Vb 1 |
| 662 |
\& my $uubuf = pack( \*(Aqu\*(Aq, $bindat ); |
| 663 |
.Ve |
| 664 |
.PP |
| 665 |
A repeat count after \f(CW\*(C`u\*(C'\fR sets the number of bytes to put into an |
| 666 |
uuencoded line, which is the maximum of 45 by default, but could be |
| 667 |
set to some (smaller) integer multiple of three. \f(CW\*(C`unpack\*(C'\fR simply ignores |
| 668 |
the repeat count. |
| 669 |
.SS "Doing Sums" |
| 670 |
.IX Subsection "Doing Sums" |
| 671 |
An even stranger template code is \f(CW\*(C`%\*(C'\fR<\fInumber\fR>. First, because |
| 672 |
it's used as a prefix to some other template code. Second, because it |
| 673 |
cannot be used in \f(CW\*(C`pack\*(C'\fR at all, and third, in \f(CW\*(C`unpack\*(C'\fR, doesn't return the |
| 674 |
data as defined by the template code it precedes. Instead it'll give you an |
| 675 |
integer of \fInumber\fR bits that is computed from the data value by |
| 676 |
doing sums. For numeric unpack codes, no big feat is achieved: |
| 677 |
.PP |
| 678 |
.Vb 2 |
| 679 |
\& my $buf = pack( \*(Aqiii\*(Aq, 100, 20, 3 ); |
| 680 |
\& print unpack( \*(Aq%32i3\*(Aq, $buf ), "\en"; # prints 123 |
| 681 |
.Ve |
| 682 |
.PP |
| 683 |
For string values, \f(CW\*(C`%\*(C'\fR returns the sum of the byte values saving |
| 684 |
you the trouble of a sum loop with \f(CW\*(C`substr\*(C'\fR and \f(CW\*(C`ord\*(C'\fR: |
| 685 |
.PP |
| 686 |
.Vb 1 |
| 687 |
\& print unpack( \*(Aq%32A*\*(Aq, "\ex01\ex10" ), "\en"; # prints 17 |
| 688 |
.Ve |
| 689 |
.PP |
| 690 |
Although the \f(CW\*(C`%\*(C'\fR code is documented as returning a "checksum": |
| 691 |
don't put your trust in such values! Even when applied to a small number |
| 692 |
of bytes, they won't guarantee a noticeable Hamming distance. |
| 693 |
.PP |
| 694 |
In connection with \f(CW\*(C`b\*(C'\fR or \f(CW\*(C`B\*(C'\fR, \f(CW\*(C`%\*(C'\fR simply adds bits, and this can be put |
| 695 |
to good use to count set bits efficiently: |
| 696 |
.PP |
| 697 |
.Vb 1 |
| 698 |
\& my $bitcount = unpack( \*(Aq%32b*\*(Aq, $mask ); |
| 699 |
.Ve |
| 700 |
.PP |
| 701 |
And an even parity bit can be determined like this: |
| 702 |
.PP |
| 703 |
.Vb 1 |
| 704 |
\& my $evenparity = unpack( \*(Aq%1b*\*(Aq, $mask ); |
| 705 |
.Ve |
| 706 |
.SS Unicode |
| 707 |
.IX Subsection "Unicode" |
| 708 |
Unicode is a character set that can represent most characters in most of |
| 709 |
the world's languages, providing room for over one million different |
| 710 |
characters. Unicode 3.1 specifies 94,140 characters: The Basic Latin |
| 711 |
characters are assigned to the numbers 0 \- 127. The Latin\-1 Supplement with |
| 712 |
characters that are used in several European languages is in the next |
| 713 |
range, up to 255. After some more Latin extensions we find the character |
| 714 |
sets from languages using non-Roman alphabets, interspersed with a |
| 715 |
variety of symbol sets such as currency symbols, Zapf Dingbats or Braille. |
| 716 |
(You might want to visit <https://www.unicode.org/> for a look at some of |
| 717 |
them \- my personal favourites are Telugu and Kannada.) |
| 718 |
.PP |
| 719 |
The Unicode character sets associates characters with integers. Encoding |
| 720 |
these numbers in an equal number of bytes would more than double the |
| 721 |
requirements for storing texts written in Latin alphabets. |
| 722 |
The UTF\-8 encoding avoids this by storing the most common (from a western |
| 723 |
point of view) characters in a single byte while encoding the rarer |
| 724 |
ones in three or more bytes. |
| 725 |
.PP |
| 726 |
Perl uses UTF\-8, internally, for most Unicode strings. |
| 727 |
.PP |
| 728 |
So what has this got to do with \f(CW\*(C`pack\*(C'\fR? Well, if you want to compose a |
| 729 |
Unicode string (that is internally encoded as UTF\-8), you can do so by |
| 730 |
using template code \f(CW\*(C`U\*(C'\fR. As an example, let's produce the Euro currency |
| 731 |
symbol (code number 0x20AC): |
| 732 |
.PP |
| 733 |
.Vb 2 |
| 734 |
\& $UTF8{Euro} = pack( \*(AqU\*(Aq, 0x20AC ); |
| 735 |
\& # Equivalent to: $UTF8{Euro} = "\ex{20ac}"; |
| 736 |
.Ve |
| 737 |
.PP |
| 738 |
Inspecting \f(CW$UTF8{Euro}\fR shows that it contains 3 bytes: |
| 739 |
"\exe2\ex82\exac". However, it contains only 1 character, number 0x20AC. |
| 740 |
The round trip can be completed with \f(CW\*(C`unpack\*(C'\fR: |
| 741 |
.PP |
| 742 |
.Vb 1 |
| 743 |
\& $Unicode{Euro} = unpack( \*(AqU\*(Aq, $UTF8{Euro} ); |
| 744 |
.Ve |
| 745 |
.PP |
| 746 |
Unpacking using the \f(CW\*(C`U\*(C'\fR template code also works on UTF\-8 encoded byte |
| 747 |
strings. |
| 748 |
.PP |
| 749 |
Usually you'll want to pack or unpack UTF\-8 strings: |
| 750 |
.PP |
| 751 |
.Vb 3 |
| 752 |
\& # pack and unpack the Hebrew alphabet |
| 753 |
\& my $alefbet = pack( \*(AqU*\*(Aq, 0x05d0..0x05ea ); |
| 754 |
\& my @hebrew = unpack( \*(AqU*\*(Aq, $utf ); |
| 755 |
.Ve |
| 756 |
.PP |
| 757 |
Please note: in the general case, you're better off using |
| 758 |
\&\f(CW\*(C`Encode::decode(\*(AqUTF\-8\*(Aq, $utf)\*(C'\fR to decode a UTF\-8 |
| 759 |
encoded byte string to a Perl Unicode string, and |
| 760 |
\&\f(CW\*(C`Encode::encode(\*(AqUTF\-8\*(Aq, $str)\*(C'\fR to encode a Perl Unicode |
| 761 |
string to UTF\-8 bytes. These functions provide means of handling invalid byte |
| 762 |
sequences and generally have a friendlier interface. |
| 763 |
.SS "Another Portable Binary Encoding" |
| 764 |
.IX Subsection "Another Portable Binary Encoding" |
| 765 |
The pack code \f(CW\*(C`w\*(C'\fR has been added to support a portable binary data |
| 766 |
encoding scheme that goes way beyond simple integers. (Details can |
| 767 |
be found at <https://github.com/mworks\-project/mw_scarab/blob/master/Scarab\-0.1.00d19/doc/binary\-serialization.txt>, |
| 768 |
the Scarab project.) A BER (Binary Encoded |
| 769 |
Representation) compressed unsigned integer stores base 128 |
| 770 |
digits, most significant digit first, with as few digits as possible. |
| 771 |
Bit eight (the high bit) is set on each byte except the last. There |
| 772 |
is no size limit to BER encoding, but Perl won't go to extremes. |
| 773 |
.PP |
| 774 |
.Vb 1 |
| 775 |
\& my $berbuf = pack( \*(Aqw*\*(Aq, 1, 128, 128+1, 128*128+127 ); |
| 776 |
.Ve |
| 777 |
.PP |
| 778 |
A hex dump of \f(CW$berbuf\fR, with spaces inserted at the right places, |
| 779 |
shows 01 8100 8101 81807F. Since the last byte is always less than |
| 780 |
128, \f(CW\*(C`unpack\*(C'\fR knows where to stop. |
| 781 |
.SH "Template Grouping" |
| 782 |
.IX Header "Template Grouping" |
| 783 |
Prior to Perl 5.8, repetitions of templates had to be made by |
| 784 |
\&\f(CW\*(C`x\*(C'\fR\-multiplication of template strings. Now there is a better way as |
| 785 |
we may use the pack codes \f(CW\*(C`(\*(C'\fR and \f(CW\*(C`)\*(C'\fR combined with a repeat count. |
| 786 |
The \f(CW\*(C`unpack\*(C'\fR template from the Stack Frame example can simply |
| 787 |
be written like this: |
| 788 |
.PP |
| 789 |
.Vb 1 |
| 790 |
\& unpack( \*(Aqv2 (vXXCC)5 v5\*(Aq, $frame ) |
| 791 |
.Ve |
| 792 |
.PP |
| 793 |
Let's explore this feature a little more. We'll begin with the equivalent of |
| 794 |
.PP |
| 795 |
.Vb 1 |
| 796 |
\& join( \*(Aq\*(Aq, map( substr( $_, 0, 1 ), @str ) ) |
| 797 |
.Ve |
| 798 |
.PP |
| 799 |
which returns a string consisting of the first character from each string. |
| 800 |
Using pack, we can write |
| 801 |
.PP |
| 802 |
.Vb 1 |
| 803 |
\& pack( \*(Aq(A)\*(Aq.@str, @str ) |
| 804 |
.Ve |
| 805 |
.PP |
| 806 |
or, because a repeat count \f(CW\*(C`*\*(C'\fR means "repeat as often as required", |
| 807 |
simply |
| 808 |
.PP |
| 809 |
.Vb 1 |
| 810 |
\& pack( \*(Aq(A)*\*(Aq, @str ) |
| 811 |
.Ve |
| 812 |
.PP |
| 813 |
(Note that the template \f(CW\*(C`A*\*(C'\fR would only have packed \f(CW$str[0]\fR in full |
| 814 |
length.) |
| 815 |
.PP |
| 816 |
To pack dates stored as triplets ( day, month, year ) in an array \f(CW@dates\fR |
| 817 |
into a sequence of byte, byte, short integer we can write |
| 818 |
.PP |
| 819 |
.Vb 1 |
| 820 |
\& $pd = pack( \*(Aq(CCS)*\*(Aq, map( @$_, @dates ) ); |
| 821 |
.Ve |
| 822 |
.PP |
| 823 |
To swap pairs of characters in a string (with even length) one could use |
| 824 |
several techniques. First, let's use \f(CW\*(C`x\*(C'\fR and \f(CW\*(C`X\*(C'\fR to skip forward and back: |
| 825 |
.PP |
| 826 |
.Vb 1 |
| 827 |
\& $s = pack( \*(Aq(A)*\*(Aq, unpack( \*(Aq(xAXXAx)*\*(Aq, $s ) ); |
| 828 |
.Ve |
| 829 |
.PP |
| 830 |
We can also use \f(CW\*(C`@\*(C'\fR to jump to an offset, with 0 being the position where |
| 831 |
we were when the last \f(CW\*(C`(\*(C'\fR was encountered: |
| 832 |
.PP |
| 833 |
.Vb 1 |
| 834 |
\& $s = pack( \*(Aq(A)*\*(Aq, unpack( \*(Aq(@1A @0A @2)*\*(Aq, $s ) ); |
| 835 |
.Ve |
| 836 |
.PP |
| 837 |
Finally, there is also an entirely different approach by unpacking big |
| 838 |
endian shorts and packing them in the reverse byte order: |
| 839 |
.PP |
| 840 |
.Vb 1 |
| 841 |
\& $s = pack( \*(Aq(v)*\*(Aq, unpack( \*(Aq(n)*\*(Aq, $s ); |
| 842 |
.Ve |
| 843 |
.SH "Lengths and Widths" |
| 844 |
.IX Header "Lengths and Widths" |
| 845 |
.SS "String Lengths" |
| 846 |
.IX Subsection "String Lengths" |
| 847 |
In the previous section we've seen a network message that was constructed |
| 848 |
by prefixing the binary message length to the actual message. You'll find |
| 849 |
that packing a length followed by so many bytes of data is a |
| 850 |
frequently used recipe since appending a null byte won't work |
| 851 |
if a null byte may be part of the data. Here is an example where both |
| 852 |
techniques are used: after two null terminated strings with source and |
| 853 |
destination address, a Short Message (to a mobile phone) is sent after |
| 854 |
a length byte: |
| 855 |
.PP |
| 856 |
.Vb 1 |
| 857 |
\& my $msg = pack( \*(AqZ*Z*CA*\*(Aq, $src, $dst, length( $sm ), $sm ); |
| 858 |
.Ve |
| 859 |
.PP |
| 860 |
Unpacking this message can be done with the same template: |
| 861 |
.PP |
| 862 |
.Vb 1 |
| 863 |
\& ( $src, $dst, $len, $sm ) = unpack( \*(AqZ*Z*CA*\*(Aq, $msg ); |
| 864 |
.Ve |
| 865 |
.PP |
| 866 |
There's a subtle trap lurking in the offing: Adding another field after |
| 867 |
the Short Message (in variable \f(CW$sm\fR) is all right when packing, but this |
| 868 |
cannot be unpacked naively: |
| 869 |
.PP |
| 870 |
.Vb 2 |
| 871 |
\& # pack a message |
| 872 |
\& my $msg = pack( \*(AqZ*Z*CA*C\*(Aq, $src, $dst, length( $sm ), $sm, $prio ); |
| 873 |
\& |
| 874 |
\& # unpack fails \- $prio remains undefined! |
| 875 |
\& ( $src, $dst, $len, $sm, $prio ) = unpack( \*(AqZ*Z*CA*C\*(Aq, $msg ); |
| 876 |
.Ve |
| 877 |
.PP |
| 878 |
The pack code \f(CW\*(C`A*\*(C'\fR gobbles up all remaining bytes, and \f(CW$prio\fR remains |
| 879 |
undefined! Before we let disappointment dampen the morale: Perl's got |
| 880 |
the trump card to make this trick too, just a little further up the sleeve. |
| 881 |
Watch this: |
| 882 |
.PP |
| 883 |
.Vb 2 |
| 884 |
\& # pack a message: ASCIIZ, ASCIIZ, length/string, byte |
| 885 |
\& my $msg = pack( \*(AqZ* Z* C/A* C\*(Aq, $src, $dst, $sm, $prio ); |
| 886 |
\& |
| 887 |
\& # unpack |
| 888 |
\& ( $src, $dst, $sm, $prio ) = unpack( \*(AqZ* Z* C/A* C\*(Aq, $msg ); |
| 889 |
.Ve |
| 890 |
.PP |
| 891 |
Combining two pack codes with a slash (\f(CW\*(C`/\*(C'\fR) associates them with a single |
| 892 |
value from the argument list. In \f(CW\*(C`pack\*(C'\fR, the length of the argument is |
| 893 |
taken and packed according to the first code while the argument itself |
| 894 |
is added after being converted with the template code after the slash. |
| 895 |
This saves us the trouble of inserting the \f(CW\*(C`length\*(C'\fR call, but it is |
| 896 |
in \f(CW\*(C`unpack\*(C'\fR where we really score: The value of the length byte marks the |
| 897 |
end of the string to be taken from the buffer. Since this combination |
| 898 |
doesn't make sense except when the second pack code isn't \f(CW\*(C`a*\*(C'\fR, \f(CW\*(C`A*\*(C'\fR |
| 899 |
or \f(CW\*(C`Z*\*(C'\fR, Perl won't let you. |
| 900 |
.PP |
| 901 |
The pack code preceding \f(CW\*(C`/\*(C'\fR may be anything that's fit to represent a |
| 902 |
number: All the numeric binary pack codes, and even text codes such as |
| 903 |
\&\f(CW\*(C`A4\*(C'\fR or \f(CW\*(C`Z*\*(C'\fR: |
| 904 |
.PP |
| 905 |
.Vb 4 |
| 906 |
\& # pack/unpack a string preceded by its length in ASCII |
| 907 |
\& my $buf = pack( \*(AqA4/A*\*(Aq, "Humpty\-Dumpty" ); |
| 908 |
\& # unpack $buf: \*(Aq13 Humpty\-Dumpty\*(Aq |
| 909 |
\& my $txt = unpack( \*(AqA4/A*\*(Aq, $buf ); |
| 910 |
.Ve |
| 911 |
.PP |
| 912 |
\&\f(CW\*(C`/\*(C'\fR is not implemented in Perls before 5.6, so if your code is required to |
| 913 |
work on ancient Perls you'll need to \f(CW\*(C`unpack( \*(AqZ* Z* C\*(Aq)\*(C'\fR to get the length, |
| 914 |
then use it to make a new unpack string. For example |
| 915 |
.PP |
| 916 |
.Vb 3 |
| 917 |
\& # pack a message: ASCIIZ, ASCIIZ, length, string, byte |
| 918 |
\& # (5.005 compatible) |
| 919 |
\& my $msg = pack( \*(AqZ* Z* C A* C\*(Aq, $src, $dst, length $sm, $sm, $prio ); |
| 920 |
\& |
| 921 |
\& # unpack |
| 922 |
\& ( undef, undef, $len) = unpack( \*(AqZ* Z* C\*(Aq, $msg ); |
| 923 |
\& ($src, $dst, $sm, $prio) = unpack ( "Z* Z* x A$len C", $msg ); |
| 924 |
.Ve |
| 925 |
.PP |
| 926 |
But that second \f(CW\*(C`unpack\*(C'\fR is rushing ahead. It isn't using a simple literal |
| 927 |
string for the template. So maybe we should introduce... |
| 928 |
.SS "Dynamic Templates" |
| 929 |
.IX Subsection "Dynamic Templates" |
| 930 |
So far, we've seen literals used as templates. If the list of pack |
| 931 |
items doesn't have fixed length, an expression constructing the |
| 932 |
template is required (whenever, for some reason, \f(CW\*(C`()*\*(C'\fR cannot be used). |
| 933 |
Here's an example: To store named string values in a way that can be |
| 934 |
conveniently parsed by a C program, we create a sequence of names and |
| 935 |
null terminated ASCII strings, with \f(CW\*(C`=\*(C'\fR between the name and the value, |
| 936 |
followed by an additional delimiting null byte. Here's how: |
| 937 |
.PP |
| 938 |
.Vb 2 |
| 939 |
\& my $env = pack( \*(Aq(A*A*Z*)\*(Aq . keys( %Env ) . \*(AqC\*(Aq, |
| 940 |
\& map( { ( $_, \*(Aq=\*(Aq, $Env{$_} ) } keys( %Env ) ), 0 ); |
| 941 |
.Ve |
| 942 |
.PP |
| 943 |
Let's examine the cogs of this byte mill, one by one. There's the \f(CW\*(C`map\*(C'\fR |
| 944 |
call, creating the items we intend to stuff into the \f(CW$env\fR buffer: |
| 945 |
to each key (in \f(CW$_\fR) it adds the \f(CW\*(C`=\*(C'\fR separator and the hash entry value. |
| 946 |
Each triplet is packed with the template code sequence \f(CW\*(C`A*A*Z*\*(C'\fR that |
| 947 |
is repeated according to the number of keys. (Yes, that's what the \f(CW\*(C`keys\*(C'\fR |
| 948 |
function returns in scalar context.) To get the very last null byte, |
| 949 |
we add a \f(CW0\fR at the end of the \f(CW\*(C`pack\*(C'\fR list, to be packed with \f(CW\*(C`C\*(C'\fR. |
| 950 |
(Attentive readers may have noticed that we could have omitted the 0.) |
| 951 |
.PP |
| 952 |
For the reverse operation, we'll have to determine the number of items |
| 953 |
in the buffer before we can let \f(CW\*(C`unpack\*(C'\fR rip it apart: |
| 954 |
.PP |
| 955 |
.Vb 2 |
| 956 |
\& my $n = $env =~ tr/\e0// \- 1; |
| 957 |
\& my %env = map( split( /=/, $_ ), unpack( "(Z*)$n", $env ) ); |
| 958 |
.Ve |
| 959 |
.PP |
| 960 |
The \f(CW\*(C`tr\*(C'\fR counts the null bytes. The \f(CW\*(C`unpack\*(C'\fR call returns a list of |
| 961 |
name-value pairs each of which is taken apart in the \f(CW\*(C`map\*(C'\fR block. |
| 962 |
.SS "Counting Repetitions" |
| 963 |
.IX Subsection "Counting Repetitions" |
| 964 |
Rather than storing a sentinel at the end of a data item (or a list of items), |
| 965 |
we could precede the data with a count. Again, we pack keys and values of |
| 966 |
a hash, preceding each with an unsigned short length count, and up front |
| 967 |
we store the number of pairs: |
| 968 |
.PP |
| 969 |
.Vb 1 |
| 970 |
\& my $env = pack( \*(AqS(S/A* S/A*)*\*(Aq, scalar keys( %Env ), %Env ); |
| 971 |
.Ve |
| 972 |
.PP |
| 973 |
This simplifies the reverse operation as the number of repetitions can be |
| 974 |
unpacked with the \f(CW\*(C`/\*(C'\fR code: |
| 975 |
.PP |
| 976 |
.Vb 1 |
| 977 |
\& my %env = unpack( \*(AqS/(S/A* S/A*)\*(Aq, $env ); |
| 978 |
.Ve |
| 979 |
.PP |
| 980 |
Note that this is one of the rare cases where you cannot use the same |
| 981 |
template for \f(CW\*(C`pack\*(C'\fR and \f(CW\*(C`unpack\*(C'\fR because \f(CW\*(C`pack\*(C'\fR can't determine |
| 982 |
a repeat count for a \f(CW\*(C`()\*(C'\fR\-group. |
| 983 |
.SS "Intel HEX" |
| 984 |
.IX Subsection "Intel HEX" |
| 985 |
Intel HEX is a file format for representing binary data, mostly for |
| 986 |
programming various chips, as a text file. (See |
| 987 |
<https://en.wikipedia.org/wiki/.hex> for a detailed description, and |
| 988 |
<https://en.wikipedia.org/wiki/SREC_(file_format)> for the Motorola |
| 989 |
S\-record format, which can be unravelled using the same technique.) |
| 990 |
Each line begins with a colon (':') and is followed by a sequence of |
| 991 |
hexadecimal characters, specifying a byte count \fIn\fR (8 bit), |
| 992 |
an address (16 bit, big endian), a record type (8 bit), \fIn\fR data bytes |
| 993 |
and a checksum (8 bit) computed as the least significant byte of the two's |
| 994 |
complement sum of the preceding bytes. Example: \f(CW\*(C`:0300300002337A1E\*(C'\fR. |
| 995 |
.PP |
| 996 |
The first step of processing such a line is the conversion, to binary, |
| 997 |
of the hexadecimal data, to obtain the four fields, while checking the |
| 998 |
checksum. No surprise here: we'll start with a simple \f(CW\*(C`pack\*(C'\fR call to |
| 999 |
convert everything to binary: |
| 1000 |
.PP |
| 1001 |
.Vb 1 |
| 1002 |
\& my $binrec = pack( \*(AqH*\*(Aq, substr( $hexrec, 1 ) ); |
| 1003 |
.Ve |
| 1004 |
.PP |
| 1005 |
The resulting byte sequence is most convenient for checking the checksum. |
| 1006 |
Don't slow your program down with a for loop adding the \f(CW\*(C`ord\*(C'\fR values |
| 1007 |
of this string's bytes \- the \f(CW\*(C`unpack\*(C'\fR code \f(CW\*(C`%\*(C'\fR is the thing to use |
| 1008 |
for computing the 8\-bit sum of all bytes, which must be equal to zero: |
| 1009 |
.PP |
| 1010 |
.Vb 1 |
| 1011 |
\& die unless unpack( "%8C*", $binrec ) == 0; |
| 1012 |
.Ve |
| 1013 |
.PP |
| 1014 |
Finally, let's get those four fields. By now, you shouldn't have any |
| 1015 |
problems with the first three fields \- but how can we use the byte count |
| 1016 |
of the data in the first field as a length for the data field? Here |
| 1017 |
the codes \f(CW\*(C`x\*(C'\fR and \f(CW\*(C`X\*(C'\fR come to the rescue, as they permit jumping |
| 1018 |
back and forth in the string to unpack. |
| 1019 |
.PP |
| 1020 |
.Vb 1 |
| 1021 |
\& my( $addr, $type, $data ) = unpack( "x n C X4 C x3 /a", $bin ); |
| 1022 |
.Ve |
| 1023 |
.PP |
| 1024 |
Code \f(CW\*(C`x\*(C'\fR skips a byte, since we don't need the count yet. Code \f(CW\*(C`n\*(C'\fR takes |
| 1025 |
care of the 16\-bit big-endian integer address, and \f(CW\*(C`C\*(C'\fR unpacks the |
| 1026 |
record type. Being at offset 4, where the data begins, we need the count. |
| 1027 |
\&\f(CW\*(C`X4\*(C'\fR brings us back to square one, which is the byte at offset 0. |
| 1028 |
Now we pick up the count, and zoom forth to offset 4, where we are |
| 1029 |
now fully furnished to extract the exact number of data bytes, leaving |
| 1030 |
the trailing checksum byte alone. |
| 1031 |
.SH "Packing and Unpacking C Structures" |
| 1032 |
.IX Header "Packing and Unpacking C Structures" |
| 1033 |
In previous sections we have seen how to pack numbers and character |
| 1034 |
strings. If it were not for a couple of snags we could conclude this |
| 1035 |
section right away with the terse remark that C structures don't |
| 1036 |
contain anything else, and therefore you already know all there is to it. |
| 1037 |
Sorry, no: read on, please. |
| 1038 |
.PP |
| 1039 |
If you have to deal with a lot of C structures, and don't want to |
| 1040 |
hack all your template strings manually, you'll probably want to have |
| 1041 |
a look at the CPAN module \f(CW\*(C`Convert::Binary::C\*(C'\fR. Not only can it parse |
| 1042 |
your C source directly, but it also has built-in support for all the |
| 1043 |
odds and ends described further on in this section. |
| 1044 |
.SS "The Alignment Pit" |
| 1045 |
.IX Subsection "The Alignment Pit" |
| 1046 |
In the consideration of speed against memory requirements the balance |
| 1047 |
has been tilted in favor of faster execution. This has influenced the |
| 1048 |
way C compilers allocate memory for structures: On architectures |
| 1049 |
where a 16\-bit or 32\-bit operand can be moved faster between places in |
| 1050 |
memory, or to or from a CPU register, if it is aligned at an even or |
| 1051 |
multiple-of-four or even at a multiple-of eight address, a C compiler |
| 1052 |
will give you this speed benefit by stuffing extra bytes into structures. |
| 1053 |
If you don't cross the C shoreline this is not likely to cause you any |
| 1054 |
grief (although you should care when you design large data structures, |
| 1055 |
or you want your code to be portable between architectures (you do want |
| 1056 |
that, don't you?)). |
| 1057 |
.PP |
| 1058 |
To see how this affects \f(CW\*(C`pack\*(C'\fR and \f(CW\*(C`unpack\*(C'\fR, we'll compare these two |
| 1059 |
C structures: |
| 1060 |
.PP |
| 1061 |
.Vb 6 |
| 1062 |
\& typedef struct { |
| 1063 |
\& char c1; |
| 1064 |
\& short s; |
| 1065 |
\& char c2; |
| 1066 |
\& long l; |
| 1067 |
\& } gappy_t; |
| 1068 |
\& |
| 1069 |
\& typedef struct { |
| 1070 |
\& long l; |
| 1071 |
\& short s; |
| 1072 |
\& char c1; |
| 1073 |
\& char c2; |
| 1074 |
\& } dense_t; |
| 1075 |
.Ve |
| 1076 |
.PP |
| 1077 |
Typically, a C compiler allocates 12 bytes to a \f(CW\*(C`gappy_t\*(C'\fR variable, but |
| 1078 |
requires only 8 bytes for a \f(CW\*(C`dense_t\*(C'\fR. After investigating this further, |
| 1079 |
we can draw memory maps, showing where the extra 4 bytes are hidden: |
| 1080 |
.PP |
| 1081 |
.Vb 5 |
| 1082 |
\& 0 +4 +8 +12 |
| 1083 |
\& +\-\-+\-\-+\-\-+\-\-+\-\-+\-\-+\-\-+\-\-+\-\-+\-\-+\-\-+\-\-+ |
| 1084 |
\& |c1|xx| s |c2|xx|xx|xx| l | xx = fill byte |
| 1085 |
\& +\-\-+\-\-+\-\-+\-\-+\-\-+\-\-+\-\-+\-\-+\-\-+\-\-+\-\-+\-\-+ |
| 1086 |
\& gappy_t |
| 1087 |
\& |
| 1088 |
\& 0 +4 +8 |
| 1089 |
\& +\-\-+\-\-+\-\-+\-\-+\-\-+\-\-+\-\-+\-\-+ |
| 1090 |
\& | l | h |c1|c2| |
| 1091 |
\& +\-\-+\-\-+\-\-+\-\-+\-\-+\-\-+\-\-+\-\-+ |
| 1092 |
\& dense_t |
| 1093 |
.Ve |
| 1094 |
.PP |
| 1095 |
And that's where the first quirk strikes: \f(CW\*(C`pack\*(C'\fR and \f(CW\*(C`unpack\*(C'\fR |
| 1096 |
templates have to be stuffed with \f(CW\*(C`x\*(C'\fR codes to get those extra fill bytes. |
| 1097 |
.PP |
| 1098 |
The natural question: "Why can't Perl compensate for the gaps?" warrants |
| 1099 |
an answer. One good reason is that C compilers might provide (non-ANSI) |
| 1100 |
extensions permitting all sorts of fancy control over the way structures |
| 1101 |
are aligned, even at the level of an individual structure field. And, if |
| 1102 |
this were not enough, there is an insidious thing called \f(CW\*(C`union\*(C'\fR where |
| 1103 |
the amount of fill bytes cannot be derived from the alignment of the next |
| 1104 |
item alone. |
| 1105 |
.PP |
| 1106 |
OK, so let's bite the bullet. Here's one way to get the alignment right |
| 1107 |
by inserting template codes \f(CW\*(C`x\*(C'\fR, which don't take a corresponding item |
| 1108 |
from the list: |
| 1109 |
.PP |
| 1110 |
.Vb 1 |
| 1111 |
\& my $gappy = pack( \*(Aqcxs cxxx l!\*(Aq, $c1, $s, $c2, $l ); |
| 1112 |
.Ve |
| 1113 |
.PP |
| 1114 |
Note the \f(CW\*(C`!\*(C'\fR after \f(CW\*(C`l\*(C'\fR: We want to make sure that we pack a long |
| 1115 |
integer as it is compiled by our C compiler. And even now, it will only |
| 1116 |
work for the platforms where the compiler aligns things as above. |
| 1117 |
And somebody somewhere has a platform where it doesn't. |
| 1118 |
[Probably a Cray, where \f(CW\*(C`short\*(C'\fRs, \f(CW\*(C`int\*(C'\fRs and \f(CW\*(C`long\*(C'\fRs are all 8 bytes. :\-)] |
| 1119 |
.PP |
| 1120 |
Counting bytes and watching alignments in lengthy structures is bound to |
| 1121 |
be a drag. Isn't there a way we can create the template with a simple |
| 1122 |
program? Here's a C program that does the trick: |
| 1123 |
.PP |
| 1124 |
.Vb 2 |
| 1125 |
\& #include <stdio.h> |
| 1126 |
\& #include <stddef.h> |
| 1127 |
\& |
| 1128 |
\& typedef struct { |
| 1129 |
\& char fc1; |
| 1130 |
\& short fs; |
| 1131 |
\& char fc2; |
| 1132 |
\& long fl; |
| 1133 |
\& } gappy_t; |
| 1134 |
\& |
| 1135 |
\& #define Pt(struct,field,tchar) \e |
| 1136 |
\& printf( "@%d%s ", offsetof(struct,field), # tchar ); |
| 1137 |
\& |
| 1138 |
\& int main() { |
| 1139 |
\& Pt( gappy_t, fc1, c ); |
| 1140 |
\& Pt( gappy_t, fs, s! ); |
| 1141 |
\& Pt( gappy_t, fc2, c ); |
| 1142 |
\& Pt( gappy_t, fl, l! ); |
| 1143 |
\& printf( "\en" ); |
| 1144 |
\& } |
| 1145 |
.Ve |
| 1146 |
.PP |
| 1147 |
The output line can be used as a template in a \f(CW\*(C`pack\*(C'\fR or \f(CW\*(C`unpack\*(C'\fR call: |
| 1148 |
.PP |
| 1149 |
.Vb 1 |
| 1150 |
\& my $gappy = pack( \*(Aq@0c @2s! @4c @8l!\*(Aq, $c1, $s, $c2, $l ); |
| 1151 |
.Ve |
| 1152 |
.PP |
| 1153 |
Gee, yet another template code \- as if we hadn't plenty. But |
| 1154 |
\&\f(CW\*(C`@\*(C'\fR saves our day by enabling us to specify the offset from the beginning |
| 1155 |
of the pack buffer to the next item: This is just the value |
| 1156 |
the \f(CW\*(C`offsetof\*(C'\fR macro (defined in \f(CW\*(C`<stddef.h>\*(C'\fR) returns when |
| 1157 |
given a \f(CW\*(C`struct\*(C'\fR type and one of its field names ("member-designator" in |
| 1158 |
C standardese). |
| 1159 |
.PP |
| 1160 |
Neither using offsets nor adding \f(CW\*(C`x\*(C'\fR's to bridge the gaps is satisfactory. |
| 1161 |
(Just imagine what happens if the structure changes.) What we really need |
| 1162 |
is a way of saying "skip as many bytes as required to the next multiple of N". |
| 1163 |
In fluent templates, you say this with \f(CW\*(C`x!N\*(C'\fR where N is replaced by the |
| 1164 |
appropriate value. Here's the next version of our struct packaging: |
| 1165 |
.PP |
| 1166 |
.Vb 1 |
| 1167 |
\& my $gappy = pack( \*(Aqc x!2 s c x!4 l!\*(Aq, $c1, $s, $c2, $l ); |
| 1168 |
.Ve |
| 1169 |
.PP |
| 1170 |
That's certainly better, but we still have to know how long all the |
| 1171 |
integers are, and portability is far away. Rather than \f(CW2\fR, |
| 1172 |
for instance, we want to say "however long a short is". But this can be |
| 1173 |
done by enclosing the appropriate pack code in brackets: \f(CW\*(C`[s]\*(C'\fR. So, here's |
| 1174 |
the very best we can do: |
| 1175 |
.PP |
| 1176 |
.Vb 1 |
| 1177 |
\& my $gappy = pack( \*(Aqc x![s] s c x![l!] l!\*(Aq, $c1, $s, $c2, $l ); |
| 1178 |
.Ve |
| 1179 |
.SS "Dealing with Endian-ness" |
| 1180 |
.IX Subsection "Dealing with Endian-ness" |
| 1181 |
Now, imagine that we want to pack the data for a machine with a |
| 1182 |
different byte-order. First, we'll have to figure out how big the data |
| 1183 |
types on the target machine really are. Let's assume that the longs are |
| 1184 |
32 bits wide and the shorts are 16 bits wide. You can then rewrite the |
| 1185 |
template as: |
| 1186 |
.PP |
| 1187 |
.Vb 1 |
| 1188 |
\& my $gappy = pack( \*(Aqc x![s] s c x![l] l\*(Aq, $c1, $s, $c2, $l ); |
| 1189 |
.Ve |
| 1190 |
.PP |
| 1191 |
If the target machine is little-endian, we could write: |
| 1192 |
.PP |
| 1193 |
.Vb 1 |
| 1194 |
\& my $gappy = pack( \*(Aqc x![s] s< c x![l] l<\*(Aq, $c1, $s, $c2, $l ); |
| 1195 |
.Ve |
| 1196 |
.PP |
| 1197 |
This forces the short and the long members to be little-endian, and is |
| 1198 |
just fine if you don't have too many struct members. But we could also |
| 1199 |
use the byte-order modifier on a group and write the following: |
| 1200 |
.PP |
| 1201 |
.Vb 1 |
| 1202 |
\& my $gappy = pack( \*(Aq( c x![s] s c x![l] l )<\*(Aq, $c1, $s, $c2, $l ); |
| 1203 |
.Ve |
| 1204 |
.PP |
| 1205 |
This is not as short as before, but it makes it more obvious that we |
| 1206 |
intend to have little-endian byte-order for a whole group, not only |
| 1207 |
for individual template codes. It can also be more readable and easier |
| 1208 |
to maintain. |
| 1209 |
.SS "Alignment, Take 2" |
| 1210 |
.IX Subsection "Alignment, Take 2" |
| 1211 |
I'm afraid that we're not quite through with the alignment catch yet. The |
| 1212 |
hydra raises another ugly head when you pack arrays of structures: |
| 1213 |
.PP |
| 1214 |
.Vb 4 |
| 1215 |
\& typedef struct { |
| 1216 |
\& short count; |
| 1217 |
\& char glyph; |
| 1218 |
\& } cell_t; |
| 1219 |
\& |
| 1220 |
\& typedef cell_t buffer_t[BUFLEN]; |
| 1221 |
.Ve |
| 1222 |
.PP |
| 1223 |
Where's the catch? Padding is neither required before the first field \f(CW\*(C`count\*(C'\fR, |
| 1224 |
nor between this and the next field \f(CW\*(C`glyph\*(C'\fR, so why can't we simply pack |
| 1225 |
like this: |
| 1226 |
.PP |
| 1227 |
.Vb 3 |
| 1228 |
\& # something goes wrong here: |
| 1229 |
\& pack( \*(Aqs!a\*(Aq x @buffer, |
| 1230 |
\& map{ ( $_\->{count}, $_\->{glyph} ) } @buffer ); |
| 1231 |
.Ve |
| 1232 |
.PP |
| 1233 |
This packs \f(CW\*(C`3*@buffer\*(C'\fR bytes, but it turns out that the size of |
| 1234 |
\&\f(CW\*(C`buffer_t\*(C'\fR is four times \f(CW\*(C`BUFLEN\*(C'\fR! The moral of the story is that |
| 1235 |
the required alignment of a structure or array is propagated to the |
| 1236 |
next higher level where we have to consider padding \fIat the end\fR |
| 1237 |
of each component as well. Thus the correct template is: |
| 1238 |
.PP |
| 1239 |
.Vb 2 |
| 1240 |
\& pack( \*(Aqs!ax\*(Aq x @buffer, |
| 1241 |
\& map{ ( $_\->{count}, $_\->{glyph} ) } @buffer ); |
| 1242 |
.Ve |
| 1243 |
.SS "Alignment, Take 3" |
| 1244 |
.IX Subsection "Alignment, Take 3" |
| 1245 |
And even if you take all the above into account, ANSI still lets this: |
| 1246 |
.PP |
| 1247 |
.Vb 3 |
| 1248 |
\& typedef struct { |
| 1249 |
\& char foo[2]; |
| 1250 |
\& } foo_t; |
| 1251 |
.Ve |
| 1252 |
.PP |
| 1253 |
vary in size. The alignment constraint of the structure can be greater than |
| 1254 |
any of its elements. [And if you think that this doesn't affect anything |
| 1255 |
common, dismember the next cellphone that you see. Many have ARM cores, and |
| 1256 |
the ARM structure rules make \f(CW\*(C`sizeof (foo_t)\*(C'\fR == 4] |
| 1257 |
.SS "Pointers for How to Use Them" |
| 1258 |
.IX Subsection "Pointers for How to Use Them" |
| 1259 |
The title of this section indicates the second problem you may run into |
| 1260 |
sooner or later when you pack C structures. If the function you intend |
| 1261 |
to call expects a, say, \f(CW\*(C`void *\*(C'\fR value, you \fIcannot\fR simply take |
| 1262 |
a reference to a Perl variable. (Although that value certainly is a |
| 1263 |
memory address, it's not the address where the variable's contents are |
| 1264 |
stored.) |
| 1265 |
.PP |
| 1266 |
Template code \f(CW\*(C`P\*(C'\fR promises to pack a "pointer to a fixed length string". |
| 1267 |
Isn't this what we want? Let's try: |
| 1268 |
.PP |
| 1269 |
.Vb 3 |
| 1270 |
\& # allocate some storage and pack a pointer to it |
| 1271 |
\& my $memory = "\ex00" x $size; |
| 1272 |
\& my $memptr = pack( \*(AqP\*(Aq, $memory ); |
| 1273 |
.Ve |
| 1274 |
.PP |
| 1275 |
But wait: doesn't \f(CW\*(C`pack\*(C'\fR just return a sequence of bytes? How can we pass this |
| 1276 |
string of bytes to some C code expecting a pointer which is, after all, |
| 1277 |
nothing but a number? The answer is simple: We have to obtain the numeric |
| 1278 |
address from the bytes returned by \f(CW\*(C`pack\*(C'\fR. |
| 1279 |
.PP |
| 1280 |
.Vb 1 |
| 1281 |
\& my $ptr = unpack( \*(AqL!\*(Aq, $memptr ); |
| 1282 |
.Ve |
| 1283 |
.PP |
| 1284 |
Obviously this assumes that it is possible to typecast a pointer |
| 1285 |
to an unsigned long and vice versa, which frequently works but should not |
| 1286 |
be taken as a universal law. \- Now that we have this pointer the next question |
| 1287 |
is: How can we put it to good use? We need a call to some C function |
| 1288 |
where a pointer is expected. The \fBread\fR\|(2) system call comes to mind: |
| 1289 |
.PP |
| 1290 |
.Vb 1 |
| 1291 |
\& ssize_t read(int fd, void *buf, size_t count); |
| 1292 |
.Ve |
| 1293 |
.PP |
| 1294 |
After reading perlfunc explaining how to use \f(CW\*(C`syscall\*(C'\fR we can write |
| 1295 |
this Perl function copying a file to standard output: |
| 1296 |
.PP |
| 1297 |
.Vb 12 |
| 1298 |
\& require \*(Aqsyscall.ph\*(Aq; # run h2ph to generate this file |
| 1299 |
\& sub cat($){ |
| 1300 |
\& my $path = shift(); |
| 1301 |
\& my $size = \-s $path; |
| 1302 |
\& my $memory = "\ex00" x $size; # allocate some memory |
| 1303 |
\& my $ptr = unpack( \*(AqL\*(Aq, pack( \*(AqP\*(Aq, $memory ) ); |
| 1304 |
\& open( F, $path ) || die( "$path: cannot open ($!)\en" ); |
| 1305 |
\& my $fd = fileno(F); |
| 1306 |
\& my $res = syscall( &SYS_read, fileno(F), $ptr, $size ); |
| 1307 |
\& print $memory; |
| 1308 |
\& close( F ); |
| 1309 |
\& } |
| 1310 |
.Ve |
| 1311 |
.PP |
| 1312 |
This is neither a specimen of simplicity nor a paragon of portability but |
| 1313 |
it illustrates the point: We are able to sneak behind the scenes and |
| 1314 |
access Perl's otherwise well-guarded memory! (Important note: Perl's |
| 1315 |
\&\f(CW\*(C`syscall\*(C'\fR does \fInot\fR require you to construct pointers in this roundabout |
| 1316 |
way. You simply pass a string variable, and Perl forwards the address.) |
| 1317 |
.PP |
| 1318 |
How does \f(CW\*(C`unpack\*(C'\fR with \f(CW\*(C`P\*(C'\fR work? Imagine some pointer in the buffer |
| 1319 |
about to be unpacked: If it isn't the null pointer (which will smartly |
| 1320 |
produce the \f(CW\*(C`undef\*(C'\fR value) we have a start address \- but then what? |
| 1321 |
Perl has no way of knowing how long this "fixed length string" is, so |
| 1322 |
it's up to you to specify the actual size as an explicit length after \f(CW\*(C`P\*(C'\fR. |
| 1323 |
.PP |
| 1324 |
.Vb 2 |
| 1325 |
\& my $mem = "abcdefghijklmn"; |
| 1326 |
\& print unpack( \*(AqP5\*(Aq, pack( \*(AqP\*(Aq, $mem ) ); # prints "abcde" |
| 1327 |
.Ve |
| 1328 |
.PP |
| 1329 |
As a consequence, \f(CW\*(C`pack\*(C'\fR ignores any number or \f(CW\*(C`*\*(C'\fR after \f(CW\*(C`P\*(C'\fR. |
| 1330 |
.PP |
| 1331 |
Now that we have seen \f(CW\*(C`P\*(C'\fR at work, we might as well give \f(CW\*(C`p\*(C'\fR a whirl. |
| 1332 |
Why do we need a second template code for packing pointers at all? The |
| 1333 |
answer lies behind the simple fact that an \f(CW\*(C`unpack\*(C'\fR with \f(CW\*(C`p\*(C'\fR promises |
| 1334 |
a null-terminated string starting at the address taken from the buffer, |
| 1335 |
and that implies a length for the data item to be returned: |
| 1336 |
.PP |
| 1337 |
.Vb 2 |
| 1338 |
\& my $buf = pack( \*(Aqp\*(Aq, "abc\ex00efhijklmn" ); |
| 1339 |
\& print unpack( \*(Aqp\*(Aq, $buf ); # prints "abc" |
| 1340 |
.Ve |
| 1341 |
.PP |
| 1342 |
Albeit this is apt to be confusing: As a consequence of the length being |
| 1343 |
implied by the string's length, a number after pack code \f(CW\*(C`p\*(C'\fR is a repeat |
| 1344 |
count, not a length as after \f(CW\*(C`P\*(C'\fR. |
| 1345 |
.PP |
| 1346 |
Using \f(CW\*(C`pack(..., $x)\*(C'\fR with \f(CW\*(C`P\*(C'\fR or \f(CW\*(C`p\*(C'\fR to get the address where \f(CW$x\fR is |
| 1347 |
actually stored must be used with circumspection. Perl's internal machinery |
| 1348 |
considers the relation between a variable and that address as its very own |
| 1349 |
private matter and doesn't really care that we have obtained a copy. Therefore: |
| 1350 |
.IP \(bu 4 |
| 1351 |
Do not use \f(CW\*(C`pack\*(C'\fR with \f(CW\*(C`p\*(C'\fR or \f(CW\*(C`P\*(C'\fR to obtain the address of variable |
| 1352 |
that's bound to go out of scope (and thereby freeing its memory) before you |
| 1353 |
are done with using the memory at that address. |
| 1354 |
.IP \(bu 4 |
| 1355 |
Be very careful with Perl operations that change the value of the |
| 1356 |
variable. Appending something to the variable, for instance, might require |
| 1357 |
reallocation of its storage, leaving you with a pointer into no-man's land. |
| 1358 |
.IP \(bu 4 |
| 1359 |
Don't think that you can get the address of a Perl variable |
| 1360 |
when it is stored as an integer or double number! \f(CW\*(C`pack(\*(AqP\*(Aq, $x)\*(C'\fR will |
| 1361 |
force the variable's internal representation to string, just as if you |
| 1362 |
had written something like \f(CW\*(C`$x .= \*(Aq\*(Aq\*(C'\fR. |
| 1363 |
.PP |
| 1364 |
It's safe, however, to P\- or p\-pack a string literal, because Perl simply |
| 1365 |
allocates an anonymous variable. |
| 1366 |
.SH "Pack Recipes" |
| 1367 |
.IX Header "Pack Recipes" |
| 1368 |
Here are a collection of (possibly) useful canned recipes for \f(CW\*(C`pack\*(C'\fR |
| 1369 |
and \f(CW\*(C`unpack\*(C'\fR: |
| 1370 |
.PP |
| 1371 |
.Vb 2 |
| 1372 |
\& # Convert IP address for socket functions |
| 1373 |
\& pack( "C4", split /\e./, "123.4.5.6" ); |
| 1374 |
\& |
| 1375 |
\& # Count the bits in a chunk of memory (e.g. a select vector) |
| 1376 |
\& unpack( \*(Aq%32b*\*(Aq, $mask ); |
| 1377 |
\& |
| 1378 |
\& # Determine the endianness of your system |
| 1379 |
\& $is_little_endian = unpack( \*(Aqc\*(Aq, pack( \*(Aqs\*(Aq, 1 ) ); |
| 1380 |
\& $is_big_endian = unpack( \*(Aqxc\*(Aq, pack( \*(Aqs\*(Aq, 1 ) ); |
| 1381 |
\& |
| 1382 |
\& # Determine the number of bits in a native integer |
| 1383 |
\& $bits = unpack( \*(Aq%32I!\*(Aq, ~0 ); |
| 1384 |
\& |
| 1385 |
\& # Prepare argument for the nanosleep system call |
| 1386 |
\& my $timespec = pack( \*(AqL!L!\*(Aq, $secs, $nanosecs ); |
| 1387 |
.Ve |
| 1388 |
.PP |
| 1389 |
For a simple memory dump we unpack some bytes into just as |
| 1390 |
many pairs of hex digits, and use \f(CW\*(C`map\*(C'\fR to handle the traditional |
| 1391 |
spacing \- 16 bytes to a line: |
| 1392 |
.PP |
| 1393 |
.Vb 4 |
| 1394 |
\& my $i; |
| 1395 |
\& print map( ++$i % 16 ? "$_ " : "$_\en", |
| 1396 |
\& unpack( \*(AqH2\*(Aq x length( $mem ), $mem ) ), |
| 1397 |
\& length( $mem ) % 16 ? "\en" : \*(Aq\*(Aq; |
| 1398 |
.Ve |
| 1399 |
.SH "Funnies Section" |
| 1400 |
.IX Header "Funnies Section" |
| 1401 |
.Vb 5 |
| 1402 |
\& # Pulling digits out of nowhere... |
| 1403 |
\& print unpack( \*(AqC\*(Aq, pack( \*(Aqx\*(Aq ) ), |
| 1404 |
\& unpack( \*(Aq%B*\*(Aq, pack( \*(AqA\*(Aq ) ), |
| 1405 |
\& unpack( \*(AqH\*(Aq, pack( \*(AqA\*(Aq ) ), |
| 1406 |
\& unpack( \*(AqA\*(Aq, unpack( \*(AqC\*(Aq, pack( \*(AqA\*(Aq ) ) ), "\en"; |
| 1407 |
\& |
| 1408 |
\& # One for the road ;\-) |
| 1409 |
\& my $advice = pack( \*(Aqall u can in a van\*(Aq ); |
| 1410 |
.Ve |
| 1411 |
.SH Authors |
| 1412 |
.IX Header "Authors" |
| 1413 |
Simon Cozens and Wolfgang Laun. |