| 1 |
NAME |
| 2 |
File::Write::Rotate - Write to files that archive/rotate themselves |
| 3 |
|
| 4 |
VERSION |
| 5 |
This document describes version 0.321 of File::Write::Rotate (from Perl |
| 6 |
distribution File-Write-Rotate), released on 2019-06-27. |
| 7 |
|
| 8 |
SYNOPSIS |
| 9 |
use File::Write::Rotate; |
| 10 |
|
| 11 |
my $fwr = File::Write::Rotate->new( |
| 12 |
dir => '/var/log', # required |
| 13 |
prefix => 'myapp', # required |
| 14 |
#suffix => '.log', # default is '' |
| 15 |
size => 25*1024*1024, # default is 10MB, unless period is set |
| 16 |
histories => 12, # default is 10 |
| 17 |
#buffer_size => 100, # default is none |
| 18 |
); |
| 19 |
|
| 20 |
# write, will write to /var/log/myapp.log, automatically rotate old log files |
| 21 |
# to myapp.log.1 when myapp.log reaches 25MB. will keep old log files up to |
| 22 |
# myapp.log.12. |
| 23 |
$fwr->write("This is a line\n"); |
| 24 |
$fwr->write("This is", " another line\n"); |
| 25 |
|
| 26 |
To compressing old log files: |
| 27 |
|
| 28 |
$fwr->compress; |
| 29 |
|
| 30 |
This is usually done in a separate process, because it potentially takes |
| 31 |
a long time if the files to compress are large; we are rotating |
| 32 |
automatically in write() so doing automatic compression too would |
| 33 |
annoyingly block writer for a potentially long time. |
| 34 |
|
| 35 |
DESCRIPTION |
| 36 |
This module can be used to write to file, usually for logging, that can |
| 37 |
rotate itself. File will be opened in append mode. By default, locking |
| 38 |
will be done to avoid conflict when there are multiple writers. Rotation |
| 39 |
can be done by size (after a certain size is reached), by time |
| 40 |
(daily/monthly/yearly), or both. |
| 41 |
|
| 42 |
I first wrote this module for logging script STDERR output to files (see |
| 43 |
Tie::Handle::FileWriteRotate). |
| 44 |
|
| 45 |
ATTRIBUTES |
| 46 |
buffer_size => int |
| 47 |
Get or set buffer size. If set to a value larger than 0, then when a |
| 48 |
write() failed, instead of dying, the message will be stored in an |
| 49 |
internal buffer first (a regular Perl array). When the number of items |
| 50 |
in the buffer exceeds this size, then write() will die upon failure. |
| 51 |
Otherwise, every write() will try to flush the buffer. |
| 52 |
|
| 53 |
Can be used for example when a program runs as superuser/root then |
| 54 |
temporarily drops privilege to a normal user. During this period, |
| 55 |
logging can fail because the program cannot lock the lock file or write |
| 56 |
to the logging directory. Before dropping privilege, the program can set |
| 57 |
buffer_size to some larger-than-zero value to hold the messages emitted |
| 58 |
during dropping privilege. The next write() as the superuser/root will |
| 59 |
succeed and flush the buffer to disk (provided there is no other error |
| 60 |
condition, of course). |
| 61 |
|
| 62 |
path => str (ro) |
| 63 |
Current file's path. |
| 64 |
|
| 65 |
handle => (ro) |
| 66 |
Current file handle. You should not use this directly, but use write() |
| 67 |
instead. This attribute is provided for special circumstances (e.g. in |
| 68 |
hooks, see example in the hook section). |
| 69 |
|
| 70 |
hook_before_write => code |
| 71 |
Will be called by write() before actually writing to filehandle (but |
| 72 |
after locking is done). Code will be passed ($self, \@msgs, $fh) where |
| 73 |
@msgs is an array of strings to be written (the contents of buffer, if |
| 74 |
any, plus arguments passed to write()) and $fh is the filehandle. |
| 75 |
|
| 76 |
hook_before_rotate => code |
| 77 |
Will be called by the rotating routine before actually doing rotating. |
| 78 |
Code will be passed ($self). |
| 79 |
|
| 80 |
This can be used to write a footer to the end of each file, e.g.: |
| 81 |
|
| 82 |
# hook_before_rotate |
| 83 |
my ($self) = @_; |
| 84 |
my $fh = $self->handle; |
| 85 |
print $fh "Some footer\n"; |
| 86 |
|
| 87 |
Since this hook is indirectly called by write(), locking is already |
| 88 |
done. |
| 89 |
|
| 90 |
hook_after_rotate => code |
| 91 |
Will be called by the rotating routine after the rotating process. Code |
| 92 |
will be passed ($self, \@renamed, \@deleted) where @renamed is array of |
| 93 |
new filenames that have been renamed, @deleted is array of new filenames |
| 94 |
that have been deleted. |
| 95 |
|
| 96 |
hook_after_create => code |
| 97 |
Will be called by after a new file is created. Code will be passed |
| 98 |
($self). |
| 99 |
|
| 100 |
This hook can be used to write a header to each file, e.g.: |
| 101 |
|
| 102 |
# hook_after_create |
| 103 |
my ($self) = @_; |
| 104 |
my $fh $self->handle; |
| 105 |
print $fh "header\n"; |
| 106 |
|
| 107 |
Since this is called indirectly by write(), locking is also already |
| 108 |
done. |
| 109 |
|
| 110 |
binmode => str |
| 111 |
If set to "1", will cause the file handle to be set: |
| 112 |
|
| 113 |
binmode $fh; |
| 114 |
|
| 115 |
which might be necessary on some OS, e.g. Windows when writing binary |
| 116 |
data. Otherwise, other defined values will cause the file handle to be |
| 117 |
set: |
| 118 |
|
| 119 |
binmode $fh, $value |
| 120 |
|
| 121 |
which can be used to set PerlIO layer(s). |
| 122 |
|
| 123 |
METHODS |
| 124 |
$obj = File::Write::Rotate->new(%args) |
| 125 |
Create new object. Known arguments: |
| 126 |
|
| 127 |
* dir => STR (required) |
| 128 |
|
| 129 |
Directory to put the files in. |
| 130 |
|
| 131 |
* prefix => STR (required) |
| 132 |
|
| 133 |
Name of files. The files will be named like the following: |
| 134 |
|
| 135 |
<prefix><period><suffix><rotate_suffix> |
| 136 |
|
| 137 |
"<period>" will only be given if the "period" argument is set. If |
| 138 |
"period" is set to "yearly", "<period>" will be "YYYY" (4-digit |
| 139 |
year). If "period" is "monthly", "<period>" will be "YYYY-MM" |
| 140 |
(4-digit year and 2-digit month). If "period" is "daily", "<period>" |
| 141 |
will be "YYYY-MM-DD" (4-digit year, 2-digit month, and 2-digit day). |
| 142 |
|
| 143 |
"<rotate_suffix>" is either empty string for current file; or .1, .2 |
| 144 |
and so on for rotated files. .1 is the most recent rotated file, .2 |
| 145 |
is the next most recent, and so on. |
| 146 |
|
| 147 |
An example, with "prefix" set to "myapp": |
| 148 |
|
| 149 |
myapp # current file |
| 150 |
myapp.1 # most recently rotated |
| 151 |
myapp.2 # the next most recently rotated |
| 152 |
|
| 153 |
With "prefix" set to "myapp", "period" set to "monthly", "suffix" |
| 154 |
set to ".log": |
| 155 |
|
| 156 |
myapp.2012-12.log # file name for december 2012 |
| 157 |
myapp.2013-01.log # file name for january 2013 |
| 158 |
|
| 159 |
Like previous, but additionally with "size" also set (which will |
| 160 |
also rotate each period file if it exceeds specified size): |
| 161 |
|
| 162 |
myapp.2012-12.log # file(s) for december 2012 |
| 163 |
myapp.2012-12.log.1 |
| 164 |
myapp.2012-12.log.2 |
| 165 |
myapp.2013-01.log # file(s) for january 2013 |
| 166 |
|
| 167 |
All times will use local time, so you probably want to set "TZ" |
| 168 |
environment variable or equivalent methods to set time zone. |
| 169 |
|
| 170 |
* suffix => STR (default: '') |
| 171 |
|
| 172 |
Suffix to give to file names, usually file extension like ".log". |
| 173 |
See "prefix" for more details. |
| 174 |
|
| 175 |
If you use a yearly period, setting suffix is advised to avoid |
| 176 |
ambiguity with rotate suffix (for example, is "myapp.2012" the |
| 177 |
current file for year 2012 or file with 2012 rotate suffix?) |
| 178 |
|
| 179 |
* size => INT (default: 10*1024*1024) |
| 180 |
|
| 181 |
Maximum file size, in bytes, before rotation is triggered. The |
| 182 |
default is 10MB (10*1024*1024) *if* "period" is not set. If "period" |
| 183 |
is set, no default for "size" is provided, which means files will |
| 184 |
not be rotated for size (only for period). |
| 185 |
|
| 186 |
* period => STR |
| 187 |
|
| 188 |
Can be set to either "daily", "monthly", or "yearly". If set, will |
| 189 |
automatically rotate after period change. See "prefix" for more |
| 190 |
details. |
| 191 |
|
| 192 |
* histories => INT (default: 10) |
| 193 |
|
| 194 |
Number of rotated files to keep. After the number of files exceeds |
| 195 |
this, the oldest one will be deleted. 0 means not to keep any |
| 196 |
history, 1 means to only keep .1 file, and so on. |
| 197 |
|
| 198 |
* buffer_size => INT (default: 0) |
| 199 |
|
| 200 |
Set initial value of buffer. See the "buffer_size" attribute for |
| 201 |
more information. |
| 202 |
|
| 203 |
* lock_mode => STR (default: 'write') |
| 204 |
|
| 205 |
Can be set to either "none", "write", or "exclusive". "none" |
| 206 |
disables locking and increases write performance, but should only be |
| 207 |
used when there is only one writer. "write" acquires and holds the |
| 208 |
lock for each write. "exclusive" acquires the lock at object |
| 209 |
creation and holds it until the the object is destroyed. |
| 210 |
|
| 211 |
Lock file is named "<prefix>"".lck". Will wait for up to 1 minute to |
| 212 |
acquire lock, will die if failed to acquire lock. |
| 213 |
|
| 214 |
* hook_before_write => CODE |
| 215 |
|
| 216 |
* hook_before_rotate => CODE |
| 217 |
|
| 218 |
* hook_after_rotate => CODE |
| 219 |
|
| 220 |
* hook_after_create => CODE |
| 221 |
|
| 222 |
See "ATTRIBUTES". |
| 223 |
|
| 224 |
* buffer_size => int |
| 225 |
|
| 226 |
* rotate_probability => float (between 0 < x < 1) |
| 227 |
|
| 228 |
If set, instruct to only check for rotation under a certain |
| 229 |
probability, for example if value is set to 0.1 then will only check |
| 230 |
for rotation 10% of the time. |
| 231 |
|
| 232 |
lock_file_path => STR |
| 233 |
Returns a string representing the complete pathname to the lock file, |
| 234 |
based on "dir" and "prefix" attributes. |
| 235 |
|
| 236 |
$fwr->write(@args) |
| 237 |
Write to file. Will automatically rotate file if period changes or file |
| 238 |
size exceeds specified limit. When rotating, will only keep a specified |
| 239 |
number of histories and delete the older ones. |
| 240 |
|
| 241 |
Does not append newline so you'll have to do it yourself. |
| 242 |
|
| 243 |
$fwr->compress |
| 244 |
Compress old rotated files and remove the uncompressed originals. |
| 245 |
Currently uses IO::Compress::Gzip to do the compression. Extension given |
| 246 |
to compressed file is ".gz". |
| 247 |
|
| 248 |
Will not lock writers, but will create "<prefix>""-compress.pid" PID |
| 249 |
file to prevent multiple compression processes running and to signal the |
| 250 |
writers to postpone rotation. |
| 251 |
|
| 252 |
After compression is finished, will remove the PID file, so rotation can |
| 253 |
be done again on the next "write()" if necessary. |
| 254 |
|
| 255 |
FAQ |
| 256 |
Why use autorotating file? |
| 257 |
Mainly convenience and low maintenance. You no longer need a separate |
| 258 |
rotator process like the Unix logrotate utility (which when accidentally |
| 259 |
disabled or misconfigured will cause your logs to stop being rotated and |
| 260 |
grow indefinitely). |
| 261 |
|
| 262 |
What is the downside of using FWR (and LDFR)? |
| 263 |
Mainly (significant) performance overhead. At (almost) every "write()", |
| 264 |
FWR needs to check file sizes and/or dates for rotation. Under default |
| 265 |
configuration (where "lock_mode" is "write"), it also performs locking |
| 266 |
on each "write()" to make it safe to use with multiple processes. Below |
| 267 |
is a casual benchmark to give a sense of the overhead, tested on my Core |
| 268 |
i5-2400 3.1GHz desktop: |
| 269 |
|
| 270 |
Writing lines in the size of ~ 200 bytes, raw writing to disk (SSD) has |
| 271 |
the speed of around 3.4mil/s, while using FWR it goes down to around |
| 272 |
~13k/s. Using "lock_mode" "none" or "exclusive", the speed is ~52k/s. |
| 273 |
|
| 274 |
However, this is not something you'll notice or need to worry about |
| 275 |
unless you're writing near that speed. |
| 276 |
|
| 277 |
If you need more speed, you can try setting "rotate_probability" which |
| 278 |
will cause FWR to only check for rotation probabilistically, e.g. if you |
| 279 |
set this to 0.1 then checks will only be done in about 1 of 10 writes. |
| 280 |
This can significantly reduce the overhead and increase write speed |
| 281 |
several times (e.g. 5-8 times), but understand that this will make the |
| 282 |
writes "overflow" a bit, e.g. file sizes will exceed for a bit if you do |
| 283 |
size-based rotation. More suitable if you only do size-based rotation |
| 284 |
since it is usually okay to exceed sizes for a bit. |
| 285 |
|
| 286 |
I want a filehandle instead of a File::Write::Rotate object! |
| 287 |
Use Tie::Handle::FileWriteRotate. |
| 288 |
|
| 289 |
HOMEPAGE |
| 290 |
Please visit the project's homepage at |
| 291 |
<https://metacpan.org/release/File-Write-Rotate>. |
| 292 |
|
| 293 |
SOURCE |
| 294 |
Source repository is at |
| 295 |
<https://github.com/perlancar/perl-File-Write-Rotate>. |
| 296 |
|
| 297 |
BUGS |
| 298 |
Please report any bugs or feature requests on the bugtracker website |
| 299 |
<https://rt.cpan.org/Public/Dist/Display.html?Name=File-Write-Rotate> |
| 300 |
|
| 301 |
When submitting a bug or request, please include a test-file or a patch |
| 302 |
to an existing test-file that illustrates the bug or desired feature. |
| 303 |
|
| 304 |
SEE ALSO |
| 305 |
Log::Dispatch::FileRotate, which inspires this module. Differences |
| 306 |
between File::Write::Rotate (FWR) and Log::Dispatch::FileRotate (LDFR) |
| 307 |
are as follows: |
| 308 |
|
| 309 |
* FWR is not part of the Log::Dispatch family. |
| 310 |
|
| 311 |
This makes FWR more general to use. |
| 312 |
|
| 313 |
For using together with Log::Dispatch/Log4perl, I have also written |
| 314 |
Log::Dispatch::FileWriteRotate which is a direct (although not a |
| 315 |
perfect drop-in) replacement for Log::Dispatch::FileRotate. |
| 316 |
|
| 317 |
* Secondly, FWR does not use Date::Manip. |
| 318 |
|
| 319 |
Date::Manip is relatively large (loading Date::Manip 6.37 equals to |
| 320 |
loading 34 files and ~ 22k lines; while FWR itself is only < 1k |
| 321 |
lines!) |
| 322 |
|
| 323 |
As a consequence of this, FWR does not support DatePattern; instead, |
| 324 |
FWR replaces it with a simple daily/monthly/yearly period. |
| 325 |
|
| 326 |
* And lastly, FWR supports compressing and rotating compressed old |
| 327 |
files. |
| 328 |
|
| 329 |
Using separate processes like the Unix logrotate utility means |
| 330 |
having to deal with yet another race condition. FWR takes care of |
| 331 |
that for you (see the compress() method). You also have the option |
| 332 |
to do file compression in the same script/process if you want, which |
| 333 |
is convenient. |
| 334 |
|
| 335 |
There is no significant overhead difference between FWR and LDFR (FWR is |
| 336 |
slightly faster than LDFR on my testing). |
| 337 |
|
| 338 |
Tie::Handle::FileWriteRotate and Log::Dispatch::FileWriteRotate, which |
| 339 |
use this module. |
| 340 |
|
| 341 |
AUTHOR |
| 342 |
perlancar <[email protected]> |
| 343 |
|
| 344 |
COPYRIGHT AND LICENSE |
| 345 |
This software is copyright (c) 2019, 2016, 2015, 2014, 2013, 2012 by |
| 346 |
[email protected]. |
| 347 |
|
| 348 |
This is free software; you can redistribute it and/or modify it under |
| 349 |
the same terms as the Perl 5 programming language system itself. |
| 350 |
|