Debian Codegraph @

doc/html/ncftpspooler.html

raw
1 <HTML>
2 <HEAD>
3 <TITLE>ncftpspooler(1) manual page</TITLE>
4 </HEAD>
5 <BODY bgcolor=white>
6
7 <H2><B>Name</B></H2>
8
9 <ul>
10 <P>
11 <b>ncftpspooler</b> - Global batch FTP job processor daemon
12 <P>
13 </ul>
14
15 <H2><B>Synopsis</B></H2>
16
17 <ul>
18 <P>
19 <B>ncftpspooler</B> -d [<I>options</I>]
20 <P>
21 <P>
22 <B>ncftpspooler</B> -l [<I>options</I>]
23 <P>
24 </ul>
25
26 <center>
27 <p>
28 <form action="https://www.paypal.com/cgi-bin/webscr" method="post">
29 <input type="hidden" name="cmd" value="_s-xclick">
30 <input type="hidden" name="hosted_button_id" value="5JS76LUNSD3HU">
31 <input type="image" src="https://www.paypalobjects.com/en_US/i/btn/btn_donateCC_LG.gif" border="0" name="submit" alt="PayPal - The safer, easier way to pay online!">
32 <img alt="" border="0" src="https://www.paypalobjects.com/en_US/i/scr/pixel.gif" width="1" height="1">
33 </form>
34 </center>
35
36 <H2><B>Options</B></H2>
37
38 <P>
39 Command line flags:
40 <ul>
41
42
43 <DL>
44
45 <DT><B>-d</B> </DT>
46 <DD> Begin background processing of FTP jobs in the
47 designated FTP job queue directory.
48 <P>
49 </DD>
50
51 <DT><B>-D</B> </DT>
52 <DD> This is like <B>-d</B>, except that the process does not
53 become a background daemon process.
54 </DD>
55
56 <DT><B>-q</B> <I>XX</I> </DT>
57 <DD>Use this option to specify a directory to use as
58 the FTP job queue instead of the default directory,
59 <tt>/var/spool/ncftp</tt>.
60 <P>
61 </DD>
62
63 <DT><B>-o</B> <I>XX</I> </DT>
64 <DD>Use this option to specify a filename to use as
65 the log file. By default, (and rather inappropriately)
66 the program simply uses a file called <I>log</I>
67 in the job queue directory. If you don't want a
68 log, use this option to specify
69 <tt>/dev/null</tt>.
70 <P>
71 </DD>
72
73 <DT><B>-l</B> </DT>
74 <DD> Lists the contents of the job queue directory.
75 <P>
76 </DD>
77
78 <DT><B>-s</B> <I>XX</I> </DT>
79 <DD>When the job queue is empty, the program sleeps
80 120 seconds and then checks again to see if a new
81 job has been submitted. Use this option to change
82 the number of seconds used for this delay.
83 <P>
84 </DD>
85 </DL>
86 </ul>
87
88 <H2><B>Description</B></H2>
89
90 <P>
91 The <I>ncftpspooler</I> program evolved from the <I>ncftpbatch</I> program.
92 The <I>ncftpbatch</I> program was originally designed as a
93 ``personal FTP spooler'' which would process a single
94 background job a particular user and exit when it finished;
95 the <I>ncftpspooler</I> program is a ``global FTP
96 spooler'' which stays running and processes background
97 jobs as they are submitted.
98 <P>
99 <P>
100 The job queue directory is monitored for specially-named
101 and formatted text files. Each file serves as a single
102 FTP job. The name of the job file contains the type of
103 FTP job (<I>get</I> or <I>put</I>), a timestamp indicating the earliest
104 the job should be processed, and optionally some additional
105 information to make it easier to create unique job
106 files (i.e. a sequence number). The contents of the job
107 files have information such as the remote server machine
108 to FTP to, username, password, remote pathname, etc.
109 <P>
110 <P>
111 Your job queue directory must be readable and writable by
112 the user that you plan to run <I>ncftpspooler</I> as, so that
113 jobs can be removed or renamed within the queue.
114 <P>
115 <P>
116 More importantly, the user that is running the program
117 will need adequate privileges to access the local files
118 that are involved in the FTPing. I.e., if your spooler is
119 going to be processing jobs which upload files to remote
120 servers, then the user will need read permission on the
121 local files that will be uploaded (and directory access
122 permission the parent directories). Likewise, if your
123 spooler is going to be processing jobs which download
124 files, then the user would need to be able to write to the
125 local directories.
126 <P>
127 <P>
128 Once you have created your spool directory with appropriate
129 permissions and ownerships, you can run
130 <I>ncftpspooler</I> <I>-d</I> to launch the spooler daemon. You can run
131 additional spoolers if you want to process more than FTP
132 job from the same job queue directory simultaneously. You
133 can then monitor the log file (i.e., using <tt>tail -f</tt> ) to
134 track the progress of the spooler. Most of the time it
135 won't be doing anything, unless job files have appeared in
136 the job queue directory.
137 <P>
138
139 <H2><B>Job File Names</B></H2>
140
141 <P>
142 When the <I>ncftpspooler</I> program monitors the job queue
143 directory, it ignores any files that do not follow the
144 naming convention for job files. The job files must be
145 prefixed in the format of <I>X-YYYYMMDD-hhmmss</I> where <I>X</I>
146 denotes a job type, <I>YYYY</I> is the four-digit year, <I>MM</I> is the
147 two-digit month number, <I>DD</I> is the two-digit day of the
148 month, <I>hh</I> is the two-digit hour of the day (00-23), <I>mm</I> is
149 the two-digit minute, and <I>ss</I> is the two-digit second. The
150 date and time represent the earliest time you want the job
151 to be run.
152 <P>
153 <P>
154 The job type can be <I>g</I> for a get (download from remote
155 host), or <I>p</I> for aput (upload to remote host).
156 <P>
157 <P>
158 As an example, if you wanted to schedule an upload to
159 occur at 11:45 PM on December 7, 2001, a job file could be
160 named
161 <P>
162 <ul>
163 <tt>p-20011207-234500</tt>
164 </ul>
165 <P>
166 In practice, the job files include additional information
167 such as a sequence number or process ID. This makes it
168 easier to create unique job file names. Here is the same
169 example, with a process ID and a sequence number:
170 <P>
171 <ul>
172 <tt>p-20011207-234500-1234-2</tt>
173 </ul>
174 <P>
175 When submitting job files to the queue directory, be sure
176 to use a dash character after the <I>hhmmss</I> field if you
177 choose to append any additional data to the job file name.
178 <P>
179
180 <H2><B>Job File Contents</B></H2>
181
182 <P>
183 Job files are ordinary text files, so that they can be
184 created by hand. Each line of the file is a key-pair in
185 the format <I>variable</I>=<I>value</I>, or is a comment line beginning
186 with an octothorpe character (<I>#</I>), or is a blank line.
187 Here is an example job file:
188 <P>
189 <ul>
190 <pre>
191 # This is a NcFTP spool file entry.
192 job-name=g-20011016-100656-008299-1
193 op=get
194 hostname=ftp.freebsd.org
195 xtype=I
196 passive=1
197 remote-dir=pub/FreeBSD
198 local-dir=/tmp
199 remote-file=README.TXT
200 local-file=readme.txt
201 </pre>
202 </ul>
203 <P>
204 Job files are flexible since they follow an easy-to-use
205 format and do not have many requirements, but there are a
206 few mandatory parameters that must appear for the spooler
207 to be able to process the job.
208 <P>
209
210 <UL><DL>
211 <DT><I>op</I></DT>
212 <DD>
213 The operation (job type) to perform. Valid values
214 are <I>get</I> and <I>put</I>.
215 </DD>
216
217 <P>
218 <DT><I>hostname</I></DT>
219 <DD>
220 The remote host to FTP to. This may be an IP
221 address or a DNS name (i.e. ftp.example.com).
222 </DD>
223 </DL></UL>
224
225 <P>
226 For a regular <I>get</I> job, these parameters are required:
227
228 <UL><DL>
229 <DT><I>remote-file</I></DT>
230 <DD>
231 The pathname of the file to download from the
232 remote server.
233 </DD>
234
235 <P>
236 <DT><I>local-file</I></DT>
237 <DD>
238 The pathname to use on the local server for the
239 downloaded file.
240 </DD>
241 </DL></UL>
242
243 <P>
244 For a regular <I>put</I> job, these parameters are required:
245
246 <P>
247 <UL><DL>
248 <DT><I>local-file</I></DT>
249 <DD>
250 The pathname of the file to upload to the remote
251 server.
252 </DD>
253
254 <P>
255 <DT><I>remote-file</I></DT>
256 <DD>
257 The pathname to use on the remote server for the
258 uploaded file.
259 </DD>
260 </DL></UL>
261
262 <P>
263 For a recursive <I>get</I> job, these parameters are required:
264
265 <P>
266 <UL><DL>
267 <DT><I>remote-file</I></DT>
268 <DD>
269 The pathname of the file or directory to download
270 from the remote server.
271 </DD>
272
273 <P>
274 <DT><I>local-dir</I></DT>
275 <DD>
276 The directory pathname to use on the local server
277 to contain the downloaded items.
278 </DD>
279 </DL></UL>
280
281 <P>
282 For a recursive <I>put</I> job, these parameters are required:
283
284 <P>
285 <UL><DL>
286 <DT><I>local-file</I></DT>
287 <DD>
288 The pathname of the file or directory to upload to
289 the remote server.
290 </DD>
291
292 <P>
293 <DT><I>remote-dir</I></DT>
294 <DD>
295 The directory pathname to use on the remote server
296 to contain the uploaded items.
297 </DD>
298 </DL></UL>
299
300 <P>
301 The rest of the parameters are optional. The spooler will
302 attempt to use reasonable defaults for these parameters if
303 necessary.
304
305 <P>
306 <UL><DL>
307 <DT><I>user</I> </DT>
308 <DD> The username to use to login to the remote server.
309 Defaults to ``anonymous'' for guest access.
310 </DD>
311
312 <P>
313 <DT><I>pass</I> </DT>
314 <DD> The password to use in conjunction with the username
315 to login to the remote server.
316 </DD>
317
318 <P>
319 <DT><I>acct</I> </DT>
320 <DD> The account to use in conjunction with the username
321 to login to the remote server. The need to
322 specify this parameter is extremely rare.
323 </DD>
324
325 <P>
326 <DT><I>port</I> </DT>
327 <DD> The port number to use in conjunction with the
328 remote hostname to connect to the remote server.
329 Defaults to the standard FTP port number, 21.
330 </DD>
331
332 <P>
333 <DT><I>host-ip</I></DT>
334 <DD>
335 The IP address to use in conjunction with the
336 remote hostname to connect to the remote server.
337 This parameter can be used in place of the <I>host</I><B>_</B>n<I>ame</I>
338 parameter, but one or the other must be used.
339 This parameter is commonly included along with the
340 <I>hostname</I> parameter as supplemental information.
341 </DD>
342
343 <P>
344 <DT><I>xtype</I></DT>
345 <DD>
346 The transfer type to use. Defaults to binary
347 transfer type (TYPE I). Valid values are <I>I</I> for
348 binary, <I>A</I> for ASCII text.
349 </DD>
350
351 <P>
352 <DT><I>passive</I></DT>
353 <DD>
354 Whether to use FTP passive data connections (PASV)
355 or FTP active data connections (PORT). Valid values
356 are <I>0</I> for active, <I>1</I> for passive, or <I>2</I> to try
357 passive, then fallback to active. The default is
358 <I>2</I>.
359 </DD>
360
361 <P>
362 <DT><I>recursive</I></DT>
363 <DD>
364 This can be used to transfer entire directory
365 trees. By default, only a single file is transferred.
366 Valid values are <I>yes</I> or <I>no</I>.
367 </DD>
368
369 <P>
370 <DT><I>delete</I></DT>
371 <DD>
372 This can be used to delete the source file on the
373 source machine after successfully transferring the
374 file to the destination machine. By default,
375 source files are not deleted. Valid values are
376 <I>yes</I> or <I>no</I>.
377 </DD>
378
379 <P>
380 <DT><I>job-name</I></DT>
381 <DD>
382 This isn't used by the program, but can be used by
383 an entity which is automatically generating job
384 files. As an example, when using the <I>-bbb</I> flag
385 with <I>ncftpput</I>, it creates a job file on stdout
386 with a <I>job-name</I> parameter so you can easily copy
387 the file to the job queue directory with the suggested
388 job name as the job file name.
389 </DD>
390
391 <P>
392 <DT><I>pre-ftp-command</I><BR>
393 <I>post-ftp-command</I></DT>
394 <DD>
395 These parameters correspond to the <I>-W</I>, and <I>-Y</I>
396 options of <I>ncftpget</I> and <I>ncftpput</I>. It is important
397 to note that these refer to RFC959 File Transfer
398 Protocol commands and <B>not</B> shell commands, nor commands
399 used from within <tt>/usr/bin/ftp</tt> or <i>ncftp</i>.
400 </DD>
401
402 <P>
403 <DT><I>pre-shell-command</I><BR>
404 <I>post-shell-command</I></DT>
405 <DD>
406 These parameters provide hooks so you can run a
407 custom program when an item is processed by the
408 spooler. Valid values are pathnames to scripts or
409 executable programs. Note that the value must not
410 contain any command-line arguments -- if you want
411 to do that, create a shell script and have it run
412 your program with the command-line arguments it
413 requires.
414 </DD>
415 </DL></UL>
416
417 <P>
418 Generally speaking, <I>post-shell-command</I> is much more useful
419 than <I>pre-shell-command</I> since if you need to use these
420 options you're more likely to want to do something after
421 the FTP transfer has completed rather than before. For
422 example, you might want to run a shell script which pages
423 an administrator to notify her that her 37 gigabyte file
424 download has completed.
425 <P>
426 <P>
427 When your custom program is run, it receives on standard
428 input the contents of the job file (i.e. several lines of
429 <I>variable</I>=<I>value</I> key-pairs), as well as additional data the
430 spooler may provide, such as a <I>result</I> key-pair with a textual
431 description of the job's completion status.
432 <P>
433 <P>
434 <I>post-shell-command</I> update a log file named
435 <tt>/var/log/ncftp_spooler</tt>.
436
437 <ul>
438 <pre>
439 #!/usr/bin/perl -w
440
441 use strict;
442
443 my ($line);
444 my (%params) = ();
445
446 while (defined($line = &lt;STDIN&gt;)) {
447 $params{$1} = $2 if ($line =~ /^([^=\#\s]+)=(.*)/);
448 }
449
450 if ((defined($params{"result"})) &amp;&amp; ($params{"result"} =~ /^Succeeded/)) {
451 open(LOG, "&gt;&gt; /var/log/ncftp_spooler.log") or exit(1);
452 print LOG "DOWNLOAD" if ($params{"op"} eq "get");
453 print LOG "UPLOAD" if ($params{"op"} eq "put");
454 print LOG " ", $params{"local-file"}, "\n";
455 close(LOG);
456 }
457 </pre>
458 </ul>
459
460 <H2><B>Diagnostics</B></H2>
461
462 <P>
463 The log file should be examined to determine if any <I>ncftpspooler</I>
464 processes are actively working on jobs. The log
465 contains copious amounts of useful information, including
466 the entire FTP control connection conversation between the
467 FTP client and server.
468 <P>
469
470 <H2><B>Bugs</B></H2>
471
472 <P>
473 The <I>recursive</I> option may not be reliable since <I>ncftpspooler</I>
474 depends on functionality which may or may not be
475 present in the remote server software. Additionally, even
476 if the functionality is available, <I>ncftpspooler</I> may need
477 to use heuristics which cannot be considered 100% accurate.
478 Therefore it is best to create individual jobs for
479 each file in the directory tree, rather than a single
480 recursive directory job.
481 <P>
482 <P>
483 For resumption of downloads to work, the remote server
484 must support the FTP <I>SIZE</I> and <I>MDTM</I> primitives. Most modern
485 FTP server software can do this, but there are still a
486 number of bare-bones <I>ftpd</I> implementations which do not.
487 In these cases, <I>ncftpspooler</I> will re-download the file in
488 entirety each time until the download succeeds.
489 <P>
490 <P>
491 The program needs to be improved to detect jobs that have
492 no chance of ever completing successfully. There are
493 still a number of cases where jobs can get spooled but get
494 retried over and over again until a vigilant sysadmin manually
495 removes the jobs.
496 <P>
497 <P>
498 The spool files may contain usernames and passwords stored
499 in cleartext. These files should not be readable by any
500 user except the user running the program!
501 <P>
502
503 <H2>Author</H2>
504 <blockquote>
505 <P>
506 Mike Gleason, <a href="http://www.ncftp.com">NcFTP Software</a>.
507 </blockquote>
508 <H2>See Also</H2>
509 <blockquote>
510 <P>
511 <I>ncftpput(1)</I>, <I>ncftpget(1),</I>
512 <I>ncftp(1)</I>, <I>ftp(1)</I>, <I>rcp(1)</I>, <I>tftp(1)</I>. <P>
513 <I>LibNcFTP</I> (<a href="http://www.ncftp.com/libncftp/">http://www.ncftp.com/libncftp/</a>).
514 </blockquote>
515
516 </BODY></HTML>