Debian Codegraph @

ChangeLog

raw
1
2 Enhanced CTorrent Change Log
3 _________________________________________________________________
4
5 Changes for "dnh3.3.2" Release
6
7 * [1954631] Fixed incorrect ordering of dictionary keys in created
8 torrent metainfo file.
9 * Remove unneeded buffer allocation in bencode_buf.
10 * Insure appropriate file can be closed if needed.
11 * Fix double-cancel in StopDownload.
12 * Incorrectly set not_interested state in RequestPiece when it
13 should set m_standby, as there are still outstanding requests.
14 * Corrected a problem in choke oscillation handling.
15 * Avoid re-requesting the most recently requested piece from a peer,
16 to reduce choke oscillation impact.
17 * Set multpeer on choke oscillation, to accept the affected piece.
18 * Don't print file completion details with -c if no files existed.
19 * Set errno in low-level btfiles functions to facilitate better
20 error messages.
21 * Set ENOMEM on alloc failure in truncate routine
22 * Corrected some CTCS error handling, which could produce an
23 odd/incorrect message about sending the torrent.
24 * Fix CTCS/Console recursion loop triggered by a network problem,
25 eventually blowing out the stack.
26 * Remove the .bf file when -c is used. This prevents an old bitfield
27 state hanging around if the torrent is found to be complete.
28 * Indicate when a file is being created, even when trying to use
29 sparse files.
30 * Don't print file completion details when no data is present.
31 * Buffer was not freed when using -a (memory waste).
32 * Additional "now" time updates to correct timing flaws. Too many
33 disk accesses could be performed because the time was not being
34 updated between operations.
35 * Fix request URL encoding for non-ASCII locales.
36 * Cache management bug due to expiration during I/O. This could
37 manifest as a bad hash check or incomplete piece after download.
38 * Correction in cache aging of a piece.
39 * Console is unresponsive when downloading slowly (IsIdle problem).
40 _________________________________________________________________
41
42 Changes for "dnh3.3.1" Release
43
44 Bug Fixes
45 * [1941536] Fix restoration of terminal mode at exit.
46 * Handle errors from select(). (Could cause high CPU utilization and
47 unresponsiveness.)
48 * [1946210] Fix incorrect string allocation sizes. (Could cause
49 memory corruption or double-free errors.)
50 * Delete file I/O buffers at exit.
51 * Fix build warning from bitfield.cpp.
52 _________________________________________________________________
53
54 Changes for "dnh3.3" Release
55
56 Features and Performance
57 * Faster restart: Download of pieces that are known to be missing
58 can begin immediately.
59 * Added a workaround for the current lack of announce-list (multiple
60 trackers) support. If the torrent file contains an announce-list,
61 up to 9 alternate announce URLs (trackers) will be listed in the
62 torrent information. The -u option can be used to specify the
63 announce URL, either by number from the list of alternates or
64 directly as a string.
65 * Allow -X in conjunction with -c (for automated completion check).
66 * SO_REUSEADDR is now used when -p is specified, so that the client
67 can restart immediately on the given port.
68 * Daemon mode (-d) now closes the standard file descriptors and
69 reopens them to /dev/null. This complies with most people's
70 expectations of a daemon. If -d is specified twice, standard I/O
71 descriptors which are not a terminal (i.e. which have been
72 redirected) are kept intact as was the previous behavior. This
73 allows a fully detached process from which output can be captured.
74 * Add tracker update & restart actions (from CTCS) to operator menu.
75 * When creating a torrent, the following new options are available:
76 -p Private (disable peer exchange)
77 -c comment Include a comment/description
78 Note that -t must now be the first option given when creating a
79 torrent.
80 * A new option -I allows specifying the public IP address which the
81 tracker should distribute to peers.
82 * Enabled changing CTCS server from CTCS.
83 * When changing bandwidth limits from the console, remind the user
84 that CTCS may override changes.
85 * It is now possible to stop using CTCS from the console. Use the
86 "S" option and give a single colon (:) as the input.
87 * The "private", "comment", and "created by" fields are now
88 recognized and printed (if present) when the torrent meta info is
89 shown.
90 * Improved on-screen advisement regarding bit field file at startup.
91 * Tracker reporting data has been added to the detailed status, as
92 well as failed hashes, duplicate blocks, and unwanted blocks
93 counts.
94 * CPU usage is printed to Verbose output when Detailed Status is
95 requested from operator menu.
96 * Input is now turned off if not needed. This makes things behave
97 better if, for example, -x is used and the output piped to less.
98 * Attempt to avoid wrapping/scrolling when printing the status line
99 on a narrow terminal/window.
100 * A couple of optimizations analyzing peer bitfields.
101 * Bitfield operation optimizations.
102 * Efficiency improvement in PendingQueue functions.
103 * Improved GetDownloads efficiency.
104 * Reduced memory usage when seeding, download is stalled, or using
105 -x.
106 * Rate measurement calculations are now cached and recomputed only
107 when necessary.
108 * Improved methods of determining local IP address.
109 * The "ip" field is sent in the tracker address if the public IP
110 address was determined.
111 * Reduction of download duplication and improvement of tracker
112 reporting
113 + Improved piece reassignment.
114 o Reassign only if proposer's queue is empty.
115 o Consider any piece in the slower peer's queue.
116 o Improve download rate comparison method.
117 o Allow reassignment in endgame mode.
118 + Improved data counting and reporting.
119 + Hash check failure handling improvement
120 o If hash check fails, don't re-request the same piece
121 from that peer.
122 o Increased weight of hash failure.
123 o Reduced peer error threshold.
124 + Improved randomness of CopyShuffle.
125 + Reduce duplicate data receipt
126 o Avoid duplication to high-latency/slow peers.
127 o When duplicating or reassigning requests, move immediate
128 incoming slices to the end of the queue.
129 + Accept cancelled data from a peer if still needed.
130 + Duplicate request management
131 o Spread request duplication more evenly among outstanding
132 pieces.
133 o Limit duplicate requests for a piece to the number of
134 outstanding slices.
135 o Track the number of pieces requested as duplicates.
136 o Track pieces for which slices have been requested or
137 received from multiple peers.
138 + Detect a duplication inconsistency in Pending.
139 + Improve choke oscillation detection and handling.
140 + HealthCheck improvement to reduce redundant receipts from
141 high latency peers.
142 * Support systems that do not provide random(3).
143
144 Bug Fixes
145 * Completed torrent could be reported as incomplete when restarting.
146 This was the reported/observed symptom; potentially it could also
147 be seen as failed hash-check of downloaded pieces or uploading of
148 bad data (failed hash-check at a remote peer).
149 * Fixed crash which could happen when sending data without an upload
150 limit.
151 * Surplus bits in the last byte of a bitfield could be set.
152 * Don't say "stalled" if still checking and nothing to download yet.
153 * Don't allow the -n or -X options (from CTCS and console) if
154 download is already complete.
155 * Correct handling if -s is used with a single-file torrent and a
156 directory of the specified name exists.
157 * Exit with error if tracker setup fails.
158 * Don't print an error when a known but inactive configuration
159 option is received from CTCS.
160 * Check for bad/invalid input from CTCS.
161 * Fixed a bug that could corrupt (loop) the peer list, resulting in
162 a hard-loop (nonresponsiveness).
163 * Exit (with error) after discovering that a file is not the correct
164 size at startup.
165 * Fixed a bug with input prompting (affects CTCS password entry).
166 * [1886549] Fix torrent creation with files over 4GB.
167 * Improved CTCS status updating, and accuracy for end-of-download
168 and terminating cases.
169 * Upload/download now stops upon initiating Quit.
170 * Improved cache allocation when download stalls or unstalls.
171 * [1863204] Skip I/O for zero-length files.
172 * Condition reversal in counting hash-check error against a peer.
173
174 Code Cleanup
175 * Removed declaration of unused variable cfg_max_bandwidth.
176 * Rearrangement of some bitfield functions & declarations.
177 * Consolidate peer error counting and handling into PeerError.
178 _________________________________________________________________
179
180 Changes for "dnh3.2" Release
181
182 Features and Performance
183 * Various code optimizations in cache searching and management.
184 * Added proactive flushing of the oldest cache entry during idle
185 time when the cache is full.
186 * Flush the cache in piece order if/when the entire cache is
187 flushed.
188 * Prefetch uncached slices when a downloading piece is nearly
189 complete.
190 * Try to reduce thrashing if a small cache is specified.
191 * Flushing completed pieces to disk is now given a higher priority
192 than hash-checking existing pieces.
193 * An informational warning is given if any filenames contain
194 non-printable characters and the -T option is not used.
195 * Added more methods of detecting connections from the client to
196 itself.
197 * If the tracker doesn't supply a peer count, count the peers it
198 gives for connection instead. (This can be off by 1 if you're
199 using NAT and a lame tracker gives you yourself as a peer, but you
200 can't connect.)
201 * Don't let initial hash-checking prevent checking tracker, ctcs,
202 and console.
203 * Improved efficiency of peer status/prefetch checking.
204 * A 16K (default slice size) file I/O buffer is now used to
205 potentially improve read/write performance.
206 * Added support for systems without the strncasecmp() library
207 function.
208
209 Bug Fixes
210 * Fixed a crash at program exit.
211 * Fixed an error when reading an empty file.
212 * Fixed a potential cache corruption issue.
213 * Bad piece data wasn't always removed from cache.
214 * Updated builtin SHA1 code to include a fix or two and take
215 advantage of autoconf (including endianness; cross-compilers take
216 note).
217 * Improved/corrected the method of determining the local IP address.
218 * Corrected the comparison of local and peer IP address.
219 * Corrected peer connection failure detection (could cause peer
220 disconnects).
221 * Corrected a potential problem in the peer handshake handling code.
222 * Corrected a problem that could prevent or delay receiving the
223 peer's bitfield.
224 * Fixed a problem with handling of tracker response data. Now
225 separate send and receive buffers are used for tracker
226 communication.
227 * Corrected end-of-header check in httpencode.cpp (tracker
228 response).
229 * Various timing-related fixes to improve maximum transfer rates.
230 * Avoid running too many idle-time tasks at a time.
231 * Improved the upload timing when not using cache.
232 * Adjusted the CTCS seed time (-e) option processing to make the
233 result more intuitive (it could be perceived as being an hour
234 off).
235 * Fixed a problem where the torrent metainfo would not be shown if
236 input was redirected. Also added input EOF/error detection for
237 cases not previously covered.
238 * Fix console input on Solaris (and potentially some other
239 platforms).
240
241 Code Cleanup
242 * Additional verbose output for cache management.
243 * Moved strnstr() from httpencode.cpp to compat.c.
244 * Improve handling of tracker contact intervals.
245 * Optimized PeekMessage function.
246 * Removed unnecessary headers from downloader.cpp.
247 _________________________________________________________________
248
249 Changes for "dnh3.1" Release
250
251 Features and Performance
252 * Cached completed pieces are flushed to disk during idle time
253 rather than immediately.
254 * When the cache is full, a single entry is now flushed to make room
255 for new data rather than flushing the entire cache.
256 * Management of open files has been enabled in order to reduce the
257 number of active file descriptors needed for large torrents.
258 Detection and reporting of related errors is also improved.
259 * Files are now opened in read-only mode when seeding.
260 * The client will not connect to the tracker when quitting if it has
261 not successfully made contact before.
262 * A new command line option "-T" substitutes a hex representation of
263 any non-printable characters in filenames; an underscore will be
264 inserted between any such sequences and regular text. This applies
265 to printing the name of the file as well as accessing it on disk,
266 so you need to use it (or not) consistently across multiple runs
267 of the same torrent. It does not apply to names specified with the
268 -s option.
269 * A new command line option "-X" specifies a "user exit" command to
270 run upon download completion. The string parameter will be passed
271 to the sh shell for execution; see the system() man page for
272 further clarification. You will need to quote this string on the
273 command line in order to identify it as a single parameter and
274 prevent special characters from being interpreted. Some
275 substitution sequences beginning with ampersand (&) are available;
276 note that it's a good idea to quote the sequence within the string
277 so that the substituted names will be quoted in the final command.
278 + "&d" will be replaced by the name of the directory or
279 filename that contains the downloaded data.
280 + "&t" will be replaced by the name of the torrent metainfo
281 file (with path, if it was specified that way to ctorrent).
282 + "&w" will be replaced by the client's working directory.
283 Note that the fork() and system() system/library functions are
284 used for best portability. As a result, significant extra memory
285 may be used while the specified command is running. Be advised to
286 use this feature as a trigger mechanism to update a file or run a
287 short script that kicks off a background task rather than for
288 executing a longer task directly.
289 * A new command line option "-d" specifies "daemon mode". The client
290 will detach itself from the terminal session and become a
291 background process. This option is also available from the
292 operator menu.
293 * Prefetch and total sent counts are reported with the cache stats
294 in verbose mode. Note that these stats are all close
295 approximations based on the default slice size of 16KB.
296 * The -n option now accepts a list of files to download in priority
297 order. Groups of simultaneous files can be specified, and
298 "resuming normal behavior" after completion is now optional (and
299 not the default). See the User's Guide for more detailed
300 information.
301 * The partial completion indicator ("P:") now shows percentages
302 rather than number of pieces when using the new status line
303 format. The download time remaining for the current set is also
304 shown.
305 * The CTCS protocol version has been incremented. Please download
306 the new version of CTCS to take advantage of new features. This
307 provides support for additional file details and prioritization,
308 and for auto-configuration of available client configuration
309 options in CTCS.
310 * Output channels can now be changed via CTCS.
311 * The input channel can now be redirected from the operator menu and
312 CTCS.
313 * Added "off" as an additional target for input and output channels.
314 * Added support for systems that do not have snprintf() and/or
315 vsnprintf().
316
317 Bug Fixes
318 * Fixed a missing maxfd variable initialization in the main loop,
319 which could cause high CPU utilization. (maxfd patch)
320 * Detect if the clock_gettime library function is missing and
321 emulate it. This facilitates building on Mac OSX. (clock_gettime
322 patch)
323 * Avoid putting the "key" field into the tracker request twice.
324 * Tracker redirection is now handled correctly and works.
325 * Fixed problems with detection and reporting of tracker connection
326 errors.
327 * Error reading from disk with -c option is now reported.
328 * Fixed a looping problem when the torrent begins with a zero-length
329 file or a file of exactly one piece.
330 * Fixed the way variable argument lists are handled in the console
331 code, which could cause crashes. (console patch)
332 * Fixed a problem flushing downloaded data to disk in certain
333 situations. (flush patch)
334 * Remove sockets from the list to check for peers disconnected
335 during processing.
336 * Updated fdset handling in peerlist, tracker, and ctcs components
337 to set appropriate states in all cases.
338 * Fixed a potential crash when receiving a message from a peer
339 (after processing it). (msg patch)
340 * Fixed an issue with rate measurement on NSLU2 devices. This was
341 due to an apparent compiler bug (unconfirmed but raised to
342 developers). There are other areas where the same effect can show
343 itself (with the -E seed ratio option for instance) that cannot be
344 so easily worked around. (bwrate patch)
345 * Fixed an issue with null/redirected standard input.
346 * If the time changes backward by a second, it is disregarded. This
347 is not really a program bug fix, but a workaround for system clock
348 glitches (observed on an NSLU2).
349
350 Code Cleanup
351 * Tracker request construction is somewhat consolidated and
352 simplified.
353 * Split the FillFDSet function into two functions of distinct roles
354 and more manageable size.
355 _________________________________________________________________
356
357 Changes for "dnh3" Release
358
359 Important Stuff to Know
360 * You should always specify an upload bandwidth limit. With the most
361 recent changes in the program, this "option" is not just a limit
362 to stay under, but an advisement to the client as well. Enhanced
363 CTorrent now tunes its upload performance based on the limit.
364 Without a limit, the client has no idea how much bandwidth your
365 line can support and so cannot perform this tuning. It is now
366 possible to achieve better upload rates with a limit than without.
367 Due to the tit-for-tat nature of bittorrent, this can also
368 indirectly increase your download performance. If you just want
369 the client to use as much upload bandwidth as possible, then
370 choose a limit that is 10% or so less than the available upload
371 capacity of your line. ("Available" means not typically in use by
372 other applications.) Note that limits are specified in KB/s
373 (kilobytes per second), where 1KB = 1024 bytes (8192 bits). Your
374 ISP likely measures in "kilobits" (Kb, where 1Kb = 1000 bits or
375 125 bytes) or "megabits" (1Mb = 1000000 bits or 122KB). Some of
376 that [to the tune of 20% in some cases] is used by the line
377 protocol and thus not available to you at all.
378 * The CTCS protocol version has been incremented. If you've written
379 an application that interfaces with CTorrent via the CTCS
380 protocol, it should continue to work if you've used protocol level
381 negotiation (PROTOCOL message). The new changes have been
382 documented on the CTCS Protocol page.
383
384 Features and Performance
385 * Fast Restart
386 + "Background" initial hash verification: Downloading and
387 uploading will begin while pieces are being verified. Status
388 will be updated as pieces are checked, and HAVE messages are
389 sent as valid pieces are found.
390 + A bitfield save file is always used. At startup, if a file of
391 the same name as the torrent file with ".bf" appended is
392 present, it is read to initialize the piece bitfield. The
393 file is then deleted, and rewritten upon exit if downloading
394 has not completed.
395 + Pieces that are indicated as already present are verified by
396 hash check (in the background). If the bitfield file is not
397 found then all pieces are checked, unless this is the first
398 time the torrent has been started.
399 + The -b option can be used to specify a different bitfield
400 filename. Difference: The option is generally not needed
401 since it is now "on" by default. It now does not prevent hash
402 checking when used.
403 + The -f option now forces accuracy of the initial bitfield.
404 Stated differently, it disables the initial hash checking of
405 pieces on startup. This option should [still] be used with
406 caution and is generally not recommended since background
407 hash checking is now used. If the bitfield file is found at
408 startup, it is used without verification; otherwise, seed
409 mode is forced (without hash verification). Difference: The
410 effect is generalized to apply to incomplete torrents (when a
411 bitfield file is found) as well as the original behavior of
412 forcing seed mode (without a bitfield file).
413 + To emulate the original initial hash check behavior (avoiding
414 background checking), use the -c and -f options together.
415 * Cache Management
416 + Automatic cache size: The -C option now specifies the maximum
417 cache size, with the default remaining 16MB. The actual size
418 used is determined by the download and upload rates and
419 timings.
420 + The maximum cache size that can be specified is no longer
421 limited. The maximum size that will be used in any case is
422 the size of the torrent.
423 + Slices to be uploaded are prefetched into the cache during
424 idle time. This can shave a few milliseconds off the time to
425 respond to an upload request, and can also help performance
426 during periods of heavy disk activity. The effectiveness may
427 be reduced slightly when heavy upload limiting is used (as
428 uploading patterns become less predictable), but it is also
429 less critical then.
430 + A more efficient cache aging mechanism has been added. This
431 reduces CPU utilization and helps increase the accuracy of
432 cache size management.
433 + The cache is now indexed by piece to speed up searches and
434 insertion.
435 + Downloaded pieces are flushed to disk upon completion of each
436 piece.
437 * Unchoke Tuning
438 + The number of unchoked peers will be increased automatically
439 when the upload bandwidth limit is not being reached and
440 decreased when there is much contention for upload bandwidth.
441 This is done to maximize upload bandwidth utilization while
442 also maximizing the upload rate to each unchoked peer.
443 + Unchoke intervals are extended when necessary in order to
444 insure that each unchoked peer has at least a chance of an
445 opportunity to download.
446 + You must specify an upload bandwidth limit in order to use
447 these features. Without a limit, the client does not know the
448 line capacity or available bandwidth. In that case TCP
449 congestion control (the network protocol) limits the
450 bandwidth, so the number of active upload streams (unchokes)
451 is kept low in order to help it work properly when the line
452 reaches its capacity.
453 * Console I/O Management
454 This is really a mostly-transparent enabler for other new current
455 and future features.
456 + Several configuration commands are available while running.
457 Note that this requires termios, termio, or sgtty support to
458 work properly; otherwise try pressing Enter after the key. It
459 should be very rare for a system to not have one of these
460 facilities.
461 o Press "h" or "?" for a list of active keys. Inactive
462 keys will immediately update the status line.
463 o Some commands prompt for additional input. The client
464 continues to run while waiting, though the status line
465 is not updated. An exception is the CTCS password; just
466 try to be relatively quick about it.
467 o Commands shown with "+/-" use the "+" and "-" keys to
468 increment or decrement the value. After pressing the
469 command key, press "+" and/or "-" repeatedly to adjust
470 the value. After five presses, the increment is
471 increased; press the command key again if you wish to
472 reset it.
473 + An Operator Menu allows changing output destinations,
474 choosing an alternate status line format, and viewing
475 detailed status information.
476 + Presentation of messages between status line updates is
477 cleaner.
478 + A timestamp (raw clock value) is printed on "verbose" output.
479 * Options
480 + Added User-Agent header to the tracker request. A new option
481 (-A) can be used to change the string that is sent (default
482 is shown on the help screen). This feature should help with
483 trackers that try to restrict use to recognized client
484 programs. You should only need to use the -A option if the
485 tracker has been set up to allow only specific clients
486 (almost certainly out of admin ignorance in one way or
487 another).
488 + The -p option now specifies the highest port number to use.
489 The client will count down from that point looking for an
490 available port. This mimics the default behavior of counting
491 down from port 2706.
492 + A new command line option ("-a") can be used to preallocate
493 the files to download. Use this if you are concerned about
494 file fragmentation or out-of-order block storage. This option
495 is only effective when initially creating the files and will
496 cause startup to take longer as each entire file is written
497 in order to reserve physical disk space. Note that all files
498 will be created and preallocated even if the "-n" option is
499 used to download a particular file.
500 + Added some option-specific error messages to be displayed
501 instead of the help/usage screen when invalid values are
502 given on the command line.
503 * Peer Handling
504 + Peer statistics are maintained if a peer is dropped and later
505 reconnects. This can influence unchoking, and the data is
506 visible with CTCS.
507 + The client is more discerning about choosing to reconnect to
508 a lost peer.
509 + Peer download rates (the rate at which we download from each
510 peer) are now measured more precisely. This should produce
511 more accurate unchoking decisions while downloading.
512 + Inactive peers (who are interested but don't request any data
513 while unchoked) are disconnected when seeding.
514 + A newly-interested peer will be unchoked immediately if an
515 opening is available.
516 + Latency measurement is used to avoid counting errors that are
517 likely due to transmission delays. To help with this, latency
518 is now also measured when not downloading from a peer. The
519 error threshold (tolerance) has also been reduced.
520 + Don't count piece hash failures during Initial & Endgame
521 modes, since it is likely that the component slices of a
522 piece came from multiple peers.
523 * Other performance improvements
524 + Bandwidth limiting is now based on current (most recent)
525 utilization rather than the past 20-second average. This
526 should produce smoother loading of the line since we don't
527 burst above the limit to make up for a recent slow period.
528 Note that average values are still shown in the status line
529 (and CTCS) since that is a better representation of the
530 overall transfer rates.
531 + Data transmission and receiving buffers are now much smaller
532 but are still increased as necessary (and now decreased when
533 appropriate as well).
534 + Some allowance has been made for time to appear to move
535 backwards, which could happen due to a system glitch,
536 hardware problem, or clock adjustment. This has not been
537 extensively tested (and probably won't be) but hadn't been
538 reported as a problem either.
539 + Timings related to bandwidth management and data transfer
540 have been significantly reworked to allow more precise and
541 accurate control.
542 + Unneeded and bad data is no longer cached and does not cause
543 additional piece completion checks.
544 + Added additional fdset tracking to reduce processing in the
545 checking and handling loops.
546 + Other code optimizations not specifically listed here.
547 * The Pause functionality is now more useful; it will now stop
548 transferring torrent data almost immediately but still interact
549 with peers and the tracker.
550
551 Bug Fixes
552 * Fixed a potential crash when reassigning a piece to a faster peer.
553 While a serious bug, it would likely only show itself on
554 memory-contrained systems.
555 * Fixed reporting of file completion when using -n, and also with
556 CTCS.
557 * Ensure that the "completed" event is sent to the tracker during
558 the next attempt if contacting the tracker is unsuccessful.
559 * The amount of data transferred since the last update, shown in the
560 status line, would go berzerk upon becoming a seeder. Some rate
561 handling code has been cleaned up and tweaked to fix this.
562 * A piece could be incorrectly reported as complete if an error
563 occurred reading the piece data for hash verification.
564
565 Code Cleanup
566 * Moved version number into a new m4 file to make patch distribution
567 easier.
568 * Changed method of locating and using SSL library and header files
569 for SHA1 support. See the INSTALLING file for information about
570 how to override this or give a hint to configure about where to
571 find the files.
572 * Added autoconf checks for integer types that may not be defined on
573 some systems.
574 * The funny business with cfg_max_slice_size is eliminated since
575 request slice size is based on user option rather than a fixed
576 request queue size.
577 * Moved bandwidth limiting functions from peer.* to peerlist.*
578 * Relocated and rewrote get_nl and set_nl.
579 * Replaced hard-coded message component length values with
580 constants.
581 * Formatting adjustments.
582 _________________________________________________________________
583
584 Changes for "dnh2.2" Release
585
586 Code Fixes
587 * Fixed checking of seed ratio (-E option) to allow fractional
588 values.
589 * Fixed CTCS interaction when a limit was not specified on the
590 command line.
591 * Fixed peer reconnect determination.
592 * Fixed a request-queue handling error that could cause a crash upon
593 becoming a seeder.
594 * Fixed creation of metainfo (torrent) file with multiple
595 subdirectories.
596 * Torrent metainfo is now only printed twice if hash-checking is
597 performed.
598
599 Operational Enhancements
600 * Errors flushing the cache are now handled more gracefully. (This
601 happens when the disk is full.)
602 * Added additional seeder detection criteria.
603 * Cleaned up parts of the help screen.
604 _________________________________________________________________
605
606 Changes for "dnh2.1" Release
607
608 Code Fixes
609 * Fixed interaction with CTCS under Linux, resulting in "Operation
610 now in progress" errors.
611 * Fixed a flaw in optimistic unchoke logic implementation.
612 * Fixed problem with return value in _btf_ftruncate() related to my
613 update for the vfat issue [btfiles patch].
614 * Fixed handling of 301/302 redirect response from the tracker.
615
616 Operational Enhancements
617 * Attempt to reconnect to successful peers that disconnect
618 unexpectedly, if possible.
619 * Changed the unchoke choice algorithm when seeding. The new method
620 is based on page 4 of the paper "Rarest First and Choke Algorithms
621 Are Enough" by Arnaud Legout et al. An exception is that CTorrent
622 chooses new unchokes using its original method (based on wait
623 time) rather than randomly.
624 See [1]the paper for details.
625 _________________________________________________________________
626
627 Changes for "dnh2" Release
628
629 Patches
630 * The following patches or their functionality are incorporated:
631 1380164 [dnh1.2]
632 1357832 [invert] (included in dnh1.2)
633 1352866 [dnh1.1]
634 1266767 [passkey2]
635 1239547 [dnh1]
636 1170457 [standalone-sha1] Added as a fallback case in configure if
637 OpenSSL is not found. To force it to be used, define
638 USE_STANDALONE_SHA1 in config.h (after running configure).
639 1164454 [ip]
640 1119610 [vfat] This bug appears to be linux-specific; I've tried
641 to handle it in a more general way that may apply to similar
642 situations on other platforms and filesystems, but I have limited
643 capability to test this.
644 1067196 [lfs] This is the large-file support that many have asked
645 for.
646
647 Optimization
648 * Use fewer calls to random() by shifting the previously unused
649 bits.
650 * Time() calls have been greatly reduced; a global timestamp
651 variable "now" is set once per main loop iteration and referenced
652 in functions that need a timestamp (except the caching I/O
653 routines which were left alone).
654 * Overall current bandwidth rates are now computed only once per
655 main loop and referenced in any routines that evaluate or control
656 bandwidth.
657 * Avoid flushing peer output buffers except in SendModule. This
658 allows for some consolidation of messages to reduce network
659 overhead.
660
661 Code Fixes
662 * Fixed use of cfg_req_queue_length to be the actual queue size
663 (queue was half of this value).
664 * Fixed: "peer is" verbose output could fubar the terminal.
665 * Formatting: Replaced indentation tabs with spaces for consistency.
666
667 Operational Enhancements
668 * Improved piece selection methods to include rarity as a factor.
669 This is not strictly "rarest-first", as we do not make a
670 comprehensive effort to find the "rarest" piece or rank pieces by
671 rarity. Rather, we use a more efficient compromise and try to find
672 the set of pieces that have "trade value" (another peer needs
673 them) and make a random choice from that set. Here is the current
674 preference order used in each mode:
675 Trade Value is defined as:
676 1. Piece that only this peer has (not considering other
677 seeders), that a peer in which we're interested needs.
678 2. Piece that not every peer in which we're interested has.
679 3. Piece that only this peer has (not considering other
680 seeders).
681 4. Piece that not every peer has.
682 Normal Mode
683 1. Piece we tried to get from another peer but stopped due to
684 choking or lost connection. (We have part of the piece
685 already.)
686 2. Piece most recently acquired by the peer (possibly/probably
687 rare).
688 3. Piece with trade value.
689 4. Any piece not yet requested.
690 Initial-piece Mode
691 1. Piece with trade value which is already in progress.
692 2. Piece with trade value that more than one peer has.
693 3. Piece with trade value.
694 4. Any piece not yet requested.
695 Endgame Mode
696 1. Piece with trade value which is already in progress, of which
697 we have the least amount.
698 2. Piece already in progress of which we have the least amount.
699 * Advanced request queueing system.
700 + Instead of requesting all of the slices for a piece at one
701 time, we now measure latency to the peer and send requests
702 based on how long it takes the data to arrive. This avoids
703 wasting upload banwidth by having too many outstanding
704 requests: If we get choked or lose the connection, the extra
705 requests were wasted; in initial or endgame modes, more
706 requests would have to be cancelled when we completed the
707 piece.
708 + A new piece will be queued for download when there is space
709 in the queue and we've requested the slices that have been
710 queued already. We also don't wait for the current piece to
711 complete before sending requests for a new piece. This helps
712 to maintain a continuous flow of data in the download
713 pipeline.
714 + When duplicating a request in initial or endgame mode, slices
715 that have already been requested are queued last.
716 * Don't send HAVE messages to seeders (to save UL bandwidth).
717 + Since we maintain interested state, and know the peer is a
718 seeder, we'll do the right thing when we become a seeder.
719 + Not sending HAVE to all peers (leechers) that already have
720 the piece is a bad idea IMO. If everyone takes the same
721 attitude, none of us will know when another becomes a seeder
722 and connections will remain open/occupied.
723 + We do send a HAVE to seeders upon completing our first piece
724 so that we don't continue to appear empty.
725 * Endgame strategy is used in get1file mode to complete the file.
726 * Queue management:
727 + Don't accept requests from choked peers.
728 + Discard peer's reponse_q when we choke them.
729 + Don't send cancels when we get choked (according to spec &
730 discussions).
731 + Don't put full piece queues in pending.
732 + Move closing peer's request queue to pending instead of
733 discarding it.
734 * Prefer uploading to or downloading from a peer after we skip them
735 due to bandwidth limiting. This is done via the g_next_up and
736 g_defer_up global variables in peer.cpp (for UL; s/up/dn for the
737 DL versions). The peerlist Sort() function and peer "click"
738 variables have been removed since they are not needed with this
739 feature.
740
741 Options & Features
742 * The -c option now reports file completion status.
743 + As a side effect the metainfo details are printed twice. This
744 allows you to view the torrent contents while pieces are
745 being verified.
746 + Total percentage completion is also added to the output.
747 * "-E" option to seed to a specified UL:DL ratio. Seeding will stop
748 when this ratio or the timeout (-e) is reached. If CTorrent starts
749 as a seeder, the ratio is interpreted as UL:[torrent size].
750 * If "-e 0" is specified (explicitly) and -E is used, there will be
751 no timeout; seeding will continue until the ratio is reached.
752 * The "-m" option previously didn't do anything, and it isn't clear
753 what it was originally going to do. Now the default value is 1,
754 and CTorrent will try to maintain at least this many peers by
755 contacting the tracker early if the peer count falls below this
756 value. This feature was present in release dnh1 but the value was
757 not changeable.
758 Actually it seems likely that this was to be number of peers that
759 the client would try to obtain (by initiating connections), as the
760 "official" client does; this is mentioned as a note in the online
761 specification. I don't really see the value in that though. That
762 said, the option as implemented here should rarely be used. It
763 might be useful only with torrents that have significantly more
764 than max_peers total peers and use a long tracker update interval
765 (such that you tend to drop a lot of peers betwen updates).
766 * "-z" option to set the slice size (the unit of a request, i.e. the
767 discrete amount of data that will be requested from a peer at one
768 time). The slice size now defaults to 16K regardless of the piece
769 length. Request queue size is computed and set based on the slice
770 size, as it now affects only system resources (though not a lot)
771 and not the way that requests are sent.
772 * Add support for "key" and "trackerid" tracker interaction
773 parameters.
774 * Support/display tracker warning message
775 * Now able to handle torrents with more than 255 files.
776 * Support for [2]CTorrent Control Server, an application and
777 protocol for monitoring and managing multiple Enhanced CTorrent
778 clients. The "-S" option is used to connect to CTCS, as in "-S
779 localhost:2780" if CTCS is listening at port 2780 on the local
780 system. Appending a colon ("-S localhost:2780:") will prompt for a
781 password to authenticate with CTCS.
782
783 Peer Handling
784 * Count immediate choke-unchoke (either order) as an error (two
785 errors actually, since it's so wasteful).
786 It may be that some clients do this to stimulate the peer when
787 they think it hasn't responded to their last unchoke (due to high
788 latency). It would be better for them to just repeat the unchoke
789 rather than choke-unchoke, as by choking they will cause the peer
790 to send the requests again.
791 * Detect unresponsive peer connections and try to fix them or
792 disconnect them. Basically, if a peer doesn't respond to our
793 request in a reasonable time then we first assume that our request
794 was lost in transmission; if it happens again then we assume the
795 connection is unreliable.
796 * Handle peers that suppress HAVE messages so we don't always think
797 that they're empty (and thus give them preferential treatment for
798 uploading). If we've sent the peer an amount of data equivalent to
799 two pieces, assume that they now have at least one complete piece.
800 _________________________________________________________________
801
802 Changes for "dnh1.2" Release
803
804 These are just corrections to the previous release that I felt were
805 necessary. Much more improvement is coming in the dnh2 release.
806
807 Bug/code fixes
808 * Bitfield::Invert patch [1357832 on sourceforge] described below.
809 * Fixed "piece length too long" check to reflect the actual queue
810 length used.
811 * Accept 128K slice size for peer requests.
812 * "Return" keyword in Random_init() removed due to potential compile
813 error.
814 * Modified longer-wait test in the optimistic unchoke routine to
815 consider whether the peer is currently choked.
816 _________________________________________________________________
817
818 Bitfield::Invert bug
819
820 There is a bug in the Bitfield::Invert() function that affects the
821 ctorrent-1.3.4 base code as well as releases dnh1 and dnh1.1. This can
822 cause the application to fail (segmentation fault) or may affect
823 downloading of all pieces of the torrent. A patch is available in the
824 Download secion.
825 _________________________________________________________________
826
827 Changes for "dnh1.1" Release
828
829 These are just corrections to the previous release that I felt were
830 necessary. Much more improvement is coming in the next release.
831
832 Bug/code fixes
833 * Peer count would increase on each tracker update if there were no
834 seeders.
835 * RequestQueue::CopyShuffle() changed to use a pointer argument.
836 * Fixed some incorrectness in PendingQueue::Delete() and
837 PendingQueue::DeleteSlice() which could cause a memory leak.
838 * Fixed random-chance inversion bug in PeerList::UnChokeCheck()
839 affecting choice for optimistic unchoking.
840
841 Improvements
842 * Move StopDLTimer() call from RequestPiece() to RequestCheck(),
843 which could occasionally affect peer download rate measurement.
844 * Most clients do not like a slice size of 128K even though it is
845 the max allowed by the BT specification. Changed max slice size to
846 64K. Note that the maximum piece length is 2MB (2097152); if you
847 need to download a torrent with a larger piece size you can change
848 the value of cfg_req_queue_length in btconfig.h from 64 to 128.
849 * Contact tracker immediately upon becoming (or starting as) a
850 seeder.
851 * Changed SendModule() to send only one slice at a time. This will
852 help with fairly distributing upload bandwidth among the unchoked
853 peers.
854 * Changed default peer ID prefix to '-CD0101-', indicating
855 CTorrent-dnh1.1 release.
856 _________________________________________________________________
857
858 Changes for "dnh1" Release
859
860 This is the first release. "dnh" identifies this patchset, and "1"
861 indicates release version 1 of the patchset.
862
863 Patches
864 * Incorporates the following patches. The number is the Request ID
865 from the [3]SourceForge patches page, which you can reference for
866 the details of each patch. The name in brackets is the name of the
867 patch file or a name I chose to refer to the patch. Some of these
868 names are used below (in brackets) to describe a fix or change to
869 a particular patch.
870 1042808 [getcwd] (incorporated in get1file patch)
871 1084776 [passkey] (incorporated in udlimit patch)
872 1109266 [align]
873 1109287 [tracker/tracker2]
874 1114197 [fmt] (incorporated in get1file patch)
875 1114364 [resetdl]
876 1116448 [get1file]
877 1118597 [crash]
878 1119467 [stall]
879 1119492 [rate]
880 1119497 [flush]
881 1119519 [opt]
882 1119689 [status]
883 1124342 [udlimit]
884
885 Download performance
886 * If a peer socket is ready for reading and writing, perform both.
887 Previously the cases were exclusive, with preference given to
888 reading.
889 * Download requests are now made to peers when they are ready for
890 writing (in addition to the existing event-driven cases). This
891 fixes peer stalls when a request couldn't be sent in an
892 event-triggered case due to bandwidth limiting or other
893 circumstances.
894 * Additional tests added so that the above request checking doesn't
895 create hard loops.
896
897 Bandwidth measurement/management
898 * [rate] Bandwidth reporting is now not capped. Also, only the time
899 used for the samples taken is used in the calculation rather than
900 the maximum interval (this affects rate calculation for individual
901 peers).
902 * Additional upload and download bandwidth limit checks added so
903 that bw management is more accurate.
904 * Corrected condition inversion bug in Rate::StopTimer(), which
905 affects peer rate calculations.
906
907 Peer count
908 * Request our max number of peers from the tracker each time rather
909 than just taking the tracker's default, so we can try to fill up.
910 * The tracker will be contacted early if all peers disconnect so
911 that we can actively try to establish some more peer connections.
912 To avoid hammering the tracker, we must have at least one peer for
913 15 seconds in order for this to be invoked.
914 * Some clients use nonzero bytes in the "reserved" part of the
915 handshake. Added code to ignore the reserved bytes if the rest of
916 the handshake is as expected. This includes Azureus 2300 thru 2304
917 (latest) which gives 0x80 as the first reserved byte and BitComet
918 which gives 0x6578 ("ex") as the first two bytes.
919 * Update peer's timestamp on any message, not just keepalives. Any
920 receipt of data from a peer now resets its timeout, preventing
921 early disconnect.
922
923 Parallel requests
924 * Initial-piece and endgame cases have been improved so that pieces
925 will be requested from multiple peers. Cancels are sent as slices
926 (subpieces) are received. This endgame strategy is described in
927 the BitTorrent online spec. The startup strategy is also described
928 in posts and facilitates obtaining a single piece more rapidly in
929 order to obtain some trade value. In initial mode, the piece of
930 which we need the least parts is targeted. In endgame mode, the
931 piece of which we need the most parts is targeted. Slices that are
932 cancelled are also removed from the "pending" queue, as are pieces
933 that are completed.
934 * When duplicating a piece request, the slice request order is also
935 shuffled in order to minimize duplication of effort.
936
937 Tracker info
938 * [status] Seems to be missing tracker.cpp diff. Added code to get
939 tracker's total peers, but not all trackers report these fields in
940 the normal response. Also added code to count successful updates
941 from the tracker.
942 * Added tracker connection state to status line (when
943 connecting/connected).
944
945 Tracker contact
946 * When interrupting (ctrl-C), connect to the tracker immediately
947 rather than waiting 15 seconds.
948 * Contact tracker "soon" after transitioning to seeder state.
949
950 Peer interaction
951 * Manage our interested state for each peer dynamically as content
952 changes.
953 * Unchoking
954 + Use the peer's interested state (confirming via the bitfield
955 that it needs our data) to consider unchoking. Using only the
956 bitfield could cause us to waste an upload slot on a peer
957 that doesn't have all content but isn't interested (like a
958 single-file downloader).
959 + Choke peers that become uninterested or don't need our data.
960 + Try unchoking new peers only as slots open or the optimistic
961 unchoke rotates. Unchoking too many peers can temporarily
962 reduce per-peer upload rates, which would make uploading to
963 us unappealing for good peers.
964 + In a tie for download speed, prefer to unchoke the peer to
965 whom we've uploaded the least data relative to what we've
966 downloaded from him.
967 * Optimistic unchoking
968 + Fixed condition inversion bug causing opt unchoking to occur
969 too often.
970 + A peer who has no pieces is now preferred 75% vs. a peer who
971 already has at least one piece, in order to help the new peer
972 become productive. This is documented in the online spec.
973 + Set peer's last-unchoke-time when choking the peer to get
974 better rotation of the optimistic unchoke. The value is now
975 the last time that a peer was in the unchoked state rather
976 than the time of the last unchoke event.
977
978 Miscellaneous items
979 * [tracker] Fixed normal program end (stop) process, which was
980 crashing.
981 * [get1file] Restore compact tracker response support.
982 * [get1file] Made display of the status line info dependent on
983 whether the option is in use.
984 * [get1file] Apply the filter when checking for what we need from a
985 peer instead of when recording what the peer has; this prevents
986 stalls when we move on to the next file. Update interested state
987 for each peer when we begin a new file. Also move the file-done
988 check into the piece completion code and check whether the next
989 file(s) has also been completed.
990 * Reduced the slice size from 32K to 16K (same as BT & Azureus).
991 This provides more precise DL rate measurement, and helps insure
992 that we receive a productive amount of data (i.e. a complete
993 slice) even if we are unchoked by a peer for only one cycle. See
994 [4]http://groups.yahoo.com/group/BitTorrent/message/1260 for more
995 discussion/analysis on this.
996 * Added -v (verbose) option for additional debugging output.
997
998 References
999
1000 1. http://www.eurecom.fr/~michiard/pubs/bt_experiments_techRepINRIA-00001111_VERSION1_13FEBRUARY2006.pdf
1001 2. http://www.rahul.net/dholmes/ctorrent/ctcs.html
1002 3. http://sourceforge.net/tracker/?atid=598034&group_id=91688&func=browse
1003 4. http://groups.yahoo.com/group/BitTorrent/message/1260