Torrent download machine

Intro

Torrent package consist of set of shell scripts for torrent processing with lightweight libbt torrent library written entirely in C. All shell scripts are compatible with busybox /bin/sh with requirement of some convenience tools like nohup, sed, tr, date Besides command line tools, there exist also http server CGI script written in SH for WWW access to admin.

Features

• batch multiprocessing (eg. 10 torrents simultaneously with nice priority)
• automatic seeding when torrent finishes (configurable)
• configurable ports
• WWW interface @torrent.cgi@ can be public (no password) as there is no delete provided.
• Accelerator keys (ALT-key) for CGI interface. See help for underlined keys.
• command line utilities @torrent_admin, btcheck-target@ with colorized output
• Per torrent and cumulative status
• start/stop/push and other commands avalable in command line and WWW interface
• lightweight - low memory footprint
• tracker scrape info. Suggest best seed torrent from target dir.
• watchdog for auto restart/queue processing with cron
• syslog watchdog events
• mail notify when torrent finishes
• no state database required. Directory tree is database (slow, but failsafe)
• stable and portable. Deloped on Asus WL-500g Deluxe

Workflow

For queue torrent file processing there should exist tree directories:

source

User should copy .torrent files here

work

Used for torrent processing

target

Place where finished torrents are put.

Typical file flow is source -> work -> target For every torrent file from source, directory with the same name is created. Then torrent file is moved into this directory and program btget is run in background and constantly supervised with torrent_watchdog. When btget is run in backgroung it automaticly prealocates directories and files specified in torrent file. In case of crash or restart, btget checks this subdir for bad/good blocks.

Config

/opt/etc/torrent.conf

Minimum config change after first install is setting initial directories which default to

 SOURCE=/tmp/harddisk/torrent/source
 WORK=/tmp/harddisk/torrent/work
 TARGET=/tmp/harddisk/torrent/target

Mail can be sent if MAILTO/FROM is set and proper mail client is installed. Preconfigured mail client is set. Default MAILER is mini_sendmail found in wl500g Oleg firmware, but can be overriden in /opt/etc/torrent.conf.

Programs

Package torrent (1.4-3) is installed on root and has the following files:

/opt/share/www/cgi-bin/torrent.cgi

CGI interface for busybox_http, thttpd, ...

/opt/sbin/torrent_watchdog

this scirpt should be run regulary to ensure queue processing. Install cron package or use script which periodicly runs this script (ef. every 30 minutes)

/opt/etc/init.d/S80busybox_httpd

sample script for running server

/opt/sbin/torrent_admin

command line interface

/opt/etc/torrent.conf

configurable options for all utilities

/opt/bin/btcheck-target

convnience utility for checksuming finished torrents in case o of svere disk crash/fsck problems

Package libbt (1.05-4) is installed on root and has the following files:

/opt/bin/btlist

Listing contents of torrent file

/opt/bin/btget

Lightweight torrent downloading engine (also for seeding)

/opt/bin/btcheck

Checking torrent consistence for good/bad blocks

Torent admin

This is quick explanation of the options:

-u

Updates active torrents status

-q

Quick show active torrents (without update)

-c n

Shows current.log of selected torrent

-s

All active torrent processing should stop/resume imediately

-p n

Push selected torrent to other queue

-l

Lists queued, active, suspended and completed torrents

-r n

Mark torrent for purging

-g

Purge removes all logs from completed torrents and clean removed torrents

-w

Watchdog forces torrent_watchdog processing

-i n

Info shows selected torrent info ((file content and size)

-a

Update scrape info from tracker for downloaded torrents

-b

Suggest torrent with best (leeches/seeds) ratio for seeding

-ab

Equals -a -b

Usage

For copying .torrent files into source dir one could use samba, ftp, wget and other means of fetching. torrent_watchdog is meant to be run periodicly, but it can be issued at any time from command line.

Finised torrents in target can be always re/moved from target. One can use rm -rf torent-subdir or samba services. WWW interface does not consist such functionality due to inadvent removal from third party on public WWW page. Remove and Purge just do cleanup.

The most usefull command is Push which forces move from one queue to other. With Push one can start processing a torrent, stop active torrent or push from/to seeding state.

Do not push too much. Torrent_watchdog has a code for checking maximum number of active torrents. If all ports (BTGET_MAXPORT-BTGET_MINPORT+1) are occupied, then no new torrent will be enqueued, unless some are pushed away from seed queue. There is no expire time on seeding torrents! One must push it to done manually!

Files:

.info

status info

work/.watchdog

status file when watchdog is running

current.log

stdout output of btget run in background

current_error.log

stderr output of btget run in background

Issues:

• slow listing is due to directory traversal and .info file processing
• btget fails block checksum when empry directory is specified in torrent. Manual stop and move from work dir is required. Many other clients have the same issue and it is unclear if this is a bug in client or bad torrent. Most notable is eg. empty AUDIO_TS directory sprecified in .torrent file

Feature requests:

• Add check for empty directory / patch btget
• Really Quick view

Best practice:

• Run two www servers for CGI interface. Public server should reside on some high port. eg 8008. I suggest busybox_httpd for this job as it lacks autodir features. For intranet I suggest thttpd with symlinks and autodir features on standard 80 port.
• Run torrent_watchdog every 30 minutes with cron. If the time is round, watchdog will update active torrents status. Ther is no need to manualy run update for current status.
• There is no bandwith managment available. QOS should be installed on router with bulk priority on torrent protocol (L7 or port range).
  • QUESTION: Any idea how one might force upload/download bandwidth limits without a QOS-enabled router?

oleo July 05, 2006, at 08:57 AM