BitTorrent Protocol Specifications (WIP [2004-04-25)

Ok, it's been a while since the last time I looked at this page, time to do a small refresh.

The original version of this page was written while MoonlightTorrent(.com) was still in the early stages of becoming a fully capable BT client. At about the same time, I was also writing my MoonlightTorrent Protocol - Preliminary Specifications (rewrite pending [2004-04-17]) draft which I have also not touched in ages and needs a rewrite to reflect what I have experienced over the last four months. The main reason why I started rewriting the BT protocol's specifications back then was because the official specifications lacked some details that eluded me some time. Most of it is deceptively simple, the issues are in the details of the protocol's 'unwritten rules'.

Below is my interpretation of the BT protocol specifications along with the few unwritten rules that I either discovered through experimentation or chatting with some helpful folks on #btports on FreeNode, you guys probably know who you are :)

Anyhow, this page only contains the stuff I could remember 'off-the-top' at the time of writing and will be added as I discover/remember/learn more about it.

Connection handshake:

1. The connecting party sends a 68 bytes string upon connect consisting of the following:
   • CHAR[20] Protocol name string ("\x13"+"BitTorrent Protocol", Pascal-style length-prefixed string, not C-style null-terminated.)
   • CHAR[8] Option field, reserved for future protocol extensions if someone ever convince Bram to add any.
   • CHAR[20] Client ID.
   • CHAR[20] Torrent file 'info' key hash, this is the hash of the torrent file's data from the starting 'd' of the info dictionary to its closing 'e' inclusively:
     • if the .torrent file contained "d4:infod4:name9:somethingee"
     • then the data to hash would be "d4:name9:somethinge"
     • and this set would mean that key "name" has a value of "something"
2. The answering party looks up the CliendID/InfoHash
   • For single-torrent clients, if the Info hash does not match the torrent being downloaded, the connection is dropped.
   • For multi-torrent clients, the connection is dropped if none of the torrents match the received hash.
   • If the PeerID is known (usually listed as "Local"), then an IP address check is done to see if the new connection is for the same client. If the IPs differ, the connection can be either dropped or a new record created to accomodate the client as a new entry. (usually listed as "Remote")
   • If the PeerID is unknown, a new record is created (usually listed as "Remote") for the new peer.
   • After all the connection accepting 'paperwork' is done, if the peer's connection has been accepted, the answering peer sends its 68 bytes handshake for the torrent in question.
3. The connecting party verifies that the received info hash and Client ID have the expected values and drops the connection if such is not the case.
4. Either, both or neither ends then proceed to sending the initial BitField (see below) before begining normal operation.

Basic BitTorrent message format: (all DWORDs are in network byte order)

• DWORD Message length, length of everything else in the message.
• CHAR Message type ID, BitTorrent message type.
• CHAR[] Message data, size = Message length - 1

Official message types:

• General note: All DWORDs are network-byte-order, "big-endian".
• 0x00: Choke
  • Purpose: Tells the peer not to send any more piece data requests.
  • Actions:
    • Sender drops all chunk requests made by the peer.
    • Recipient considers all outstanding chunk requests to the peer as cancelled and stops requesting further chunks.
• 0x01: UnChoke
  • Purpose: Tells the peer that it can start sending piece data requests.
  • Actions:
    • Sender waits for chunk requests from the peer.
    • Recipient start sending piece requests if the peer has useful pieces.
  • Notes: Some clients will send a Choke if too many requests are sent at once or too many are queued at one time. This limit appears to typically be around 16 and most clients queue up 4 to 16 requests, Azureus being the most common 16er.
• 0x02: Interested
  • Purpose: Tells the peer that it has piece data your client needs.
  • Action:
    • Sender waits for Unchoke.
    • Recipient sends Unchoke if it has spare upload bandwidth none of its other current upload slots can use
• 0x03: Not Interested
  • Purpose: Tells the peer that your client is no longer interested in any data the peer has.
  • Action:
    • Sender clears pending requests to that peer.
    • Recipient clears queued requests from the peer.
• 0x04: Have
  • Purpose: Notifies peers about an acquired and verified new piece that is now available for requesting.
  • Action:
    • Sender does not need to take any further actions.
    • Recipient sets the specified piece's bit in peer's bitfield and flags the peer as interesting if that piece is needed.
  • Extra payload:
    • DWORD Piece number : zero-based piece number
  • Extra notes:
    • With 100 connected peers and high download speeds, this packet type can potentially generate 5-10KB/s of overhead traffic per torrent. Most clients pack multiple Have message in one transmission to reduce TCP/IP overhead which would otherwise account for 90% of the actual traffic.
    • One proposed amendment to the BT protocol is the MultiHave message which would also remove the redundant message size and message type information. By packing a minimum of 10 Haves per MultiHave, worst-case Have traffic efficiency would go from 5.5% to about 40%.
• 0x05: BitField
  • Purpose: Tells the peer what pieces your client has acquired so far.
  • Action:
    • Sender does not need to take any further actions.
    • Recipient loads the peer's bitfield data, flag the peer as interesting if it has needed pieces.
  • Extra payload:
    • CHAR[] BitField, size is 1/8 the number of pieces rounded up.
  • Encoding: The first byte's MSB corresponds to the first torrent piece.
  • Extra notes:
    • Many clients will drop the connection if they receive more than one BitField from a peer.
    • Some bandwidth could be spared by sending a bitfield whenever the amount of backlogged Have messages' worth in bytes exceeds the BitField message's size. This threshold usually stands at less than 20 Haves when ignoring all TCP/IP overhead.
• 0x06: Request
  • Purpose: Request piece data (chunk) from a peer.
  • Action:
    • Sender waits for chunk data.
    • Recipient incrementally sends the requested data whenever bandwidth is available.
  • Extra payload:
    • DWORD Piece number: zero-based piece number.
    • DWORD Piece offset: zero-based offset within the piece at which the request starts.
    • DWORD Piece size: size of the requested piece data, usually 16KB to limit junked data waste.
  • Extra notes:
    • Requests cannot cross piece boundaries.
    • 'junked data' refers to data received from multiple sources for one particular request, this usually happens when requests to one peer time out and are re-assigned to different peers then both the former and new peers send the data.
    • Most clients will drop requests received from a choked client or drop the connection.
    • Requests may be served out of order.
    • Pipelining requests (requesting multiple chunks in advance) is necessary while doing bidirectionnal exchanges since more chunks cannot be requested while oneself is in the middle of uploading a chunk. The number of requests that need to be pipelined should be slightly more than the upload/download speed ratio for the peer. If one peer uploads 10X as fast as the other, the other needs to queue up about 10 requests.
• 0x07: Piece
  • Purpose: Transfer one of the data chunk requests.
  • Action:
    • Sender removes the request being served from its pool.
    • Recipient matches the data to an active request, writes the data to disk and hashes the whole piece once all its chunks have been gathered.
  • Extra payload:
    • DWORD Piece number: zero-based piece number.
    • DWORD Piece offset: zero-based offset within the piece at which the request starts.
    • CHAR[] Piece data: The requested data, the size is the remainder of the message's size.
  • Extra notes:
    • Official, official-based and most other clients expect the Piece message to match an outstanding piece request exactly.
• 0x08: Cancel
  • Purpose: Cancel an outstanding requests to a peer.
  • Action:
    • Sender removes the request from its request list to that peer.
    • Recipient drops the specified request unless it was being served prior to receiving the Cancel.
  • Extra payload:
    • DWORD Piece number: zero-based piece number.
    • DWORD Piece offset: zero-based offset within the piece at which the request starts.
    • DWORD Piece size: size of the requested piece data, usually 16KB to limit junked data waste.
  • Extra notes:
    • Mostly used while finishing downloads.
    • Also used to cancel outstanding requests to a peer when they time out or when faster peers are identified and the requests transferred to them to reduce the amount of junked data that may be received.

Hits since December 5, 2003:

---