Reference Matrix Home Server

1 files changed, 177 insertions, 0 deletions

diff --git a/docs/server-server/specification b/docs/server-server/specification
new file mode 100644
index 0000000000..e1e49cc069
--- /dev/null
+++ b/docs/server-server/specification
@@ -0,0 +1,177 @@
+============================
+Synapse Server-to-Server API
+============================
+
+A description of the protocol used to communicate between Synapse home servers;
+also known as Federation.
+
+
+Overview
+========
+
+The server-server API is a mechanism by which two home servers can exchange
+Synapse event messages, both as a real-time push of current events, and as a
+historic fetching mechanism to synchronise past history for clients to view. It
+uses HTTP connections between each pair of servers involved as the underlying
+transport. Messages are exchanged between servers in real-time by active pushing
+from each server's HTTP client into the server of the other. Queries to fetch
+historic data for the purpose of back-filling scrollback buffers and the like
+can also be performed.
+
+
+ { Synapse entities }                            { Synapse entities }
+     ^          |                                    ^          |
+     |  events  |                                    |  events  |
+     |          V                                    |          V
+ +------------------+                            +------------------+
+ |                  |---------( HTTP )---------->|                  |
+ |   Home Server    |                            |   Home Server    |
+ |                  |<--------( HTTP )-----------|                  |
+ +------------------+                            +------------------+
+
+
+Transactions and PDUs
+=====================
+
+The communication between home servers is performed by a bidirectional exchange
+of messages. These messages are called Transactions, and are encoded as JSON
+objects with a dict as the top-level element, passed over HTTP. A Transaction is
+meaningful only to the pair of home servers that exchanged it; they are not
+globally-meaningful.
+
+Each transaction has an opaque ID and timestamp (UNIX epoch time in miliseconds)
+generated by its origin server, an origin and destination server name, a list of
+"previous IDs", and a list of PDUs - the actual message payload that the
+Transaction carries.
+
+ {"transaction_id":"916d630ea616342b42e98a3be0b74113",
+  "ts":1404835423000,
+  "origin":"red",
+  "destination":"blue",
+  "prev_ids":["e1da392e61898be4d2009b9fecce5325"],
+  "pdus":[...]}
+
+The "previous IDs" field will contain a list of previous transaction IDs that
+the origin server has sent to this destination. Its purpose is to act as a
+sequence checking mechanism - the destination server can check whether it has
+successfully received that Transaction, or ask for a retransmission if not.
+
+The "pdus" field of a transaction is a list, containing zero or more PDUs.[*]
+Each PDU is itself a dict containing a number of keys, the exact details of
+which will vary depending on the type of PDU.
+
+(* Normally the PDU list will be non-empty, but the server should cope with
+receiving an "empty" transaction, as this is useful for informing peers of other
+transaction IDs they should be aware of. This effectively acts as a push
+mechanism to encourage peers to continue to replicate content.)
+
+All PDUs have an ID, a context, a declaration of their type, a list of other PDU
+IDs that have been seen recently on that context (regardless of which origin
+sent them), and a nested content field containing the actual event content.
+
+[[TODO(paul): Update this structure so that 'pdu_id' is a two-element
+[origin,ref] pair like the prev_pdus are]]
+
+ {"pdu_id":"a4ecee13e2accdadf56c1025af232176",
+  "context":"#example.green",
+  "origin":"green",
+  "ts":1404838188000,
+  "pdu_type":"m.text",
+  "prev_pdus":[["blue","99d16afbc857975916f1d73e49e52b65"]],
+  "content":...
+  "is_state":false}
+
+In contrast to the transaction layer, it is important to note that the prev_pdus
+field of a PDU refers to PDUs that any origin server has sent, rather than
+previous IDs that this origin has sent. This list may refer to other PDUs sent
+by the same origin as the current one, or other origins.
+
+Because of the distributed nature of participants in a Synapse conversation, it
+is impossible to establish a globally-consistent total ordering on the events.
+However, by annotating each outbound PDU at its origin with IDs of other PDUs it
+has received, a partial ordering can be constructed allowing causallity
+relationships to be preserved. A client can then display these messages to the
+end-user in some order consistent with their content and ensure that no message
+that is semantically in reply of an earlier one is ever displayed before it.
+
+PDUs fall into two main categories: those that deliver Events, and those that
+synchronise State. For PDUs that relate to State synchronisation, additional
+keys exist to support this:
+
+ {...,
+  "is_state":true,
+  "state_key":TODO
+  "power_level":TODO
+  "prev_state_id":TODO
+  "prev_state_origin":TODO}
+
+[[TODO(paul): At this point we should probably have a long description of how
+State management works, with descriptions of clobbering rules, power levels, etc
+etc... But some of that detail is rather up-in-the-air, on the whiteboard, and
+so on. This part needs refining. And writing in its own document as the details
+relate to the server/system as a whole, not specifically to server-server
+federation.]]
+
+
+Protocol URLs
+=============
+
+For active pushing of messages representing live activity "as it happens":
+
+  PUT /send/:transaction_id/
+    Body: JSON encoding of a single Transaction
+
+    Response: [[TODO(paul): I don't actually understand what
+    ReplicationLayer.on_transaction() is doing here, so I'm not sure what the
+    response ought to be]]
+
+  The transaction_id path argument will override any ID given in the JSON body.
+  The destination name will be set to that of the receiving server itself. Each
+  embedded PDU in the transaction body will be processed.
+
+
+To fetch a particular PDU:
+
+  GET /pdu/:origin/:pdu_id/
+
+    Response: JSON encoding of a single Transaction containing one PDU
+
+  Retrieves a given PDU from the server. The response will contain a single new
+  Transaction, inside which will be the requested PDU.
+  
+
+To fetch all the state of a given context:
+
+  GET /state/:context/
+
+    Response: JSON encoding of a single Transaction containing multiple PDUs
+
+  Retrieves a snapshot of the entire current state of the given context. The
+  response will contain a single Transaction, inside which will be a list of
+  PDUs that encode the state.
+
+
+To paginate events on a given context:
+
+  GET /paginate/:context/
+    Query args: v, limit
+
+    Response: JSON encoding of a single Transaction containing multiple PDUs
+
+  Retrieves a sliding-window history of previous PDUs that occurred on the
+  given context. Starting from the PDU ID(s) given in the "v" argument, the
+  PDUs that preceeded it are retrieved, up to a total number given by the
+  "limit" argument. These are then returned in a new Transaction containing all
+  off the PDUs.
+
+
+To stream events all the events:
+
+  GET /pull/
+    Query args: origin, v
+
+  Response: JSON encoding of a single Transaction consisting of multiple PDUs
+
+  Retrieves all of the transactions later than any version given by the "v"
+  arguments. [[TODO(paul): I'm not sure what the "origin" argument does because
+  I think at some point in the code it's got swapped around.]]