Reference Matrix Home Server

3 files changed, 377 insertions, 0 deletions

diff --git a/docs/server-server/protocol-format b/docs/server-server/protocol-format
new file mode 100644
index 0000000000..2838253ab7
--- /dev/null
+++ b/docs/server-server/protocol-format
@@ -0,0 +1,59 @@
+
+Transaction
+===========
+
+Required keys:
+
+============ =================== ===============================================
+    Key            Type                         Description
+============ =================== ===============================================
+origin       String              DNS name of homeserver making this transaction.
+ts           Integer             Timestamp in milliseconds on originating 
+                                 homeserver when this transaction started.
+previous_ids List of Strings     List of transactions that were sent immediately
+                                 prior to this transaction.
+pdus         List of Objects     List of updates contained in this transaction.
+============ =================== ===============================================
+
+
+PDU
+===
+
+Required keys:
+
+============ ================== ================================================
+    Key            Type                         Description
+============ ================== ================================================
+context      String             Event context identifier
+origin       String             DNS name of homeserver that created this PDU.
+pdu_id       String             Unique identifier for PDU within the context for
+                                the originating homeserver.
+ts           Integer            Timestamp in milliseconds on originating 
+                                homeserver when this PDU was created.
+pdu_type     String             PDU event type.
+prev_pdus    List of Pairs      The originating homeserver and PDU ids of the
+             of Strings         most recent PDUs the homeserver was aware of for
+                                this context when it made this PDU.
+depth        Integer            The maximum depth of the previous PDUs plus one.
+============ ================== ================================================
+
+Keys for state updates:
+
+================== ============ ================================================
+    Key               Type                      Description
+================== ============ ================================================
+is_state           Boolean      True if this PDU is updating state.
+state_key          String       Optional key identifying the updated state within
+                                the context.
+power_level        Integer      The asserted power level of the user performing
+                                the update.
+min_update         Integer      The required power level needed to replace this
+                                update.
+prev_state_id      String       The homeserver of the update this replaces
+prev_state_origin  String       The PDU id of the update this replaces.
+user               String       The user updating the state.
+================== ============ ================================================
+
+
+
+
diff --git a/docs/server-server/security-threat-model b/docs/server-server/security-threat-model
new file mode 100644
index 0000000000..cf0430e43d
--- /dev/null
+++ b/docs/server-server/security-threat-model
@@ -0,0 +1,141 @@
+Overview
+========
+
+Scope
+-----
+
+This document considers threats specific to the server to server federation 
+synapse protocol.
+
+
+Attacker
+--------
+
+It is assumed that the attacker can see and manipulate all network traffic 
+between any of the servers and may be in control of one or more homeservers 
+participating in the federation protocol.
+
+Threat Model
+============
+
+Denial of Service
+-----------------
+
+The attacker could attempt to prevent delivery of messages to or from the 
+victim in order to:
+
+    * Disrupt service or marketing campaign of a commercial competitor.
+    * Censor a discussion or censor a participant in a discussion.
+    * Perform general vandalism.
+
+Threat: Resource Exhaustion
+~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+An attacker could cause the victims server to exhaust a particular resource 
+(e.g. open TCP connections, CPU, memory, disk storage)
+
+Threat: Unrecoverable Consistency Violations
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+An attacker could send messages which created an unrecoverable "split-brain"
+state in the cluster such that the victim's servers could no longer dervive a
+consistent view of the chatroom state.
+
+Threat: Bad History
+~~~~~~~~~~~~~~~~~~~
+
+An attacker could convince the victim to accept invalid messages which the 
+victim would then include in their view of the chatroom history. Other servers
+in the chatroom would reject the invalid messages and potentially reject the
+victims messages as well since they depended on the invalid messages.
+
+Threat: Block Network Traffic
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+An attacker could try to firewall traffic between the victim's server and some
+or all of the other servers in the chatroom.
+
+Threat: High Volume of Messages
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+An attacker could send large volumes of messages to a chatroom with the victim
+making the chatroom unusable.
+
+Threat: Banning users without necessary authorisation
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+An attacker could attempt to ban a user from a chatroom with the necessary
+authorisation.
+
+Spoofing
+--------
+
+An attacker could try to send a message claiming to be from the victim without 
+the victim having sent the message in order to:
+
+    * Impersonate the victim while performing illict activity.
+    * Obtain privileges of the victim.
+
+Threat: Altering Message Contents
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+An attacker could try to alter the contents of an existing message from the 
+victim.
+
+Threat: Fake Message "origin" Field
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+An attacker could try to send a new message purporting to be from the victim
+with a phony "origin" field.
+
+Spamming
+--------
+
+The attacker could try to send a high volume of solicicted or unsolicted 
+messages to the victim in order to:
+    
+    * Find victims for scams.
+    * Market unwanted products.
+
+Threat: Unsoliticted Messages
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+An attacker could try to send messages to victims who do not wish to receive 
+them.
+
+Threat: Abusive Messages
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+An attacker could send abusive or threatening messages to the victim
+
+Spying
+------
+
+The attacker could try to access message contents or metadata for messages sent
+by the victim or to the victim that were not intended to reach the attacker in
+order to:
+
+    * Gain sensitive personal or commercial information.
+    * Impersonate the victim using credentials contained in the messages.
+      (e.g. password reset messages)
+    * Discover who the victim was talking to and when.
+
+Threat: Disclosure during Transmission
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+An attacker could try to expose the message contents or metadata during 
+transmission between the servers.
+
+Threat: Disclosure to Servers Outside Chatroom
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+An attacker could try to convince servers within a chatroom to send messages to
+a server it controls that was not authorised to be within the chatroom.
+
+Threat: Disclosure to Servers Within Chatroom
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+An attacker could take control of a server within a chatroom to expose message
+contents or metadata for messages in that room.
+
+
diff --git a/docs/server-server/specification b/docs/server-server/specification
new file mode 100644
index 0000000000..e1e49cc069
--- /dev/null
+++ b/docs/server-server/specification
@@ -0,0 +1,177 @@
+============================
+Synapse Server-to-Server API
+============================
+
+A description of the protocol used to communicate between Synapse home servers;
+also known as Federation.
+
+
+Overview
+========
+
+The server-server API is a mechanism by which two home servers can exchange
+Synapse event messages, both as a real-time push of current events, and as a
+historic fetching mechanism to synchronise past history for clients to view. It
+uses HTTP connections between each pair of servers involved as the underlying
+transport. Messages are exchanged between servers in real-time by active pushing
+from each server's HTTP client into the server of the other. Queries to fetch
+historic data for the purpose of back-filling scrollback buffers and the like
+can also be performed.
+
+
+ { Synapse entities }                            { Synapse entities }
+     ^          |                                    ^          |
+     |  events  |                                    |  events  |
+     |          V                                    |          V
+ +------------------+                            +------------------+
+ |                  |---------( HTTP )---------->|                  |
+ |   Home Server    |                            |   Home Server    |
+ |                  |<--------( HTTP )-----------|                  |
+ +------------------+                            +------------------+
+
+
+Transactions and PDUs
+=====================
+
+The communication between home servers is performed by a bidirectional exchange
+of messages. These messages are called Transactions, and are encoded as JSON
+objects with a dict as the top-level element, passed over HTTP. A Transaction is
+meaningful only to the pair of home servers that exchanged it; they are not
+globally-meaningful.
+
+Each transaction has an opaque ID and timestamp (UNIX epoch time in miliseconds)
+generated by its origin server, an origin and destination server name, a list of
+"previous IDs", and a list of PDUs - the actual message payload that the
+Transaction carries.
+
+ {"transaction_id":"916d630ea616342b42e98a3be0b74113",
+  "ts":1404835423000,
+  "origin":"red",
+  "destination":"blue",
+  "prev_ids":["e1da392e61898be4d2009b9fecce5325"],
+  "pdus":[...]}
+
+The "previous IDs" field will contain a list of previous transaction IDs that
+the origin server has sent to this destination. Its purpose is to act as a
+sequence checking mechanism - the destination server can check whether it has
+successfully received that Transaction, or ask for a retransmission if not.
+
+The "pdus" field of a transaction is a list, containing zero or more PDUs.[*]
+Each PDU is itself a dict containing a number of keys, the exact details of
+which will vary depending on the type of PDU.
+
+(* Normally the PDU list will be non-empty, but the server should cope with
+receiving an "empty" transaction, as this is useful for informing peers of other
+transaction IDs they should be aware of. This effectively acts as a push
+mechanism to encourage peers to continue to replicate content.)
+
+All PDUs have an ID, a context, a declaration of their type, a list of other PDU
+IDs that have been seen recently on that context (regardless of which origin
+sent them), and a nested content field containing the actual event content.
+
+[[TODO(paul): Update this structure so that 'pdu_id' is a two-element
+[origin,ref] pair like the prev_pdus are]]
+
+ {"pdu_id":"a4ecee13e2accdadf56c1025af232176",
+  "context":"#example.green",
+  "origin":"green",
+  "ts":1404838188000,
+  "pdu_type":"m.text",
+  "prev_pdus":[["blue","99d16afbc857975916f1d73e49e52b65"]],
+  "content":...
+  "is_state":false}
+
+In contrast to the transaction layer, it is important to note that the prev_pdus
+field of a PDU refers to PDUs that any origin server has sent, rather than
+previous IDs that this origin has sent. This list may refer to other PDUs sent
+by the same origin as the current one, or other origins.
+
+Because of the distributed nature of participants in a Synapse conversation, it
+is impossible to establish a globally-consistent total ordering on the events.
+However, by annotating each outbound PDU at its origin with IDs of other PDUs it
+has received, a partial ordering can be constructed allowing causallity
+relationships to be preserved. A client can then display these messages to the
+end-user in some order consistent with their content and ensure that no message
+that is semantically in reply of an earlier one is ever displayed before it.
+
+PDUs fall into two main categories: those that deliver Events, and those that
+synchronise State. For PDUs that relate to State synchronisation, additional
+keys exist to support this:
+
+ {...,
+  "is_state":true,
+  "state_key":TODO
+  "power_level":TODO
+  "prev_state_id":TODO
+  "prev_state_origin":TODO}
+
+[[TODO(paul): At this point we should probably have a long description of how
+State management works, with descriptions of clobbering rules, power levels, etc
+etc... But some of that detail is rather up-in-the-air, on the whiteboard, and
+so on. This part needs refining. And writing in its own document as the details
+relate to the server/system as a whole, not specifically to server-server
+federation.]]
+
+
+Protocol URLs
+=============
+
+For active pushing of messages representing live activity "as it happens":
+
+  PUT /send/:transaction_id/
+    Body: JSON encoding of a single Transaction
+
+    Response: [[TODO(paul): I don't actually understand what
+    ReplicationLayer.on_transaction() is doing here, so I'm not sure what the
+    response ought to be]]
+
+  The transaction_id path argument will override any ID given in the JSON body.
+  The destination name will be set to that of the receiving server itself. Each
+  embedded PDU in the transaction body will be processed.
+
+
+To fetch a particular PDU:
+
+  GET /pdu/:origin/:pdu_id/
+
+    Response: JSON encoding of a single Transaction containing one PDU
+
+  Retrieves a given PDU from the server. The response will contain a single new
+  Transaction, inside which will be the requested PDU.
+  
+
+To fetch all the state of a given context:
+
+  GET /state/:context/
+
+    Response: JSON encoding of a single Transaction containing multiple PDUs
+
+  Retrieves a snapshot of the entire current state of the given context. The
+  response will contain a single Transaction, inside which will be a list of
+  PDUs that encode the state.
+
+
+To paginate events on a given context:
+
+  GET /paginate/:context/
+    Query args: v, limit
+
+    Response: JSON encoding of a single Transaction containing multiple PDUs
+
+  Retrieves a sliding-window history of previous PDUs that occurred on the
+  given context. Starting from the PDU ID(s) given in the "v" argument, the
+  PDUs that preceeded it are retrieved, up to a total number given by the
+  "limit" argument. These are then returned in a new Transaction containing all
+  off the PDUs.
+
+
+To stream events all the events:
+
+  GET /pull/
+    Query args: origin, v
+
+  Response: JSON encoding of a single Transaction consisting of multiple PDUs
+
+  Retrieves all of the transactions later than any version given by the "v"
+  arguments. [[TODO(paul): I'm not sure what the "origin" argument does because
+  I think at some point in the code it's got swapped around.]]