Merge branch 'release-v0.2.3' of github.com:matrix-org/synapse

Conflicts:
	webclient/room/room-controller.js

8 files changed, 1131 insertions, 5 deletions

diff --git a/docs/client-server/howto.rst b/docs/client-server/howto.rst
index c02ea8d897..ec941edddd 100644
--- a/docs/client-server/howto.rst
+++ b/docs/client-server/howto.rst
@@ -24,7 +24,7 @@ If you already have an account, you must **login** into it.
 
 `Try out the fiddle`__
 
-.. __: http://jsfiddle.net/4q2jyxng/
+.. __: http://jsfiddle.net/gh/get/jquery/1.8.3/matrix-org/synapse/tree/master/jsfiddles/register_login
 
 Registration
 ------------
@@ -87,7 +87,7 @@ user and **send a message** to that room.
 
 `Try out the fiddle`__
 
-.. __: http://jsfiddle.net/zL3zto9g/
+.. __: http://jsfiddle.net/gh/get/jquery/1.8.3/matrix-org/synapse/tree/master/jsfiddles/create_room_send_msg
 
 Creating a room
 ---------------
@@ -137,7 +137,7 @@ join a room **via a room alias** if one was set up.
 
 `Try out the fiddle`__
 
-.. __: http://jsfiddle.net/7fhotf1b/
+.. __: http://jsfiddle.net/gh/get/jquery/1.8.3/matrix-org/synapse/tree/master/jsfiddles/room_memberships
 
 Inviting a user to a room
 -------------------------
@@ -183,7 +183,7 @@ of getting events, depending on what the client already knows.
 
 `Try out the fiddle`__
 
-.. __: http://jsfiddle.net/vw11mg37/
+.. __: http://jsfiddle.net/gh/get/jquery/1.8.3/matrix-org/synapse/tree/master/jsfiddles/event_stream
 
 Getting all state
 -----------------
@@ -633,4 +633,4 @@ application.
 
 `Try out the fiddle`__
 
-.. __: http://jsfiddle.net/uztL3yme/
+.. __: http://jsfiddle.net/gh/get/jquery/1.8.3/matrix-org/synapse/tree/master/jsfiddles/example_app
diff --git a/docs/client-server/model/presence.rst b/docs/client-server/model/presence.rst
new file mode 100644
index 0000000000..7e54505364
--- /dev/null
+++ b/docs/client-server/model/presence.rst
@@ -0,0 +1,249 @@
+========
+Presence
+========
+
+A description of presence information and visibility between users.
+
+Overview
+========
+
+Each user has the concept of Presence information. This encodes a sense of the
+"availability" of that user, suitable for display on other user's clients.
+
+
+Presence Information
+====================
+
+The basic piece of presence information is an enumeration of a small set of
+state; such as "free to chat", "online", "busy", or "offline". The default state
+unless the user changes it is "online". Lower states suggest some amount of
+decreased availability from normal, which might have some client-side effect
+like muting notification sounds and suggests to other users not to bother them
+unless it is urgent. Equally, the "free to chat" state exists to let the user
+announce their general willingness to receive messages moreso than default.
+
+Home servers should also allow a user to set their state as "hidden" - a state
+which behaves as offline, but allows the user to see the client state anyway and
+generally interact with client features such as reading message history or
+accessing contacts in the address book.
+
+This basic state field applies to the user as a whole, regardless of how many
+client devices they have connected. The home server should synchronise this
+status choice among multiple devices to ensure the user gets a consistent
+experience.
+
+Idle Time
+---------
+
+As well as the basic state field, the presence information can also show a sense
+of an "idle timer". This should be maintained individually by the user's
+clients, and the homeserver can take the highest reported time as that to
+report. Likely this should be presented in fairly coarse granularity; possibly
+being limited to letting the home server automatically switch from a "free to
+chat" or "online" mode into "idle".
+
+When a user is offline, the Home Server can still report when the user was last
+seen online, again perhaps in a somewhat coarse manner.
+
+Device Type
+-----------
+
+Client devices that may limit the user experience somewhat (such as "mobile"
+devices with limited ability to type on a real keyboard or read large amounts of
+text) should report this to the home server, as this is also useful information
+to report as "presence" if the user cannot be expected to provide a good typed
+response to messages.
+
+
+Presence List
+=============
+
+Each user's home server stores a "presence list" for that user. This stores a
+list of other user IDs the user has chosen to add to it (remembering any ACL
+Pointer if appropriate).
+
+To be added to a contact list, the user being added must grant permission. Once
+granted, both user's HS(es) store this information, as it allows the user who
+has added the contact some more abilities; see below. Since such subscriptions
+are likely to be bidirectional, HSes may wish to automatically accept requests
+when a reverse subscription already exists.
+
+As a convenience, presence lists should support the ability to collect users
+into groups, which could allow things like inviting the entire group to a new
+("ad-hoc") chat room, or easy interaction with the profile information ACL
+implementation of the HS.
+
+
+Presence and Permissions
+========================
+
+For a viewing user to be allowed to see the presence information of a target
+user, either
+
+ * The target user has allowed the viewing user to add them to their presence
+   list, or
+
+ * The two users share at least one room in common
+
+In the latter case, this allows for clients to display some minimal sense of
+presence information in a user list for a room.
+
+Home servers can also use the user's choice of presence state as a signal for
+how to handle new private one-to-one chat message requests. For example, it
+might decide:
+
+  "free to chat": accept anything
+  "online": accept from anyone in my addres book list
+  "busy": accept from anyone in this "important people" group in my address
+    book list
+
+
+API Efficiency
+==============
+
+A simple implementation of presence messaging has the ability to cause a large
+amount of Internet traffic relating to presence updates. In order to minimise
+the impact of such a feature, the following observations can be made:
+
+ * There is no point in a Home Server polling status for peers in a user's
+   presence list if the user has no clients connected that care about it.
+
+ * It is highly likely that most presence subscriptions will be symmetric - a
+   given user watching another is likely to in turn be watched by that user.
+
+ * It is likely that most subscription pairings will be between users who share
+   at least one Room in common, and so their Home Servers are actively
+   exchanging message PDUs or transactions relating to that Room.
+
+ * Presence update messages do not need realtime guarantees. It is acceptable to
+   delay delivery of updates for some small amount of time (10 seconds to a
+   minute).
+
+The general model of presence information is that of a HS registering its
+interest in receiving presence status updates from other HSes, which then
+promise to send them when required. Rather than actively polling for the
+currentt state all the time, HSes can rely on their relative stability to only
+push updates when required.
+
+A Home Server should not rely on the longterm validity of this presence
+information, however, as this would not cover such cases as a user's server
+crashing and thus failing to inform their peers that users it used to host are
+no longer available online. Therefore, each promise of future updates should
+carry with a timeout value (whether explicit in the message, or implicit as some
+defined default in the protocol), after which the receiving HS should consider
+the information potentially stale and request it again.
+
+However, because of the likelyhood that two home servers are exchanging messages
+relating to chat traffic in a room common to both of them, the ongoing receipt
+of these messages can be taken by each server as an implicit notification that
+the sending server is still up and running, and therefore that no status changes
+have happened; because if they had the server would have sent them. A second,
+larger timeout should be applied to this implicit inference however, to protect
+against implementation bugs or other reasons that the presence state cache may
+become invalid; eventually the HS should re-enquire the current state of users
+and update them with its own.
+
+The following workflows can therefore be used to handle presence updates:
+
+ 1 When a user first appears online their HS sends a message to each other HS
+   containing at least one user to be watched; each message carrying both a
+   notification of the sender's new online status, and a request to obtain and
+   watch the target users' presence information. This message implicitly
+   promises the sending HS will now push updates to the target HSes.
+
+ 2 The target HSes then respond a single message each, containing the current
+   status of the requested user(s). These messages too implicitly promise the
+   target HSes will themselves push updates to the sending HS.
+
+   As these messages arrive at the sending user's HS they can be pushed to the
+   user's client(s), possibly batched again to ensure not too many small
+   messages which add extra protocol overheads.
+
+At this point, all the user's clients now have the current presence status
+information for this moment in time, and have promised to send each other
+updates in future.
+
+ 3 The HS maintains two watchdog timers per peer HS it is exchanging presence
+   information with. The first timer should have a relatively small expiry
+   (perhaps 1 minute), and the second timer should have a much longer time
+   (perhaps 1 hour).
+
+ 4 Any time any kind of message is received from a peer HS, the short-term
+   presence timer associated with it is reset.
+
+ 5 Whenever either of these timers expires, an HS should push a status reminder
+   to the target HS whose timer has now expired, and request again from that
+   server the status of the subscribed users.
+
+ 6 On receipt of one of these presence status reminders, an HS can reset both
+   of its presence watchdog timers.
+
+To avoid bursts of traffic, implementations should attempt to stagger the expiry
+of the longer-term watchdog timers for different peer HSes.
+
+When individual users actively change their status (either by explicit requests
+from clients, or inferred changes due to idle timers or client timeouts), the HS
+should batch up any status changes for some reasonable amount of time (10
+seconds to a minute). This allows for reduced protocol overheads in the case of
+multiple messages needing to be sent to the same peer HS; as is the likely
+scenario in many cases, such as a given human user having multiple user
+accounts.
+
+
+API Requirements
+================
+
+The data model presented here puts the following requirements on the APIs:
+
+Client-Server
+-------------
+
+Requests that a client can make to its Home Server
+
+ * get/set current presence state
+   Basic enumeration + ability to set a custom piece of text
+
+ * report per-device idle time
+   After some (configurable?) idle time the device should send a single message
+   to set the idle duration. The HS can then infer a "start of idle" instant and
+   use that to keep the device idleness up to date. At some later point the
+   device can cancel this idleness.
+
+ * report per-device type
+   Inform the server that this device is a "mobile" device, or perhaps some
+   other to-be-defined category of reduced capability that could be presented to
+   other users.
+
+ * start/stop presence polling for my presence list
+   It is likely that these messages could be implicitly inferred by other
+   messages, though having explicit control is always useful.
+
+ * get my presence list
+   [implicit poll start?]
+   It is possible that the HS doesn't yet have current presence information when
+   the client requests this. There should be a "don't know" type too.
+
+ * add/remove a user to my presence list
+
+Server-Server
+-------------
+
+Requests that Home Servers make to others
+
+ * request permission to add a user to presence list
+
+ * allow/deny a request to add to a presence list
+
+ * perform a combined presence state push and subscription request
+   For each sending user ID, the message contains their new status.
+   For each receiving user ID, the message should contain an indication on
+   whether the sending server is also interested in receiving status from that
+   user; either as an immediate update response now, or as a promise to send
+   future updates.
+
+Server to Client
+----------------
+
+[[TODO(paul): There also needs to be some way for a user's HS to push status
+updates of the presence list to clients, but the general server-client event
+model currently lacks a space to do that.]]
diff --git a/docs/client-server/model/profiles.rst b/docs/client-server/model/profiles.rst
new file mode 100644
index 0000000000..f7d6bd5679
--- /dev/null
+++ b/docs/client-server/model/profiles.rst
@@ -0,0 +1,232 @@
+========
+Profiles
+========
+
+A description of Synapse user profile metadata support.
+
+
+Overview
+========
+
+Internally within Synapse users are referred to by an opaque ID, which consists
+of some opaque localpart combined with the domain name of their home server.
+Obviously this does not yield a very nice user experience; users would like to
+see readable names for other users that are in some way meaningful to them.
+Additionally, users like to be able to publish "profile" details to inform other
+users of other information about them.
+
+It is also conceivable that since we are attempting to provide a
+worldwide-applicable messaging system, that users may wish to present different
+subsets of information in their profile to different other people, from a
+privacy and permissions perspective.
+
+A Profile consists of a display name, an (optional?) avatar picture, and a set
+of other metadata fields that the user may wish to publish (email address, phone
+numbers, website URLs, etc...). We put no requirements on the display name other
+than it being a valid Unicode string. Since it is likely that users will end up
+having multiple accounts (perhaps by necessity of being hosted in multiple
+places, perhaps by choice of wanting multiple distinct identifies), it would be
+useful that a metadata field type exists that can refer to another Synapse User
+ID, so that clients and HSes can make use of this information.
+
+Metadata Fields
+---------------
+
+[[TODO(paul): Likely this list is incomplete; more fields can be defined as we
+think of them. At the very least, any sort of supported ID for the 3rd Party ID
+servers should be accounted for here.]]
+
+ * Synapse Directory Server username(s)
+
+ * Email address
+
+ * Phone number - classify "home"/"work"/"mobile"/custom?
+ 
+ * Twitter/Facebook/Google+/... social networks
+
+ * Location - keep this deliberately vague to allow people to choose how
+     granular it is
+ 
+ * "Bio" information - date of birth, etc...
+
+ * Synapse User ID of another account
+
+ * Web URL
+
+ * Freeform description text
+
+
+Visibility Permissions
+======================
+
+A home server implementation could offer the ability to set permissions on
+limited visibility of those fields. When another user requests access to the
+target user's profile, their own identity should form part of that request. The
+HS implementation can then decide which fields to make available to the
+requestor.
+
+A particular detail of implementation could allow the user to create one or more
+ACLs; where each list is granted permission to see a given set of non-public
+fields (compare to Google+ Circles) and contains a set of other people allowed
+to use it. By giving these ACLs strong identities within the HS, they can be
+referenced in communications with it, granting other users who encounter these
+the "ACL Token" to use the details in that ACL.
+
+If we further allow an ACL Token to be present on Room join requests or stored
+by 3PID servers, then users of these ACLs gain the extra convenience of not
+having to manually curate people in the access list; anyone in the room or with
+knowledge of the 3rd Party ID is automatically granted access. Every HS and
+client implementation would have to be aware of the existence of these ACL
+Token, and include them in requests if present, but not every HS implementation
+needs to actually provide the full permissions model. This can be used as a
+distinguishing feature among competing implementations. However, servers MUST
+NOT serve profile information from a cache if there is a chance that its limited
+understanding could lead to information leakage.
+
+
+Client Concerns of Multiple Accounts
+====================================
+
+Because a given person may want to have multiple Synapse User accounts, client
+implementations should allow the use of multiple accounts simultaneously
+(especially in the field of mobile phone clients, which generally don't support
+running distinct instances of the same application). Where features like address
+books, presence lists or rooms are presented, the client UI should remember to
+make distinct with user account is in use for each.
+
+
+Directory Servers
+=================
+
+Directory Servers can provide a forward mapping from human-readable names to
+User IDs. These can provide a service similar to giving domain-namespaced names
+for Rooms; in this case they can provide a way for a user to reference their
+User ID in some external form (e.g. that can be printed on a business card).
+
+The format for Synapse user name will consist of a localpart specific to the
+directory server, and the domain name of that directory server:
+
+  @localname:some.domain.name
+
+The localname is separated from the domain name using a colon, so as to ensure
+the localname can still contain periods, as users may want this for similarity
+to email addresses or the like, which typically can contain them. The format is
+also visually quite distinct from email addresses, phone numbers, etc... so
+hopefully reasonably "self-describing" when written on e.g. a business card
+without surrounding context.
+
+[[TODO(paul): we might have to think about this one - too close to email?
+  Twitter? Also it suggests a format scheme for room names of
+  #localname:domain.name, which I quite like]]
+
+Directory server administrators should be able to make some kind of policy
+decision on how these are allocated. Servers within some "closed" domain (such
+as company-specific ones) may wish to verify the validity of a mapping using
+their own internal mechanisms; "public" naming servers can operate on a FCFS
+basis. There are overlapping concerns here with the idea of the 3rd party
+identity servers as well, though in this specific case we are creating a new
+namespace to allocate names into.
+
+It would also be nice from a user experience perspective if the profile that a
+given name links to can also declare that name as part of its metadata.
+Furthermore as a security and consistency perspective it would be nice if each
+end (the directory server and the user's home server) check the validity of the
+mapping in some way. This needs investigation from a security perspective to
+ensure against spoofing.
+
+One such model may be that the user starts by declaring their intent to use a
+given user name link to their home server, which then contacts the directory
+service. At some point later (maybe immediately for "public open FCFS servers",
+maybe after some kind of human intervention for verification) the DS decides to
+honour this link, and includes it in its served output. It should also tell the
+HS of this fact, so that the HS can present this as fact when requested for the
+profile information. For efficiency, it may further wish to provide the HS with
+a cryptographically-signed certificate as proof, so the HS serving the profile
+can provide that too when asked, avoiding requesting HSes from constantly having
+to contact the DS to verify this mapping. (Note: This is similar to the security
+model often applied in DNS to verify PTR <-> A bidirectional mappings).
+
+
+Identity Servers
+================
+
+The identity servers should support the concept of pointing a 3PID being able to
+store an ACL Token as well as the main User ID. It is however, beyond scope to
+do any kind of verification that any third-party IDs that the profile is
+claiming match up to the 3PID mappings.
+
+
+User Interface and Expectations Concerns
+========================================
+
+Given the weak "security" of some parts of this model as compared to what users
+might expect, some care should be taken on how it is presented to users,
+specifically in the naming or other wording of user interface components.
+
+Most notably mere knowledge of an ACL Pointer is enough to read the information
+stored in it. It is possible that Home or Identity Servers could leak this
+information, allowing others to see it. This is a security-vs-convenience
+balancing choice on behalf of the user who would choose, or not, to make use of
+such a feature to publish their information.
+
+Additionally, unless some form of strong end-to-end user-based encryption is
+used, a user of ACLs for information privacy has to trust other home servers not
+to lie about the identify of the user requesting access to the Profile.
+
+
+API Requirements
+================
+
+The data model presented here puts the following requirements on the APIs:
+
+Client-Server
+-------------
+
+Requests that a client can make to its Home Server
+
+ * get/set my Display Name
+   This should return/take a simple "text/plain" field
+
+ * get/set my Avatar URL
+   The avatar image data itself is not stored by this API; we'll just store a
+   URL to let the clients fetch it. Optionally HSes could integrate this with
+   their generic content attacmhent storage service, allowing a user to set
+   upload their profile Avatar and update the URL to point to it.
+
+ * get/add/remove my metadata fields
+   Also we need to actually define types of metadata
+
+ * get another user's Display Name / Avatar / metadata fields
+
+[[TODO(paul): At some later stage we should consider the API for:
+
+ * get/set ACL permissions on my metadata fields
+
+ * manage my ACL tokens
+]]
+
+Server-Server
+-------------
+
+Requests that Home Servers make to others
+
+ * get a user's Display Name / Avatar
+
+ * get a user's full profile - name/avatar + MD fields
+   This request must allow for specifying the User ID of the requesting user,
+   for permissions purposes. It also needs to take into account any ACL Tokens
+   the requestor has.
+
+ * push a change of Display Name to observers (overlaps with the presence API)
+
+Room Event PDU Types
+--------------------
+
+Events that are pushed from Home Servers to other Home Servers or clients.
+
+ * user Display Name change
+ 
+ * user Avatar change
+   [[TODO(paul): should the avatar image itself be stored in all the room
+   histories? maybe this event should just be a hint to clients that they should
+   re-fetch the avatar image]]
diff --git a/docs/client-server/model/protocol_examples.rst b/docs/client-server/model/protocol_examples.rst
new file mode 100644
index 0000000000..61a599b432
--- /dev/null
+++ b/docs/client-server/model/protocol_examples.rst
@@ -0,0 +1,64 @@
+PUT /send/abc/ HTTP/1.1
+Host: ...
+Content-Length: ...
+Content-Type: application/json
+
+{
+    "origin": "localhost:5000",
+    "pdus": [
+        {
+            "content": {},
+            "context": "tng",
+            "depth": 12,
+            "is_state": false,
+            "origin": "localhost:5000",
+            "pdu_id": 1404381396854,
+            "pdu_type": "feedback",
+            "prev_pdus": [
+                [
+                    "1404381395883",
+                    "localhost:6000"
+                ]
+            ],
+            "ts": 1404381427581
+        }
+    ],
+    "prev_ids": [
+        "1404381396852"
+    ],
+    "ts": 1404381427823
+}
+
+HTTP/1.1 200 OK
+...
+
+======================================
+
+GET /pull/-1/ HTTP/1.1
+Host: ...
+Content-Length: 0
+
+HTTP/1.1 200 OK
+Content-Length: ...
+Content-Type: application/json
+
+{
+    origin: ...,
+    prev_ids: ...,
+    data: [
+        {
+            data_id: ...,
+            prev_pdus: [...],
+            depth: ...,
+            ts: ...,
+            context: ...,
+            origin: ...,
+            content: {
+                ...
+            }
+        },
+        ...,
+    ]
+}
+
+
diff --git a/docs/client-server/model/room-join-workflow.rst b/docs/client-server/model/room-join-workflow.rst
new file mode 100644
index 0000000000..c321a64fab
--- /dev/null
+++ b/docs/client-server/model/room-join-workflow.rst
@@ -0,0 +1,113 @@
+==================
+Room Join Workflow
+==================
+
+An outline of the workflows required when a user joins a room.
+
+Discovery
+=========
+
+To join a room, a user has to discover the room by some mechanism in order to
+obtain the (opaque) Room ID and a candidate list of likely home servers that
+contain it.
+
+Sending an Invitation
+---------------------
+
+The most direct way a user discovers the existence of a room is from a
+invitation from some other user who is a member of that room.
+
+The inviter's HS sets the membership status of the invitee to "invited" in the
+"m.members" state key by sending a state update PDU. The HS then broadcasts this
+PDU among the existing members in the usual way. An invitation message is also
+sent to the invited user, containing the Room ID and the PDU ID of this
+invitation state change and potentially a list of some other home servers to use
+to accept the invite. The user's client can then choose to display it in some
+way to alert the user.
+
+[[TODO(paul): At present, no API has been designed or described to actually send
+that invite to the invited user. Likely it will be some facet of the larger
+user-user API required for presence, profile management, etc...]]
+
+Directory Service
+-----------------
+
+Alternatively, the user may discover the channel via a directory service; either
+by performing a name lookup, or some kind of browse or search acitivty. However
+this is performed, the end result is that the user's home server requests the
+Room ID and candidate list from the directory service.
+
+[[TODO(paul): At present, no API has been designed or described for this
+directory service]]
+
+
+Joining
+=======
+
+Once the ID and home servers are obtained, the user can then actually join the
+room.
+
+Accepting an Invite
+-------------------
+
+If a user has received and accepted an invitation to join a room, the invitee's
+home server can now send an invite acceptance message to a chosen candidate
+server from the list given in the invitation, citing also the PDU ID of the
+invitation as "proof" of their invite. (This is required as due to late message
+propagation it could be the case that the acceptance is received before the
+invite by some servers). If this message is allowed by the candidate server, it
+generates a new PDU that updates the invitee's membership status to "joined",
+referring back to the acceptance PDU, and broadcasts that as a state change in
+the usual way. The newly-invited user is now a full member of the room, and
+state propagation proceeds as usual.
+
+Joining a Public Room
+---------------------
+
+If a user has discovered the existence of a room they wish to join but does not
+have an active invitation, they can request to join it directly by sending a
+join message to a candidate server on the list provided by the directory
+service. As this list may be out of date, the HS should be prepared to retry
+other candidates if the chosen one is no longer aware of the room, because it
+has no users as members in it.
+
+Once a candidate server that is aware of the room has been found, it can
+broadcast an update PDU to add the member into the "m.members" key setting their
+state directly to "joined" (i.e. bypassing the two-phase invite semantics),
+remembering to include the new user's HS in that list.
+
+Knocking on a Semi-Public Room
+------------------------------
+
+If a user requests to join a room but the join mode of the room is "knock", the
+join is not immediately allowed. Instead, if the user wishes to proceed, they
+can instead post a "knock" message, which informs other members of the room that
+the would-be joiner wishes to become a member and sets their membership value to
+"knocked". If any of them wish to accept this, they can then send an invitation
+in the usual way described above. Knowing that the user has already knocked and
+expressed an interest in joining, the invited user's home server should
+immediately accept that invitation on the user's behalf, and go on to join the
+room in the usual way.
+
+[[NOTE(Erik): Though this may confuse users who expect 'X has joined' to
+actually be a user initiated action, i.e. they may expect that 'X' is actually
+looking at synapse right now?]]
+
+[[NOTE(paul): Yes, a fair point maybe we should suggest HSes don't do that, and
+just offer an invite to the user as normal]]
+
+Private and Non-Existent Rooms
+------------------------------
+
+If a user requests to join a room but the room is either unknown by the home
+server receiving the request, or is known by the join mode is "invite" and the
+user has not been invited, the server must respond that the room does not exist.
+This is to prevent leaking information about the existence and identity of
+private rooms.
+
+
+Outstanding Questions
+=====================
+
+ * Do invitations or knocks time out and expire at some point? If so when? Time
+   is hard in distributed systems.
diff --git a/docs/client-server/model/rooms.rst b/docs/client-server/model/rooms.rst
new file mode 100644
index 0000000000..0007e48e30
--- /dev/null
+++ b/docs/client-server/model/rooms.rst
@@ -0,0 +1,274 @@
+===========
+Rooms Model
+===========
+
+A description of the general data model used to implement Rooms, and the
+user-level visible effects and implications.
+
+
+Overview
+========
+
+"Rooms" in Synapse are shared messaging channels over which all the participant
+users can exchange messages. Rooms have an opaque persistent identify, a
+globally-replicated set of state (consisting principly of a membership set of
+users, and other management and miscellaneous metadata), and a message history.
+
+
+Room Identity and Naming
+========================
+
+Rooms can be arbitrarily created by any user on any home server; at which point
+the home server will sign the message that creates the channel, and the
+fingerprint of this signature becomes the strong persistent identify of the
+room. This now identifies the room to any home server in the network regardless
+of its original origin. This allows the identify of the room to outlive any
+particular server. Subject to appropriate permissions [to be discussed later],
+any current member of a room can invite others to join it, can post messages
+that become part of its history, and can change the persistent state of the room
+(including its current set of permissions).
+
+Home servers can provide a directory service, allowing a lookup from a
+convenient human-readable form of room label to a room ID. This mapping is
+scoped to the particular home server domain and so simply represents that server
+administrator's opinion of what room should take that label; it does not have to
+be globally replicated and does not form part of the stored state of that room.
+
+This room name takes the form
+
+  #localname:some.domain.name
+
+for similarity and consistency with user names on directories.
+
+To join a room (and therefore to be allowed to inspect past history, post new
+messages to it, and read its state), a user must become aware of the room's
+fingerprint ID. There are two mechanisms to allow this:
+
+ * An invite message from someone else in the room
+
+ * A referral from a room directory service
+
+As room IDs are opaque and ephemeral, they can serve as a mechanism to create
+"ad-hoc" rooms deliberately unnamed, for small group-chats or even private
+one-to-one message exchange.
+
+
+Stored State and Permissions
+============================
+
+Every room has a globally-replicated set of stored state. This state is a set of
+key/value or key/subkey/value pairs. The value of every (sub)key is a
+JSON-representable object. The main key of a piece of stored state establishes
+its meaning; some keys store sub-keys to allow a sub-structure within them [more
+detail below]. Some keys have special meaning to Synapse, as they relate to
+management details of the room itself, storing such details as user membership,
+and permissions of users to alter the state of the room itself. Other keys may
+store information to present to users, which the system does not directly rely
+on. The key space itself is namespaced, allowing 3rd party extensions, subject
+to suitable permission.
+
+Permission management is based on the concept of "power-levels". Every user
+within a room has an integer assigned, being their "power-level" within that
+room. Along with its actual data value, each key (or subkey) also stores the
+minimum power-level a user must have in order to write to that key, the
+power-level of the last user who actually did write to it, and the PDU ID of
+that state change.
+
+To be accepted as valid, a change must NOT:
+
+ * Be made by a user having a power-level lower than required to write to the
+   state key
+
+ * Alter the required power-level for that state key to a value higher than the
+   user has
+
+ * Increase that user's own power-level
+
+ * Grant any other user a power-level higher than the level of the user making
+   the change
+
+[[TODO(paul): consider if relaxations should be allowed; e.g. is the current
+outright-winner allowed to raise their own level, to allow for "inflation"?]]
+
+
+Room State Keys
+===============
+
+[[TODO(paul): if this list gets too big it might become necessary to move it
+into its own doc]]
+
+The following keys have special semantics or meaning to Synapse itself:
+
+m.member (has subkeys)
+  Stores a sub-key for every Synapse User ID which is currently a member of
+  this room. Its value gives the membership type ("knocked", "invited",
+  "joined").
+
+m.power_levels
+  Stores a mapping from Synapse User IDs to their power-level in the room. If
+  they are not present in this mapping, the default applies.
+
+  The reason to store this as a single value rather than a value with subkeys
+  is that updates to it are atomic; allowing a number of colliding-edit
+  problems to be avoided.
+
+m.default_level
+  Gives the default power-level for members of the room that do not have one
+  specified in their membership key.
+
+m.invite_level
+  If set, gives the minimum power-level required for members to invite others
+  to join, or to accept knock requests from non-members requesting access. If
+  absent, then invites are not allowed. An invitation involves setting their
+  membership type to "invited", in addition to sending the invite message.
+
+m.join_rules
+  Encodes the rules on how non-members can join the room. Has the following
+  possibilities:
+    "public" - a non-member can join the room directly
+    "knock" - a non-member cannot join the room, but can post a single "knock"
+        message requesting access, which existing members may approve or deny
+    "invite" - non-members cannot join the room without an invite from an
+        existing member
+    "private" - nobody who is not in the 'may_join' list or already a member
+        may join by any mechanism
+
+  In any of the first three modes, existing members with sufficient permission
+  can send invites to non-members if allowed by the "m.invite_level" key. A
+  "private" room is not allowed to have the "m.invite_level" set.
+
+  A client may use the value of this key to hint at the user interface
+  expectations to provide; in particular, a private chat with one other use
+  might warrant specific handling in the client.
+
+m.may_join
+  A list of User IDs that are always allowed to join the room, regardless of any
+  of the prevailing join rules and invite levels. These apply even to private
+  rooms. These are stored in a single list with normal update-powerlevel
+  permissions applied; users cannot arbitrarily remove themselves from the list.
+
+m.add_state_level
+  The power-level required for a user to be able to add new state keys.
+
+m.public_history
+  If set and true, anyone can request the history of the room, without needing
+  to be a member of the room.
+
+m.archive_servers
+  For "public" rooms with public history, gives a list of home servers that
+  should be included in message distribution to the room, even if no users on
+  that server are present. These ensure that a public room can still persist
+  even if no users are currently members of it. This list should be consulted by
+  the dirctory servers as the candidate list they respond with.
+
+The following keys are provided by Synapse for user benefit, but their value is
+not otherwise used by Synapse.
+
+m.name
+  Stores a short human-readable name for the room, such that clients can display
+  to a user to assist in identifying which room is which.
+  
+  This name specifically is not the strong ID used by the message transport
+  system to refer to the room, because it may be changed from time to time.
+
+m.topic
+  Stores the current human-readable topic
+
+
+Room Creation Templates
+=======================
+
+A client (or maybe home server?) could offer a few templates for the creation of
+new rooms. For example, for a simple private one-to-one chat the channel could
+assign the creator a power-level of 1, requiring a level of 1 to invite, and
+needing an invite before members can join. An invite is then sent to the other
+party, and if accepted and the other user joins, the creator's power-level can
+now be reduced to 0. This now leaves a room with two participants in it being
+unable to add more.
+
+
+Rooms that Continue History
+===========================
+
+An option that could be considered for room creation, is that when a new room is
+created the creator could specify a PDU ID into an existing room, as the history
+continuation point. This would be stored as an extra piece of meta-data on the
+initial PDU of the room's creation. (It does not appear in the normal previous
+PDU linkage).
+
+This would allow users in rooms to "fork" a room, if it is considered that the
+conversations in the room no longer fit its original purpose, and wish to
+diverge. Existing permissions on the original room would continue to apply of
+course, for viewing that history. If both rooms are considered "public" we might
+also want to define a message to post into the original room to represent this
+fork point, and give a reference to the new room.
+
+
+User Direct Message Rooms
+=========================
+
+There is no need to build a mechanism for directly sending messages between
+users, because a room can handle this ability. To allow direct user-to-user chat
+messaging we simply need to be able to create rooms with specific set of
+permissions to allow this direct messaging.
+
+Between any given pair of user IDs that wish to exchange private messages, there
+will exist a single shared Room, created lazily by either side. These rooms will
+need a certain amount of special handling in both home servers and display on
+clients, but as much as possible should be treated by the lower layers of code
+the same as other rooms.
+
+Specially, a client would likely offer a special menu choice associated with
+another user (in room member lists, presence list, etc..) as "direct chat". That
+would perform all the necessary steps to create the private chat room. Receiving
+clients should display these in a special way too as the room name is not
+important; instead it should distinguish them on the Display Name of the other
+party.
+
+Home Servers will need a client-API option to request setting up a new user-user
+chat room, which will then need special handling within the server. It will
+create a new room with the following 
+
+  m.member: the proposing user
+  m.join_rules: "private"
+  m.may_join: both users
+  m.power_levels: empty
+  m.default_level: 0
+  m.add_state_level: 0
+  m.public_history: False
+
+Having created the room, it can send an invite message to the other user in the
+normal way - the room permissions state that no users can be set to the invited
+state, but because they're in the may_join list then they'd be allowed to join
+anyway.
+
+In this arrangement there is now a room with both users may join but neither has
+the power to invite any others. Both users now have the confidence that (at
+least within the messaging system itself) their messages remain private and
+cannot later be provably leaked to a third party. They can freely set the topic
+or name if they choose and add or edit any other state of the room. The update
+powerlevel of each of these fixed properties should be 1, to lock out the users
+from being able to alter them.
+
+
+Anti-Glare
+==========
+
+There exists the possibility of a race condition if two users who have no chat
+history with each other simultaneously create a room and invite the other to it.
+This is called a "glare" situation. There are two possible ideas for how to
+resolve this:
+
+ * Each Home Server should persist the mapping of (user ID pair) to room ID, so
+   that duplicate requests can be suppressed. On receipt of a room creation
+   request that the HS thinks there already exists a room for, the invitation to
+   join can be rejected if:
+      a) the HS believes the sending user is already a member of the room (and
+         maybe their HS has forgotten this fact), or
+      b) the proposed room has a lexicographically-higher ID than the existing
+         room (to resolve true race condition conflicts)
+      
+ * The room ID for a private 1:1 chat has a special form, determined by
+   concatenting the User IDs of both members in a deterministic order, such that
+   it doesn't matter which side creates it first; the HSes can just ignore
+   (or merge?) received PDUs that create the room twice.
diff --git a/docs/client-server/model/terminology.rst b/docs/client-server/model/terminology.rst
new file mode 100644
index 0000000000..cc6e6760ac
--- /dev/null
+++ b/docs/client-server/model/terminology.rst
@@ -0,0 +1,86 @@
+===========
+Terminology
+===========
+
+A list of definitions of specific terminology used among these documents.
+These terms were originally taken from the server-server documentation, and may
+not currently match the exact meanings used in other places; though as a
+medium-term goal we should encourage the unification of this terminology.
+
+
+Terms
+=====
+
+Backfilling:
+  The process of synchronising historic state from one home server to another,
+  to backfill the event storage so that scrollback can be presented to the
+  client(s).  (Formerly, and confusingly, called 'pagination')
+
+Context:
+  A single human-level entity of interest (currently, a chat room)
+
+EDU (Ephemeral Data Unit):
+  A message that relates directly to a given pair of home servers that are
+  exchanging it. EDUs are short-lived messages that related only to one single
+  pair of servers; they are not persisted for a long time and are not forwarded
+  on to other servers. Because of this, they have no internal ID nor previous
+  EDUs reference chain.
+
+Event:
+  A record of activity that records a single thing that happened on to a context
+  (currently, a chat room). These are the "chat messages" that Synapse makes
+  available.
+  [[NOTE(paul): The current server-server implementation calls these simply
+  "messages" but the term is too ambiguous here; I've called them Events]]
+
+PDU (Persistent Data Unit):
+  A message that relates to a single context, irrespective of the server that
+  is communicating it. PDUs either encode a single Event, or a single State
+  change. A PDU is referred to by its PDU ID; the pair of its origin server
+  and local reference from that server.
+
+PDU ID:
+  The pair of PDU Origin and PDU Reference, that together globally uniquely
+  refers to a specific PDU.
+
+PDU Origin:
+  The name of the origin server that generated a given PDU. This may not be the
+  server from which it has been received, due to the way they are copied around
+  from server to server. The origin always records the original server that
+  created it.
+
+PDU Reference:
+  A local ID used to refer to a specific PDU from a given origin server. These
+  references are opaque at the protocol level, but may optionally have some
+  structured meaning within a given origin server or implementation.
+
+Presence:
+  The concept of whether a user is currently online, how available they declare
+  they are, and so on. See also: doc/model/presence
+
+Profile:
+  A set of metadata about a user, such as a display name, provided for the
+  benefit of other users. See also: doc/model/profiles
+
+Room ID:
+  An opaque string (of as-yet undecided format) that identifies a particular
+  room and used in PDUs referring to it.
+
+Room Alias:
+  A human-readable string of the form #name:some.domain that users can use as a
+  pointer to identify a room; a Directory Server will map this to its Room ID
+
+State:
+  A set of metadata maintained about a Context, which is replicated among the
+  servers in addition to the history of Events.
+
+User ID:
+  A string of the form @localpart:domain.name that identifies a user for
+  wire-protocol purposes. The localpart is meaningless outside of a particular
+  home server. This takes a human-readable form that end-users can use directly
+  if they so wish, avoiding the 3PIDs.
+
+Transaction:
+  A message which relates to the communication between a given pair of servers.
+  A transaction contains possibly-empty lists of PDUs and EDUs.
+
diff --git a/docs/client-server/model/third-party-id.rst b/docs/client-server/model/third-party-id.rst
new file mode 100644
index 0000000000..1f8138ddf7
--- /dev/null
+++ b/docs/client-server/model/third-party-id.rst
@@ -0,0 +1,108 @@
+======================
+Third Party Identities
+======================
+
+A description of how email addresses, mobile phone numbers and other third
+party identifiers can be used to authenticate and discover users in Matrix.
+
+
+Overview
+========
+
+New users need to authenticate their account. An email or SMS text message can 
+be a convenient form of authentication. Users already have email addresses 
+and phone numbers for contacts in their address book. They want to communicate
+with those contacts in Matrix without manually exchanging a Matrix User ID with 
+them.
+
+Third Party IDs
+---------------
+
+[[TODO(markjh): Describe the format of a 3PID]]
+
+
+Third Party ID Associations
+---------------------------
+
+An Associaton is a binding between a Matrix User ID and a Third Party ID (3PID).
+Each 3PID can be associated with one Matrix User ID at a time.
+
+[[TODO(markjh): JSON format of the association.]]
+
+Verification 
+------------
+
+An Assocation must be verified by a trusted Verification Server. Email 
+addresses and phone numbers can be verified by sending a token to the address 
+which a client can supply to the verifier to confirm ownership.
+
+An email Verification Server may be capable of verifying all email 3PIDs or may
+be restricted to verifying addresses for a particular domain. A phone number
+Verification Server may be capable of verifying all phone numbers or may be
+restricted to verifying numbers for a given country or phone prefix.
+
+Verification Servers fulfil a similar role to Certificate Authorities in PKI so
+a similar level of vetting should be required before clients trust their
+signatures.
+
+A Verification Server may wish to check for existing Associations for a 3PID 
+before creating a new Association.
+
+Discovery
+---------
+
+Users can discover Associations using a trusted Identity Server. Each 
+Association will be signed by the Identity Server. An Identity Server may store
+the entire space of Associations or may delegate to other Identity Servers when
+looking up Associations.
+
+Each Association returned from an Identity Server must be signed by a 
+Verification Server. Clients should check these signatures.
+
+Identity Servers fulfil a similar role to DNS servers.
+
+Privacy
+-------
+
+A User may publish the association between their phone number and Matrix User ID
+on the Identity Server without publishing the number in their Profile hosted on
+their Home Server.
+
+Identity Servers should refrain from publishing reverse mappings and should 
+take steps, such as rate limiting, to prevent attackers enumerating the space of
+mappings.
+
+Federation
+==========
+
+Delegation
+----------
+
+Verification Servers could delegate signing to another server by issuing 
+certificate to that server allowing it to verify and sign a subset of 3PID on 
+its behalf. It would be necessary to provide a language for describing which
+subset of 3PIDs that server had authority to validate. Alternatively it could 
+delegate the verification step to another server but sign the resulting
+association itself.
+
+The 3PID space will have a heirachical structure like DNS so Identity Servers
+can delegate lookups to other servers. An Identity Server should be prepared 
+to host or delegate any valid association within the subset of the 3PIDs it is 
+resonsible for.
+
+Multiple Root Verification Servers
+----------------------------------
+
+There can be multiple root Verification Servers and an Association could be
+signed by multiple servers if different clients trust different subsets of
+the verification servers.
+
+Multiple Root Identity Servers
+------------------------------
+
+There can be be multiple root Identity Servers. Clients will add each
+Association to all root Identity Servers.
+
+[[TODO(markjh): Describe how clients find the list of root Identity Servers]]
+
+