diff options
| author | Erik Johnston <erik@matrix.org> | 2014-09-03 19:47:32 +0100 |
|---|---|---|
| committer | Erik Johnston <erik@matrix.org> | 2014-09-03 19:47:32 +0100 |
| commit | 0538a4098d797eaa901c168e3a76df2ac94da2ed (patch) | |
| tree | bc1ef822fdb3184daf6d39a2ab4da1afb6370dd9 | |
| parent | Add database upgrade script (diff) | |
| parent | Bump versions. Update change logs. (diff) | |
| download | synapse-0538a4098d797eaa901c168e3a76df2ac94da2ed.tar.xz | |
Merge branch 'release-v0.2.1' v0.2.1
189 files changed, 4260 insertions, 1090 deletions
diff --git a/CHANGES.rst b/CHANGES.rst index 292f7eee62..31eee891da 100644 --- a/CHANGES.rst +++ b/CHANGES.rst @@ -1,3 +1,20 @@ +Changes in synapse 0.2.1 (2014-09-03) +===================================== + +Homeserver: + * Added support for signing up with a third party id. + * Add synctl scripts. + * Added rate limiting. + * Add option to change the external address the content repo uses. + * Presence bug fixes. + +Webclient: + * Added support for signing up with a third party id. + * Added support for banning and kicking users. + * Added support for displaying and setting ops. + * Added support for room names. + * Fix bugs with room membership event display. + Changes in synapse 0.2.0 (2014-09-02) ===================================== This update changes many configuration options, updates the diff --git a/README.rst b/README.rst index 2d355d9649..98af91ea42 100644 --- a/README.rst +++ b/README.rst @@ -2,11 +2,11 @@ Introduction ============ Matrix is an ambitious new ecosystem for open federated Instant Messaging and -VoIP[1]. The basics you need to know to get up and running are: +VoIP. The basics you need to know to get up and running are: - Chatrooms are distributed and do not exist on any single server. Rooms can be found using names like ``#matrix:matrix.org`` or - ``#test:localhost:8080`` or they can be ephemeral. + ``#test:localhost:8008`` or they can be ephemeral. - Matrix user IDs look like ``@matthew:matrix.org`` (although in the future you will normally refer to yourself and others using a 3PID: email @@ -14,8 +14,8 @@ VoIP[1]. The basics you need to know to get up and running are: The overall architecture is:: - client <----> homeserver <=================> homeserver <-----> client - e.g. matrix.org:8080 e.g. mydomain.net:8080 + client <----> homeserver <=====================> homeserver <----> client + https://matrix.org/_matrix https://mydomain.net/_matrix Quick Start =========== @@ -25,22 +25,20 @@ To get up and running: - To simply play with an **existing** homeserver you can just go straight to http://matrix.org/alpha. - - To run your own **private** homeserver on localhost:8080, install synapse + - To run your own **private** homeserver on localhost:8008, install synapse with ``python setup.py develop --user`` and then run one with ``python synapse/app/homeserver.py`` - you will find a webclient running - at http://localhost:8080 (use a recent Chrome, Safari or Firefox for now, + at http://localhost:8008 (use a recent Chrome, Safari or Firefox for now, please...) - To make the homeserver **public** and let it exchange messages with other homeservers and participate in the overall Matrix federation, open - up port 8080 and run ``python synapse/app/homeserver.py --host + up port 8448 and run ``python synapse/app/homeserver.py --host machine.my.domain.name``. Then come join ``#matrix:matrix.org`` and say hi! :) For more detailed setup instructions, please see further down this document. -[1] VoIP currently in development - About Matrix ============ @@ -50,15 +48,15 @@ which handle: - Creating and managing fully distributed chat rooms with no single points of control or failure - - Eventually-consistent cryptographically secure[2] synchronisation of room + - Eventually-consistent cryptographically secure[1] synchronisation of room state across a global open network of federated servers and services - Sending and receiving extensible messages in a room with (optional) - end-to-end encryption[3] + end-to-end encryption[2] - Inviting, joining, leaving, kicking, banning room members - Managing user accounts (registration, login, logout) - Using 3rd Party IDs (3PIDs) such as email addresses, phone numbers, Facebook accounts to authenticate, identify and discover users on Matrix. - - Placing 1:1 VoIP and Video calls (in development) + - Placing 1:1 VoIP and Video calls These APIs are intended to be implemented on a wide range of servers, services and clients, letting developers build messaging and VoIP functionality on top of @@ -92,9 +90,9 @@ https://github.com/matrix-org/synapse/issues or at matrix@matrix.org. Thanks for trying Matrix! -[2] Cryptographic signing of messages isn't turned on yet +[1] Cryptographic signing of messages isn't turned on yet -[3] End-to-end encryption is currently in development +[2] End-to-end encryption is currently in development Homeserver Installation diff --git a/UPGRADE.rst b/UPGRADE.rst index 99d58ab64f..da2a7a0a21 100644 --- a/UPGRADE.rst +++ b/UPGRADE.rst @@ -1,11 +1,6 @@ Upgrading to v0.2.0 =================== -To upgrade the database schema, run:: - - ./database-prepare-for-0.2.0.sh "<database>.db" - - The home server now requires setting up of SSL config before it can run. To automatically generate default config use:: diff --git a/VERSION b/VERSION index 0ea3a944b3..0c62199f16 100644 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -0.2.0 +0.2.1 diff --git a/cmdclient/console.py b/cmdclient/console.py index 7678b5e352..2e6b026762 100755 --- a/cmdclient/console.py +++ b/cmdclient/console.py @@ -1,6 +1,6 @@ #!/usr/bin/env python -# Copyright 2014 matrix.org +# Copyright 2014 OpenMarket Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. @@ -88,6 +88,8 @@ class SynapseCmd(cmd.Cmd): return False def _domain(self): + if "user" not in self.config or not self.config["user"]: + return None return self.config["user"].split(":")[1] def do_config(self, line): @@ -191,10 +193,12 @@ class SynapseCmd(cmd.Cmd): p = getpass.getpass("Enter your password: ") user = args["user_id"] if self._is_on("complete_usernames") and not user.startswith("@"): - user = "@" + user + ":" + self._domain() - + domain = self._domain() + if domain: + user = "@" + user + ":" + domain + reactor.callFromThread(self._do_login, user, p) - print " got %s " % p + #print " got %s " % p except Exception as e: print e @@ -312,7 +316,7 @@ class SynapseCmd(cmd.Cmd): try: args = self._parse(line, ["roomname"], force_keys=True) path = "/join/%s" % urllib.quote(args["roomname"]) - reactor.callFromThread(self._run_and_pprint, "PUT", path, {}) + reactor.callFromThread(self._run_and_pprint, "POST", path, {}) except Exception as e: print e @@ -700,7 +704,7 @@ def main(server_url, identity_server_url, username, token, config_path): if __name__ == '__main__': parser = argparse.ArgumentParser("Starts a synapse client.") parser.add_argument( - "-s", "--server", dest="server", default="http://localhost:8080", + "-s", "--server", dest="server", default="http://localhost:8008", help="The URL of the home server to talk to.") parser.add_argument( "-i", "--identity-server", dest="identityserver", default="http://localhost:8090", diff --git a/cmdclient/http.py b/cmdclient/http.py index 9de6be9b72..869f782ec1 100644 --- a/cmdclient/http.py +++ b/cmdclient/http.py @@ -1,5 +1,5 @@ # -*- coding: utf-8 -*- -# Copyright 2014 matrix.org +# Copyright 2014 OpenMarket Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. diff --git a/database-prepare-for-0.2.0.sh b/database-prepare-for-0.2.0.sh deleted file mode 100755 index e90171010b..0000000000 --- a/database-prepare-for-0.2.0.sh +++ /dev/null @@ -1,10 +0,0 @@ -#!/bin/bash - -# This is will prepare a synapse database for running with v0.2.0 of synapse. - -set -e - -cp "$1" "$1.bak" - -sqlite3 "$1" < "synapse/storage/schema/im.sql" -sqlite3 "$1" <<< "PRAGMA user_version = 2;" diff --git a/docs/client-server/specification.rst b/docs/client-server/OLD_specification.rst index 2f6645ceb9..47fba5eeac 100644 --- a/docs/client-server/specification.rst +++ b/docs/client-server/OLD_specification.rst @@ -2,6 +2,20 @@ Matrix Client-Server API ======================== + +.. WARNING:: + This specification is old. Please see /docs/specification.rst instead. + + + + + + + + + + + The following specification outlines how a client can send and receive data from a home server. diff --git a/docs/client-server/howto.rst b/docs/client-server/howto.rst index 3660c73d36..c02ea8d897 100644 --- a/docs/client-server/howto.rst +++ b/docs/client-server/howto.rst @@ -1,9 +1,8 @@ -TODO(kegan): Tweak joinalias API keys/path? Event stream historical > live needs -a token (currently doesn't). im/sync responses include outdated event formats -(room membership change messages). Room config (specifically: message history, -public rooms). /register seems super simplistic compared to /login, maybe it -would be better if /register used the same technique as /login? /register should -be "user" not "user_id". +.. TODO kegan + Room config (specifically: message history, + public rooms). /register seems super simplistic compared to /login, maybe it + would be better if /register used the same technique as /login? /register should + be "user" not "user_id". How to use the client-server API @@ -15,7 +14,7 @@ implementation, there may be variations in relation to registering/logging in which are not covered in extensive detail in this guide. If you haven't already, get a home server up and running on -``http://localhost:8080``. +``http://localhost:8008``. Accounts @@ -23,14 +22,16 @@ Accounts Before you can send and receive messages, you must **register** for an account. If you already have an account, you must **login** into it. -**Try out the fiddle: http://jsfiddle.net/jrf1h02d/** +`Try out the fiddle`__ + +.. __: http://jsfiddle.net/4q2jyxng/ Registration ------------ The aim of registration is to get a user ID and access token which you will need when accessing other APIs:: - curl -XPOST -d '{"user_id":"example", "password":"wordpass"}' "http://localhost:8080/_matrix/client/api/v1/register" + curl -XPOST -d '{"user_id":"example", "password":"wordpass"}' "http://localhost:8008/_matrix/client/api/v1/register" { "access_token": "QGV4YW1wbGU6bG9jYWxob3N0.AqdSzFmFYrLrTmteXc", @@ -51,13 +52,17 @@ Login ----- The aim when logging in is to get an access token for your existing user ID:: - curl -XGET "http://localhost:8080/_matrix/client/api/v1/login" + curl -XGET "http://localhost:8008/_matrix/client/api/v1/login" { - "type": "m.login.password" + "flows": [ + { + "type": "m.login.password" + } + ] } - curl -XPOST -d '{"type":"m.login.password", "user":"example", "password":"wordpass"}' "http://localhost:8080/_matrix/client/api/v1/login" + curl -XPOST -d '{"type":"m.login.password", "user":"example", "password":"wordpass"}' "http://localhost:8008/_matrix/client/api/v1/login" { "access_token": "QGV4YW1wbGU6bG9jYWxob3N0.vRDLTgxefmKWQEtgGd", @@ -80,14 +85,16 @@ Communicating In order to communicate with another user, you must **create a room** with that user and **send a message** to that room. -**Try out the fiddle: http://jsfiddle.net/jnwqcshc/** +`Try out the fiddle`__ + +.. __: http://jsfiddle.net/zL3zto9g/ Creating a room --------------- If you want to send a message to someone, you have to be in a room with them. To create a room:: - curl -XPOST -d '{"room_alias_name":"tutorial"}' "http://localhost:8080/_matrix/client/api/v1/rooms?access_token=QGV4YW1wbGU6bG9jYWxob3N0.vRDLTgxefmKWQEtgGd" + curl -XPOST -d '{"room_alias_name":"tutorial"}' "http://localhost:8008/_matrix/client/api/v1/createRoom?access_token=YOUR_ACCESS_TOKEN" { "room_alias": "#tutorial:localhost", @@ -98,20 +105,27 @@ The "room alias" is a human-readable string which can be shared with other users so they can join a room, rather than the room ID which is a randomly generated string. You can have multiple room aliases per room. -TODO(kegan): How to add/remove aliases from an existing room. +.. TODO(kegan) + How to add/remove aliases from an existing room. Sending messages ---------------- You can now send messages to this room:: - curl -XPUT -d '{"msgtype":"m.text", "body":"hello"}' "http://localhost:8080/_matrix/client/api/v1/rooms/%21CvcvRuDYDzTOzfKKgh:localhost/messages/%40example%3Alocalhost/msgid1?access_token=QGV4YW1wbGU6bG9jYWxob3N0.vRDLTgxefmKWQEtgGd" + curl -XPOST -d '{"msgtype":"m.text", "body":"hello"}' "http://localhost:8008/_matrix/client/api/v1/rooms/%21CvcvRuDYDzTOzfKKgh%3Alocalhost/send/m.room.message?access_token=YOUR_ACCESS_TOKEN" + + { + "event_id": "YUwRidLecu" + } + +The event ID returned is a unique ID which identifies this message. NB: There are no limitations to the types of messages which can be exchanged. -The only requirement is that ``"msgtype"`` is specified. - -NB: Depending on the room config, users who join the room may be able to see -message history from before they joined. +The only requirement is that ``"msgtype"`` is specified. The Matrix +specification outlines the following standard types: ``m.text``, ``m.image``, +``m.audio``, ``m.video``, ``m.location``, ``m.emote``. See the specification for +more information on these types. Users and rooms =============== @@ -121,33 +135,34 @@ these rules may specify if you require an **invitation** from someone already in the room in order to **join the room**. In addition, you may also be able to join a room **via a room alias** if one was set up. -**Try out the fiddle: http://jsfiddle.net/og1xokcr/** +`Try out the fiddle`__ + +.. __: http://jsfiddle.net/7fhotf1b/ Inviting a user to a room ------------------------- You can directly invite a user to a room like so:: - curl -XPUT -d '{"membership":"invite"}' "http://localhost:8080/_matrix/client/api/v1/rooms/%21CvcvRuDYDzTOzfKKgh:localhost/members/%40myfriend%3Alocalhost/state?access_token=QGV4YW1wbGU6bG9jYWxob3N0.vRDLTgxefmKWQEtgGd" + curl -XPOST -d '{"user_id":"@myfriend:localhost"}' "http://localhost:8008/_matrix/client/api/v1/rooms/%21CvcvRuDYDzTOzfKKgh%3Alocalhost/invite?access_token=YOUR_ACCESS_TOKEN" This informs ``@myfriend:localhost`` of the room ID ``!CvcvRuDYDzTOzfKKgh:localhost`` and allows them to join the room. Joining a room via an invite ---------------------------- -If you receive an invite, you can join the room by changing the membership to -join:: +If you receive an invite, you can join the room:: - curl -XPUT -d '{"membership":"join"}' "http://localhost:8080/_matrix/client/api/v1/rooms/%21CvcvRuDYDzTOzfKKgh:localhost/members/%40myfriend%3Alocalhost/state?access_token=QG15ZnJpZW5kOmxvY2FsaG9zdA...XKuGdVsovHmwMyDDvK" + curl -XPOST -d '{}' "http://localhost:8008/_matrix/client/api/v1/rooms/%21CvcvRuDYDzTOzfKKgh%3Alocalhost/join?access_token=YOUR_ACCESS_TOKEN" NB: Only the person invited (``@myfriend:localhost``) can change the membership -state to ``"join"``. +state to ``"join"``. Repeatedly joining a room does nothing. Joining a room via an alias --------------------------- Alternatively, if you know the room alias for this room and the room config allows it, you can directly join a room via the alias:: - curl -XPUT -d '{}' "http://localhost:8080/_matrix/client/api/v1/join/%23tutorial%3Alocalhost?access_token=QG15ZnJpZW5kOmxvY2FsaG9zdA...XKuGdVsovHmwMyDDvK" + curl -XPOST -d '{}' "http://localhost:8008/_matrix/client/api/v1/join/%23tutorial%3Alocalhost?access_token=YOUR_ACCESS_TOKEN" { "room_id": "!CvcvRuDYDzTOzfKKgh:localhost" @@ -166,128 +181,444 @@ An event is some interesting piece of data that a client may be interested in. It can be a message in a room, a room invite, etc. There are many different ways of getting events, depending on what the client already knows. -**Try out the fiddle: http://jsfiddle.net/5uk4dqe2/** +`Try out the fiddle`__ + +.. __: http://jsfiddle.net/vw11mg37/ Getting all state ----------------- If the client doesn't know any information on the rooms the user is invited/joined on, they can get all the user's state for all rooms:: - curl -XGET "http://localhost:8080/_matrix/client/api/v1/im/sync?access_token=QG15ZnJpZW5kOmxvY2FsaG9zdA...XKuGdVsovHmwMyDDvK" + curl -XGET "http://localhost:8008/_matrix/client/api/v1/initialSync?access_token=YOUR_ACCESS_TOKEN" - [ - { - "membership": "join", - "messages": { - "chunk": [ + { + "end": "s39_18_0", + "presence": [ + { + "content": { + "last_active_ago": 1061436, + "user_id": "@example:localhost" + }, + "type": "m.presence" + } + ], + "rooms": [ + { + "membership": "join", + "messages": { + "chunk": [ + { + "content": { + "@example:localhost": 10, + "default": 0 + }, + "event_id": "wAumPSTsWF", + "required_power_level": 10, + "room_id": "!MkDbyRqnvTYnoxjLYx:localhost", + "state_key": "", + "ts": 1409665585188, + "type": "m.room.power_levels", + "user_id": "@example:localhost" + }, + { + "content": { + "join_rule": "public" + }, + "event_id": "jrLVqKHKiI", + "required_power_level": 10, + "room_id": "!MkDbyRqnvTYnoxjLYx:localhost", + "state_key": "", + "ts": 1409665585188, + "type": "m.room.join_rules", + "user_id": "@example:localhost" + }, + { + "content": { + "level": 10 + }, + "event_id": "WpmTgsNWUZ", + "required_power_level": 10, + "room_id": "!MkDbyRqnvTYnoxjLYx:localhost", + "state_key": "", + "ts": 1409665585188, + "type": "m.room.add_state_level", + "user_id": "@example:localhost" + }, + { + "content": { + "level": 0 + }, + "event_id": "qUMBJyKsTQ", + "required_power_level": 10, + "room_id": "!MkDbyRqnvTYnoxjLYx:localhost", + "state_key": "", + "ts": 1409665585188, + "type": "m.room.send_event_level", + "user_id": "@example:localhost" + }, + { + "content": { + "ban_level": 5, + "kick_level": 5 + }, + "event_id": "YAaDmKvoUW", + "required_power_level": 10, + "room_id": "!MkDbyRqnvTYnoxjLYx:localhost", + "state_key": "", + "ts": 1409665585188, + "type": "m.room.ops_levels", + "user_id": "@example:localhost" + }, + { + "content": { + "avatar_url": null, + "displayname": null, + "membership": "join" + }, + "event_id": "RJbPMtCutf", + "membership": "join", + "room_id": "!MkDbyRqnvTYnoxjLYx:localhost", + "state_key": "@example:localhost", + "ts": 1409665586730, + "type": "m.room.member", + "user_id": "@example:localhost" + }, + { + "content": { + "body": "hello", + "hsob_ts": 1409665660439, + "msgtype": "m.text" + }, + "event_id": "YUwRidLecu", + "room_id": "!MkDbyRqnvTYnoxjLYx:localhost", + "ts": 1409665660439, + "type": "m.room.message", + "user_id": "@example:localhost" + }, + { + "content": { + "membership": "invite" + }, + "event_id": "YjNuBKnPsb", + "membership": "invite", + "room_id": "!MkDbyRqnvTYnoxjLYx:localhost", + "state_key": "@myfriend:localhost", + "ts": 1409666426819, + "type": "m.room.member", + "user_id": "@example:localhost" + }, + { + "content": { + "avatar_url": null, + "displayname": null, + "membership": "join", + "prev": "join" + }, + "event_id": "KWwdDjNZnm", + "membership": "join", + "room_id": "!MkDbyRqnvTYnoxjLYx:localhost", + "state_key": "@example:localhost", + "ts": 1409666551582, + "type": "m.room.member", + "user_id": "@example:localhost" + }, + { + "content": { + "avatar_url": null, + "displayname": null, + "membership": "join" + }, + "event_id": "JFLVteSvQc", + "membership": "join", + "room_id": "!MkDbyRqnvTYnoxjLYx:localhost", + "state_key": "@example:localhost", + "ts": 1409666587265, + "type": "m.room.member", + "user_id": "@example:localhost" + } + ], + "end": "s39_18_0", + "start": "t1-11_18_0" + }, + "room_id": "!MkDbyRqnvTYnoxjLYx:localhost", + "state": [ { "content": { - "body": "@example:localhost joined the room.", - "hsob_ts": 1408444664249, - "membership": "join", - "membership_source": "@example:localhost", - "membership_target": "@example:localhost", - "msgtype": "m.text" + "creator": "@example:localhost" }, - "event_id": "lZjmmlrEvo", - "msg_id": "m1408444664249", - "room_id": "!CvcvRuDYDzTOzfKKgh:localhost", - "type": "m.room.message", - "user_id": "_homeserver_" + "event_id": "dMUoqVTZca", + "required_power_level": 10, + "room_id": "!MkDbyRqnvTYnoxjLYx:localhost", + "state_key": "", + "ts": 1409665585188, + "type": "m.room.create", + "user_id": "@example:localhost" }, { "content": { - "body": "hello", - "hsob_ts": 1408445405672, - "msgtype": "m.text" + "@example:localhost": 10, + "default": 0 }, - "event_id": "BiBJqamISg", - "msg_id": "msgid1", - "room_id": "!CvcvRuDYDzTOzfKKgh:localhost", - "type": "m.room.message", + "event_id": "wAumPSTsWF", + "required_power_level": 10, + "room_id": "!MkDbyRqnvTYnoxjLYx:localhost", + "state_key": "", + "ts": 1409665585188, + "type": "m.room.power_levels", "user_id": "@example:localhost" }, - [...] { "content": { - "body": "@myfriend:localhost joined the room.", - "hsob_ts": 1408446501661, - "membership": "join", - "membership_source": "@myfriend:localhost", - "membership_target": "@myfriend:localhost", - "msgtype": "m.text" + "join_rule": "public" + }, + "event_id": "jrLVqKHKiI", + "required_power_level": 10, + "room_id": "!MkDbyRqnvTYnoxjLYx:localhost", + "state_key": "", + "ts": 1409665585188, + "type": "m.room.join_rules", + "user_id": "@example:localhost" + }, + { + "content": { + "level": 10 }, - "event_id": "IMmXbOzFAa", - "msg_id": "m1408446501661", - "room_id": "!CvcvRuDYDzTOzfKKgh:localhost", - "type": "m.room.message", - "user_id": "_homeserver_" + "event_id": "WpmTgsNWUZ", + "required_power_level": 10, + "room_id": "!MkDbyRqnvTYnoxjLYx:localhost", + "state_key": "", + "ts": 1409665585188, + "type": "m.room.add_state_level", + "user_id": "@example:localhost" + }, + { + "content": { + "level": 0 + }, + "event_id": "qUMBJyKsTQ", + "required_power_level": 10, + "room_id": "!MkDbyRqnvTYnoxjLYx:localhost", + "state_key": "", + "ts": 1409665585188, + "type": "m.room.send_event_level", + "user_id": "@example:localhost" + }, + { + "content": { + "ban_level": 5, + "kick_level": 5 + }, + "event_id": "YAaDmKvoUW", + "required_power_level": 10, + "room_id": "!MkDbyRqnvTYnoxjLYx:localhost", + "state_key": "", + "ts": 1409665585188, + "type": "m.room.ops_levels", + "user_id": "@example:localhost" + }, + { + "content": { + "membership": "invite" + }, + "event_id": "YjNuBKnPsb", + "membership": "invite", + "room_id": "!MkDbyRqnvTYnoxjLYx:localhost", + "state_key": "@myfriend:localhost", + "ts": 1409666426819, + "type": "m.room.member", + "user_id": "@example:localhost" + }, + { + "content": { + "avatar_url": null, + "displayname": null, + "membership": "join" + }, + "event_id": "JFLVteSvQc", + "membership": "join", + "room_id": "!MkDbyRqnvTYnoxjLYx:localhost", + "state_key": "@example:localhost", + "ts": 1409666587265, + "type": "m.room.member", + "user_id": "@example:localhost" } - ], - "end": "20", - "start": "0" - }, - "room_id": "!CvcvRuDYDzTOzfKKgh:localhost" - } - ] + ] + } + ] + } -This returns all the room IDs of rooms the user is invited/joined on, as well as -all of the messages and feedback for these rooms. This can be a LOT of data. You -may just want the most recent message for each room. This can be achieved by -applying pagination stream parameters to this request:: +This returns all the room information the user is invited/joined on, as well as +all of the presences relevant for these rooms. This can be a LOT of data. You +may just want the most recent event for each room. This can be achieved by +applying query parameters to ``limit`` this request:: - curl -XGET "http://localhost:8080/_matrix/client/api/v1/im/sync?access_token=QG15ZnJpZW5kOmxvY2FsaG9zdA...XKuGdVsovHmwMyDDvK&from=END&to=START&limit=1" + curl -XGET "http://localhost:8008/_matrix/client/api/v1/initialSync?limit=1&access_token=YOUR_ACCESS_TOKEN" - [ - { - "membership": "join", - "messages": { - "chunk": [ + { + "end": "s39_18_0", + "presence": [ + { + "content": { + "last_active_ago": 1279484, + "user_id": "@example:localhost" + }, + "type": "m.presence" + } + ], + "rooms": [ + { + "membership": "join", + "messages": { + "chunk": [ + { + "content": { + "avatar_url": null, + "displayname": null, + "membership": "join" + }, + "event_id": "JFLVteSvQc", + "membership": "join", + "room_id": "!MkDbyRqnvTYnoxjLYx:localhost", + "state_key": "@example:localhost", + "ts": 1409666587265, + "type": "m.room.member", + "user_id": "@example:localhost" + } + ], + "end": "s39_18_0", + "start": "t10-30_18_0" + }, + "room_id": "!MkDbyRqnvTYnoxjLYx:localhost", + "state": [ { "content": { - "body": "@myfriend:localhost joined the room.", - "hsob_ts": 1408446501661, - "membership": "join", - "membership_source": "@myfriend:localhost", - "membership_target": "@myfriend:localhost", - "msgtype": "m.text" + "creator": "@example:localhost" + }, + "event_id": "dMUoqVTZca", + "required_power_level": 10, + "room_id": "!MkDbyRqnvTYnoxjLYx:localhost", + "state_key": "", + "ts": 1409665585188, + "type": "m.room.create", + "user_id": "@example:localhost" + }, + { + "content": { + "@example:localhost": 10, + "default": 0 + }, + "event_id": "wAumPSTsWF", + "required_power_level": 10, + "room_id": "!MkDbyRqnvTYnoxjLYx:localhost", + "state_key": "", + "ts": 1409665585188, + "type": "m.room.power_levels", + "user_id": "@example:localhost" + }, + { + "content": { + "join_rule": "public" + }, + "event_id": "jrLVqKHKiI", + "required_power_level": 10, + "room_id": "!MkDbyRqnvTYnoxjLYx:localhost", + "state_key": "", + "ts": 1409665585188, + "type": "m.room.join_rules", + "user_id": "@example:localhost" + }, + { + "content": { + "level": 10 }, - "event_id": "IMmXbOzFAa", - "msg_id": "m1408446501661", - "room_id": "!CvcvRuDYDzTOzfKKgh:localhost", - "type": "m.room.message", - "user_id": "_homeserver_" + "event_id": "WpmTgsNWUZ", + "required_power_level": 10, + "room_id": "!MkDbyRqnvTYnoxjLYx:localhost", + "state_key": "", + "ts": 1409665585188, + "type": "m.room.add_state_level", + "user_id": "@example:localhost" + }, + { + "content": { + "level": 0 + }, + "event_id": "qUMBJyKsTQ", + "required_power_level": 10, + "room_id": "!MkDbyRqnvTYnoxjLYx:localhost", + "state_key": "", + "ts": 1409665585188, + "type": "m.room.send_event_level", + "user_id": "@example:localhost" + }, + { + "content": { + "ban_level": 5, + "kick_level": 5 + }, + "event_id": "YAaDmKvoUW", + "required_power_level": 10, + "room_id": "!MkDbyRqnvTYnoxjLYx:localhost", + "state_key": "", + "ts": 1409665585188, + "type": "m.room.ops_levels", + "user_id": "@example:localhost" + }, + { + "content": { + "membership": "invite" + }, + "event_id": "YjNuBKnPsb", + "membership": "invite", + "room_id": "!MkDbyRqnvTYnoxjLYx:localhost", + "state_key": "@myfriend:localhost", + "ts": 1409666426819, + "type": "m.room.member", + "user_id": "@example:localhost" + }, + { + "content": { + "avatar_url": null, + "displayname": null, + "membership": "join" + }, + "event_id": "JFLVteSvQc", + "membership": "join", + "room_id": "!MkDbyRqnvTYnoxjLYx:localhost", + "state_key": "@example:localhost", + "ts": 1409666587265, + "type": "m.room.member", + "user_id": "@example:localhost" } - ], - "end": "20", - "start": "21" - }, - "room_id": "!CvcvRuDYDzTOzfKKgh:localhost" - } - ] + ] + } + ] + } Getting live state ------------------ Once you know which rooms the client has previously interacted with, you need to listen for incoming events. This can be done like so:: - curl -XGET "http://localhost:8080/_matrix/client/api/v1/events?access_token=QG15ZnJpZW5kOmxvY2FsaG9zdA...XKuGdVsovHmwMyDDvK&from=END" + curl -XGET "http://localhost:8008/_matrix/client/api/v1/events?access_token=YOUR_ACCESS_TOKEN" { "chunk": [], - "end": "215", - "start": "215" + "end": "s39_18_0", + "start": "s39_18_0" } This will block waiting for an incoming event, timing out after several seconds. Even if there are no new events (as in the example above), there will be some pagination stream response keys. The client should make subsequent requests -using the value of the ``"end"`` key (in this case ``215``) as the ``from`` -query parameter. This value should be stored so when the client reopens your app -after a period of inactivity, you can resume from where you got up to in the -event stream. If it has been a long period of inactivity, there may be LOTS of -events waiting for the user. In this case, you may wish to get all state instead -and then resume getting live state from a newer end token. +using the value of the ``"end"`` key (in this case ``s39_18_0``) as the ``from`` +query parameter e.g. ``http://localhost:8008/_matrix/client/api/v1/events?access +_token=YOUR_ACCESS_TOKEN&from=s39_18_0``. This value should be stored so when the +client reopens your app after a period of inactivity, you can resume from where +you got up to in the event stream. If it has been a long period of inactivity, +there may be LOTS of events waiting for the user. In this case, you may wish to +get all state instead and then resume getting live state from a newer end token. NB: The timeout can be changed by adding a ``timeout`` query parameter, which is in milliseconds. A timeout of 0 will not block. @@ -300,4 +631,6 @@ creating and joining rooms, sending messages, getting member lists and getting historical messages for a room. This covers most functionality of a messaging application. -**Try out the fiddle: http://jsfiddle.net/L8r3o1wr/** +`Try out the fiddle`__ + +.. __: http://jsfiddle.net/uztL3yme/ diff --git a/docs/client-server/swagger_matrix/api-docs-directory b/docs/client-server/swagger_matrix/api-docs-directory index 98109a0fbc..ce12be8c96 100644 --- a/docs/client-server/swagger_matrix/api-docs-directory +++ b/docs/client-server/swagger_matrix/api-docs-directory @@ -1,7 +1,7 @@ { "apiVersion": "1.0.0", "swaggerVersion": "1.2", - "basePath": "http://localhost:8080/_matrix/client/api/v1", + "basePath": "http://localhost:8008/_matrix/client/api/v1", "resourcePath": "/directory", "produces": [ "application/json" @@ -13,6 +13,7 @@ { "method": "GET", "summary": "Get the room ID corresponding to this room alias.", + "notes": "Volatile: This API is likely to change.", "type": "DirectoryResponse", "nickname": "get_room_id_for_alias", "parameters": [ @@ -28,6 +29,7 @@ { "method": "PUT", "summary": "Create a new mapping from room alias to room ID.", + "notes": "Volatile: This API is likely to change.", "type": "void", "nickname": "add_room_alias", "parameters": [ diff --git a/docs/client-server/swagger_matrix/api-docs-events b/docs/client-server/swagger_matrix/api-docs-events index e5dd3a6113..1bdb9b034a 100644 --- a/docs/client-server/swagger_matrix/api-docs-events +++ b/docs/client-server/swagger_matrix/api-docs-events @@ -1,7 +1,7 @@ { "apiVersion": "1.0.0", "swaggerVersion": "1.2", - "basePath": "http://localhost:8080/_matrix/client/api/v1", + "basePath": "http://localhost:8008/_matrix/client/api/v1", "resourcePath": "/events", "produces": [ "application/json" diff --git a/docs/client-server/swagger_matrix/api-docs-login b/docs/client-server/swagger_matrix/api-docs-login index 8cc598b3c1..d6f8d84f29 100644 --- a/docs/client-server/swagger_matrix/api-docs-login +++ b/docs/client-server/swagger_matrix/api-docs-login @@ -8,7 +8,7 @@ "nickname": "get_login_info", "notes": "All login stages MUST be mentioned if there is >1 login type.", "summary": "Get the login mechanism to use when logging in.", - "type": "LoginInfo" + "type": "LoginFlows" }, { "method": "POST", @@ -40,17 +40,31 @@ "path": "/login" } ], - "basePath": "http://localhost:8080/_matrix/client/api/v1", + "basePath": "http://localhost:8008/_matrix/client/api/v1", "consumes": [ "application/json" ], "models": { + "LoginFlows": { + "id": "LoginFlows", + "properties": { + "flows": { + "description": "A list of valid login flows.", + "type": "array", + "items": { + "$ref": "LoginInfo" + } + } + } + }, "LoginInfo": { "id": "LoginInfo", "properties": { "stages": { "description": "Multi-stage login only: An array of all the login types required to login.", - "format": "string", + "items": { + "$ref": "string" + }, "type": "array" }, "type": { @@ -65,6 +79,10 @@ "access_token": { "description": "The access token for this user's login if this is the final stage of the login process.", "type": "string" + }, + "user_id": { + "description": "The user's fully-qualified user ID.", + "type": "string" }, "next": { "description": "Multi-stage login only: The next login type to submit.", diff --git a/docs/client-server/swagger_matrix/api-docs-presence b/docs/client-server/swagger_matrix/api-docs-presence index d52ce2164a..6b22446024 100644 --- a/docs/client-server/swagger_matrix/api-docs-presence +++ b/docs/client-server/swagger_matrix/api-docs-presence @@ -1,7 +1,7 @@ { "apiVersion": "1.0.0", "swaggerVersion": "1.2", - "basePath": "http://localhost:8080/_matrix/client/api/v1", + "basePath": "http://localhost:8008/_matrix/client/api/v1", "resourcePath": "/presence", "produces": [ "application/json" @@ -106,7 +106,7 @@ "PresenceUpdate": { "id": "PresenceUpdate", "properties": { - "state": { + "presence": { "type": "string", "description": "Enum: The presence state.", "enum": [ @@ -128,10 +128,10 @@ "Presence": { "id": "Presence", "properties": { - "mtime_age": { + "last_active_ago": { "type": "integer", "format": "int64", - "description": "The last time this user's presence state changed, in milliseconds." + "description": "The last time this user performed an action on their home server." }, "user_id": { "type": "string", diff --git a/docs/client-server/swagger_matrix/api-docs-profile b/docs/client-server/swagger_matrix/api-docs-profile index 188259fa3d..d2fccaa67d 100644 --- a/docs/client-server/swagger_matrix/api-docs-profile +++ b/docs/client-server/swagger_matrix/api-docs-profile @@ -1,7 +1,7 @@ { "apiVersion": "1.0.0", "swaggerVersion": "1.2", - "basePath": "http://localhost:8080/_matrix/client/api/v1", + "basePath": "http://localhost:8008/_matrix/client/api/v1", "resourcePath": "/profile", "produces": [ "application/json" diff --git a/docs/client-server/swagger_matrix/api-docs-registration b/docs/client-server/swagger_matrix/api-docs-registration index 2048aec1d2..f4669ea2f0 100644 --- a/docs/client-server/swagger_matrix/api-docs-registration +++ b/docs/client-server/swagger_matrix/api-docs-registration @@ -37,7 +37,7 @@ "path": "/register" } ], - "basePath": "http://localhost:8080/_matrix/client/api/v1", + "basePath": "http://localhost:8008/_matrix/client/api/v1", "consumes": [ "application/json" ], @@ -52,6 +52,10 @@ "user_id": { "description": "The fully-qualified user ID.", "type": "string" + }, + "home_server": { + "description": "The name of the home server.", + "type": "string" } } }, diff --git a/docs/client-server/swagger_matrix/api-docs-rooms b/docs/client-server/swagger_matrix/api-docs-rooms index 0a8bb3c2a5..0e1fa452a2 100644 --- a/docs/client-server/swagger_matrix/api-docs-rooms +++ b/docs/client-server/swagger_matrix/api-docs-rooms @@ -1,7 +1,7 @@ { "apiVersion": "1.0.0", "swaggerVersion": "1.2", - "basePath": "http://localhost:8080/_matrix/client/api/v1", + "basePath": "http://localhost:8008/_matrix/client/api/v1", "resourcePath": "/rooms", "produces": [ "application/json" @@ -181,6 +181,59 @@ ] }, { + "path": "/rooms/{roomId}/state/m.room.name", + "operations": [ + { + "method": "PUT", + "summary": "Set the name of this room.", + "notes": "Set the name of this room.", + "type": "void", + "nickname": "set_room_name", + "consumes": [ + "application/json" + ], + "parameters": [ + { + "name": "body", + "description": "The name contents", + "required": true, + "type": "RoomName", + "paramType": "body" + }, + { + "name": "roomId", + "description": "The room to set the name of.", + "required": true, + "type": "string", + "paramType": "path" + } + ] + }, + { + "method": "GET", + "summary": "Get the room's name.", + "notes": "", + "type": "RoomName", + "nickname": "get_room_name", + "parameters": [ + { + "name": "roomId", + "description": "The room to get the name of.", + "required": true, + "type": "string", + "paramType": "path" + } + ], + "responseMessages": [ + { + "code": 404, + "message": "Name not found." + } + ] + } + ] + }, + { "path": "/rooms/{roomId}/send/m.room.message.feedback", "operations": [ { @@ -267,6 +320,12 @@ "required": true, "type": "string", "paramType": "path" + }, + { + "name": "body", + "required": true, + "type": "JoinRequest", + "paramType": "body" } ] } @@ -291,6 +350,12 @@ "required": true, "type": "string", "paramType": "path" + }, + { + "name": "body", + "required": true, + "type": "LeaveRequest", + "paramType": "body" } ] } @@ -424,10 +489,10 @@ "path": "/join/{roomAliasOrId}", "operations": [ { - "method": "PUT", + "method": "POST", "summary": "Join a room via a room alias or room ID.", "notes": "Join a room via a room alias or room ID.", - "type": "RoomInfo", + "type": "JoinRoomInfo", "nickname": "join", "consumes": [ "application/json" @@ -574,7 +639,7 @@ { "method": "GET", "summary": "Get a list of all the current state events for this room.", - "notes": "Get a list of all the current state events for this room.", + "notes": "NOT YET IMPLEMENTED.", "type": "array", "items": { "$ref": "Event" @@ -598,7 +663,7 @@ { "method": "GET", "summary": "Get all the current information for this room, including messages and state events.", - "notes": "Get all the current information for this room, including messages and state events.", + "notes": "NOT YET IMPLEMENTED.", "type": "InitialSyncRoomData", "nickname": "get_room_sync_data", "parameters": [ @@ -624,6 +689,15 @@ } } }, + "RoomName": { + "id": "RoomName", + "properties": { + "name": { + "type": "string", + "description": "The human-readable name for the room. Can contain spaces." + } + } + }, "Message": { "id": "Message", "properties": { @@ -640,6 +714,16 @@ "Feedback": { "id": "Feedback", "properties": { + "target_event_id": { + "type": "string", + "description": "The event ID being acknowledged.", + "required": true + }, + "type": { + "type": "string", + "description": "The type of feedback. Either 'delivered' or 'read'.", + "required": true + } } }, "Member": { @@ -652,7 +736,7 @@ "invite", "join", "leave", - "knock" + "ban" ] } } @@ -672,6 +756,16 @@ } } }, + "JoinRoomInfo": { + "id": "JoinRoomInfo", + "properties": { + "room_id": { + "type": "string", + "description": "The room ID joined, if joined via a room alias only.", + "required": true + } + } + }, "RoomConfig": { "id": "RoomConfig", "properties": { @@ -830,6 +924,14 @@ } } }, + "JoinRequest": { + "id": "JoinRequest", + "properties": {} + }, + "LeaveRequest": { + "id": "LeaveRequest", + "properties": {} + }, "BanRequest": { "id": "BanRequest", "properties": { diff --git a/docs/server-server/security-threat-model.rst b/docs/server-server/security-threat-model.rst deleted file mode 100644 index cf0430e43d..0000000000 --- a/docs/server-server/security-threat-model.rst +++ /dev/null @@ -1,141 +0,0 @@ -Overview -======== - -Scope ------ - -This document considers threats specific to the server to server federation -synapse protocol. - - -Attacker --------- - -It is assumed that the attacker can see and manipulate all network traffic -between any of the servers and may be in control of one or more homeservers -participating in the federation protocol. - -Threat Model -============ - -Denial of Service ------------------ - -The attacker could attempt to prevent delivery of messages to or from the -victim in order to: - - * Disrupt service or marketing campaign of a commercial competitor. - * Censor a discussion or censor a participant in a discussion. - * Perform general vandalism. - -Threat: Resource Exhaustion -~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -An attacker could cause the victims server to exhaust a particular resource -(e.g. open TCP connections, CPU, memory, disk storage) - -Threat: Unrecoverable Consistency Violations -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -An attacker could send messages which created an unrecoverable "split-brain" -state in the cluster such that the victim's servers could no longer dervive a -consistent view of the chatroom state. - -Threat: Bad History -~~~~~~~~~~~~~~~~~~~ - -An attacker could convince the victim to accept invalid messages which the -victim would then include in their view of the chatroom history. Other servers -in the chatroom would reject the invalid messages and potentially reject the -victims messages as well since they depended on the invalid messages. - -Threat: Block Network Traffic -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -An attacker could try to firewall traffic between the victim's server and some -or all of the other servers in the chatroom. - -Threat: High Volume of Messages -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -An attacker could send large volumes of messages to a chatroom with the victim -making the chatroom unusable. - -Threat: Banning users without necessary authorisation -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -An attacker could attempt to ban a user from a chatroom with the necessary -authorisation. - -Spoofing --------- - -An attacker could try to send a message claiming to be from the victim without -the victim having sent the message in order to: - - * Impersonate the victim while performing illict activity. - * Obtain privileges of the victim. - -Threat: Altering Message Contents -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -An attacker could try to alter the contents of an existing message from the -victim. - -Threat: Fake Message "origin" Field -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -An attacker could try to send a new message purporting to be from the victim -with a phony "origin" field. - -Spamming --------- - -The attacker could try to send a high volume of solicicted or unsolicted -messages to the victim in order to: - - * Find victims for scams. - * Market unwanted products. - -Threat: Unsoliticted Messages -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -An attacker could try to send messages to victims who do not wish to receive -them. - -Threat: Abusive Messages -~~~~~~~~~~~~~~~~~~~~~~~~ - -An attacker could send abusive or threatening messages to the victim - -Spying ------- - -The attacker could try to access message contents or metadata for messages sent -by the victim or to the victim that were not intended to reach the attacker in -order to: - - * Gain sensitive personal or commercial information. - * Impersonate the victim using credentials contained in the messages. - (e.g. password reset messages) - * Discover who the victim was talking to and when. - -Threat: Disclosure during Transmission -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -An attacker could try to expose the message contents or metadata during -transmission between the servers. - -Threat: Disclosure to Servers Outside Chatroom -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -An attacker could try to convince servers within a chatroom to send messages to -a server it controls that was not authorised to be within the chatroom. - -Threat: Disclosure to Servers Within Chatroom -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -An attacker could take control of a server within a chatroom to expose message -contents or metadata for messages in that room. - - diff --git a/docs/specification.rst b/docs/specification.rst index 2b47009187..239e51b4f3 100644 --- a/docs/specification.rst +++ b/docs/specification.rst @@ -1,11 +1,87 @@ Matrix Specification ==================== -TODO(Introduction) : Matthew - - Similar to intro paragraph from README. - - Explaining the overall mission, what this spec describes... - - "What is Matrix?" - - Draw parallels with email? +WARNING +======= + +.. WARNING:: + The Matrix specification is still very much evolving: the API is not yet frozen + and this document is in places incomplete, stale, and may contain security + issues. Needless to say, we have made every effort to highlight the problem + areas that we're aware of. + + We're publishing it at this point because it's complete enough to be more than + useful and provide a canonical reference to how Matrix is evolving. Our end + goal is to mirror WHATWG's `Living Standard <http://wiki.whatwg.org/wiki/FAQ#What_does_.22Living_Standard.22_mean.3F>`_ + approach except right now Matrix is more in the process of being born than actually being + living! + +.. contents:: Table of Contents +.. sectnum:: + +Introduction +============ + +Matrix is a new set of open APIs for open-federated Instant Messaging and VoIP +functionality, designed to create and support a new global real-time +communication ecosystem on the internet. This specification is the ongoing +result of standardising the APIs used by the various components of the Matrix +ecosystem to communicate with one another. + +The principles that Matrix attempts to follow are: + +- Pragmatic Web-friendly APIs (i.e. JSON over REST) +- Keep It Simple & Stupid + + + provide a simple architecture with minimal third-party dependencies. + +- Fully open: + + + Fully open federation - anyone should be able to participate in the global + Matrix network + + Fully open standard - publicly documented standard with no IP or patent + licensing encumbrances + + Fully open source reference implementation - liberally-licensed example + implementations with no IP or patent licensing encumbrances + +- Empowering the end-user + + + The user should be able to choose the server and clients they use + + The user should be control how private their communication is + + The user should know precisely where their data is stored + +- Fully decentralised - no single points of control over conversations or the + network as a whole +- Learning from history to avoid repeating it + + + Trying to take the best aspects of XMPP, SIP, IRC, SMTP, IMAP and NNTP + whilst trying to avoid their failings + +The functionality that Matrix provides includes: + +- Creation and management of fully distributed chat rooms with no + single points of control or failure +- Eventually-consistent cryptographically secure synchronisation of room + state across a global open network of federated servers and services +- Sending and receiving extensible messages in a room with (optional) + end-to-end encryption +- Extensible user management (inviting, joining, leaving, kicking, banning) + mediated by a power-level based user privilege system. +- Extensible room state management (room naming, aliasing, topics, bans) +- Extensible user profile management (avatars, displaynames, etc) +- Managing user accounts (registration, login, logout) +- Use of 3rd Party IDs (3PIDs) such as email addresses, phone numbers, + Facebook accounts to authenticate, identify and discover users on Matrix. +- Trusted federation of Identity servers for: + + + Publishing user public keys for PKI + + Mapping of 3PIDs to Matrix IDs + +The end goal of Matrix is to be a ubiquitous messaging layer for synchronising +arbitrary data between sets of people, devices and services - be that for instant +messages, VoIP call setups, or any other objects that need to be reliably and +persistently pushed from A to B in an interoperable and federated manner. + Architecture ============ @@ -28,38 +104,43 @@ other directly. | |<--------( HTTP )-----------| | +------------------+ Federation +------------------+ -A "Client" is an end-user, typically a human using a web application or mobile app. Clients use the -"Client-to-Server" (C-S) API to communicate with their home server. A single Client is usually -responsible for a single user account. A user account is represented by their "User ID". This ID is -namespaced to the home server which allocated the account and looks like:: +A "Client" typically represents a human using a web application or mobile app. Clients use the +"Client-to-Server" (C-S) API to communicate with their home server, which stores their profile data and +their record of the conversations in which they participate. Each client is associated with a user account +(and may optionally support multiple user accounts). A user account is represented by a unique "User ID". This +ID is namespaced to the home server which allocated the account and looks like:: @localpart:domain The ``localpart`` of a user ID may be a user name, or an opaque ID identifying this user. They are case-insensitive. +.. TODO + - Need to specify precise grammar for Matrix IDs + A "Home Server" is a server which provides C-S APIs and has the ability to federate with other HSes. It is typically responsible for multiple clients. "Federation" is the term used to describe the sharing of data between two or more home servers. -Data in Matrix is encapsulated in an "Event". An event is an action within the system. Typically each -action (e.g. sending a message) correlates with exactly one event. Each event has a ``type`` which is -used to differentiate different kinds of data. ``type`` values SHOULD be namespaced according to standard -Java package naming conventions, e.g. ``com.example.myapp.event``. Events are usually sent in the context -of a "Room". +Data in Matrix is encapsulated in an "event". An event is an action within the system. Typically each +action (e.g. sending a message) correlates with exactly one event. Each event has a ``type`` which is used +to differentiate different kinds of data. ``type`` values MUST be uniquely globally namespaced following +Java's `package naming conventions <http://docs.oracle.com/javase/specs/jls/se5.0/html/packages.html#7.7>`, +e.g. ``com.example.myapp.event``. The special top-level namespace ``m.`` is reserved for events defined +in the Matrix specification. Events are usually sent in the context of a "Room". Room structure -------------- A room is a conceptual place where users can send and receive events. Rooms can be created, joined and left. Events are sent to a room, and all -participants in that room will receive the event. Rooms are uniquely -identified via a "Room ID", which look like:: +participants in that room with sufficient access will receive the event. Rooms are uniquely +identified internally via a "Room ID", which look like:: !opaque_id:domain There is exactly one room ID for each room. Whilst the room ID does contain a -domain, it is simply for namespacing room IDs. The room does NOT reside on the +domain, it is simply for globally namespacing room IDs. The room does NOT reside on the domain specified. Room IDs are not meant to be human readable. They ARE case-sensitive. @@ -101,9 +182,12 @@ Each room can also have multiple "Room Aliases", which looks like:: #room_alias:domain -A room alias "points" to a room ID. The room ID the alias is pointing to can be obtained -by visiting the domain specified. Room aliases are designed to be human readable strings -which can be used to publicise rooms. They are case-insensitive. Note that the mapping + .. TODO + - Need to specify precise grammar for Room IDs + +A room alias "points" to a room ID and is the human-readable label by which rooms are +publicised and discovered. The room ID the alias is pointing to can be obtained +by visiting the domain specified. They are case-insensitive. Note that the mapping from a room alias to a room ID is not fixed, and may change over time to point to a different room ID. For this reason, Clients SHOULD resolve the room alias to a room ID once and then use that ID on subsequent requests. @@ -118,24 +202,61 @@ once and then use that ID on subsequent requests. | domain.com | | Mappings: | | #matrix >> !aaabaa:matrix.org | - | #golf >> !wfeiofh:sport.com | - | #bike >> !4rguxf:matrix.org | + | #golf >> !wfeiofh:sport.com | + | #bike >> !4rguxf:matrix.org | |________________________________| +.. TODO kegan + - show the actual API rather than pseudo-API? + Identity -------- -- Identity in relation to 3PIDs. Discovery of users based on 3PIDs. -- Identity servers; trusted clique of servers which replicate content. -- They govern the mapping of 3PIDs to user IDs and the creation of said mappings. -- Not strictly required in order to communicate. +Users in Matrix are identified via their user ID. However, existing ID namespaces can also +be used in order to identify Matrix users. A Matrix "Identity" describes both the user ID +and any other existing IDs from third party namespaces *linked* to their account. + +Matrix users can *link* third-party IDs (3PIDs) such as email addresses, social +network accounts and phone numbers to their +user ID. Linking 3PIDs creates a mapping from a 3PID to a user ID. This mapping +can then be used by other Matrix users in order to discover other users, according +to a strict set of privacy permissions. + +In order to ensure that the mapping from 3PID to user ID is genuine, a globally federated +cluster of trusted "Identity Servers" (IS) are used to perform authentication of the 3PID. +Identity servers are also used to preserve the mapping indefinitely, by replicating the +mappings across multiple ISes. + +Usage of an IS is not required in order for a client application to be part of +the Matrix ecosystem. However, by not using an IS, discovery of users is greatly +impacted. API Standards ------------- -All communication in Matrix is performed over HTTP[S] using a Content-Type of ``application/json``. -Any errors which occur on the Matrix API level MUST return a "standard error response". This is a -JSON object which looks like:: + +The mandatory baseline for communication in Matrix is exchanging JSON objects over RESTful +HTTP APIs. HTTPS is mandated as the baseline for server-server (federation) communication. +HTTPS is recommended for client-server communication, although HTTP may be supported as a +fallback to support basic HTTP clients. More efficient optional transports for +client-server communication will in future be supported as optional extensions - e.g. a +packed binary encoding over stream-cipher encrypted TCP socket for +low-bandwidth/low-roundtrip mobile usage. + +.. TODO + We need to specify capability negotiation for extensible transports + +For the default HTTP transport, all API calls use a Content-Type of ``application/json``. +In addition, all strings MUST be encoded as UTF-8. + +Clients are authenticated using opaque ``access_token`` strings (see `Registration and +Login`_ for details), passed as a querystring parameter on all requests. + +.. TODO + Need to specify any HMAC or access_token lifetime/ratcheting tricks + +Any errors which occur on the Matrix API level +MUST return a "standard error response". This is a JSON object which looks like:: { "errcode": "<error code>", @@ -186,55 +307,57 @@ Some requests have unique error codes: :``M_LOGIN_EMAIL_URL_NOT_YET``: Encountered when polling for an email link which has not been clicked yet. -The C-S API typically uses ``HTTP POST`` to submit requests. This means these requests -are not idempotent. The C-S API also allows ``HTTP PUT`` to make requests idempotent. -In order to use a ``PUT``, paths should be suffixed with ``/{txnId}``. ``{txnId}`` is a -client-generated transaction ID which identifies the request. Crucially, it **only** -serves to identify new requests from retransmits. After the request has finished, the -``{txnId}`` value should be changed (how is not specified, it could be a monotonically -increasing integer, etc). It is preferable to use ``HTTP PUT`` to make sure requests to -send messages do not get sent more than once should clients need to retransmit requests. +The C-S API typically uses ``HTTP POST`` to submit requests. This means these requests are +not idempotent. The C-S API also allows ``HTTP PUT`` to make requests idempotent. In order +to use a ``PUT``, paths should be suffixed with ``/{txnId}``. ``{txnId}`` is a +unique client-generated transaction ID which identifies the request, and is scoped to a given +Client (identified by that client's ``access_token``). Crucially, it **only** serves to +identify new requests from retransmits. After the request has finished, the ``{txnId}`` +value should be changed (how is not specified; a monotonically increasing integer is +recommended). It is preferable to use ``HTTP PUT`` to make sure requests to send messages +do not get sent more than once should clients need to retransmit requests. Valid requests look like:: - POST /some/path/here + POST /some/path/here?access_token=secret { "key": "This is a post." } - PUT /some/path/here/11 + PUT /some/path/here/11?access_token=secret { "key": "This is a put with a txnId of 11." } In contrast, these are invalid requests:: - POST /some/path/here/11 + POST /some/path/here/11?access_token=secret { "key": "This is a post, but it has a txnId." } - PUT /some/path/here + PUT /some/path/here?access_token=secret { "key": "This is a put but it is missing a txnId." } - - -- TODO: All strings everywhere are UTF-8 - - - Receiving live updates on a client ---------------------------------- + Clients can receive new events by long-polling the home server. This will hold open the HTTP connection for a short period of time waiting for new events, returning early if an -event occurs. This is called the "Event Stream". All events which the client is authorised -to view will appear in the event stream. When the stream is closed, an ``end`` token is -returned. This token can be used in the next request to continue where the client left off. +event occurs. This is called the `Event Stream`_. All events which are visible to the +client and match the client's query will appear in the event stream. When the request +returns, an ``end`` token is included in the response. This token can be used in the next +request to continue where the client left off. + +.. TODO + Do we ever return multiple events in a single request? Don't we get lots of request + setup RTT latency if we only do one event per request? Do we ever support streaming + requests? Why not websockets? When the client first logs in, they will need to initially synchronise with their home -server. This is achieved via the ``/initialSync`` API. This API also returns an ``end`` +server. This is achieved via the |initialSync|_ API. This API also returns an ``end`` token which can be used with the event stream. Rooms @@ -242,7 +365,10 @@ Rooms Creation -------- -To create a room, a client has to use the ``/createRoom`` API. There are various options +.. TODO kegan + - TODO: Key for invite these users? + +To create a room, a client has to use the |createRoom|_ API. There are various options which can be set when creating a room: ``visibility`` @@ -278,7 +404,7 @@ which can be set when creating a room: The ``name`` value for the ``m.room.name`` state event. Description: If this is included, an ``m.room.name`` event will be sent into the room to indicate the - name of the room. See "Room Events" for more information on ``m.room.name``. + name of the room. See `Room Events`_ for more information on ``m.room.name``. ``topic`` Type: @@ -289,7 +415,7 @@ which can be set when creating a room: The ``topic`` value for the ``m.room.topic`` state event. Description: If this is included, an ``m.room.topic`` event will be sent into the room to indicate the - topic for the room. See "Room Events" for more information on ``m.room.topic``. + topic for the room. See `Room Events`_ for more information on ``m.room.topic``. Example:: @@ -300,35 +426,81 @@ Example:: "topic": "All about happy hour" } -- TODO: This creates a room creation event which serves as the root of the PDU graph for this room. -- TODO: Keys for speccing a room name / room topic / invite these users? +The home server will create a ``m.room.create`` event when the room is +created, which serves as the root of the PDU graph for this room. This +event also has a ``creator`` key which contains the user ID of the room +creator. It will also generate several other events in order to manage +permissions in this room. This includes: + + - ``m.room.power_levels`` : Sets the power levels of users. + - ``m.room.join_rules`` : Whether the room is "invite-only" or not. + - ``m.room.add_state_level``: The power level required in order to + add new state to the room (as opposed to updating exisiting state) + - ``m.room.send_event_level`` : The power level required in order to + send a message in this room. + - ``m.room.ops_level`` : The power level required in order to kick or + ban a user from the room. + +See `Room Events`_ for more information on these events. Modifying aliases ----------------- -- path to edit aliases -- format when retrieving list of aliases. NOT complete list. -- format for adding aliases. +.. NOTE:: + This section is a work in progress. + +.. TODO kegan + - path to edit aliases + - PUT /directory/room/<room alias> { room_id : foo } + - GET /directory/room/<room alias> { room_id : foo, servers: [a.com, b.com] } + - format when retrieving list of aliases. NOT complete list. + - format for adding/removing aliases. Permissions ----------- -- TODO: What is a power level? How do they work? Defaults / required levels for X. How do they change - as people join and leave rooms? What do you do if you get a clash? Examples. -- TODO: List all actions which use power levels (sending msgs, inviting users, banning people, etc...) -- TODO: Room config - what is the event and what are the keys/values and explanations for them. - Link through to respective sections where necessary. How does this tie in with permissions, e.g. - give example of creating a read-only room. +.. NOTE:: + This section is a work in progress. + +.. TODO kegan + - TODO: What is a power level? How do they work? Defaults / required levels for X. How do they change + as people join and leave rooms? What do you do if you get a clash? Examples. + - TODO: List all actions which use power levels (sending msgs, inviting users, banning people, etc...) + - TODO: Room config - what is the event and what are the keys/values and explanations for them. + Link through to respective sections where necessary. How does this tie in with permissions, e.g. + give example of creating a read-only room. + +Permissions for rooms are done via the concept of power levels - to do any +action in a room a user must have a suitable power level. + +Power levels for users are defined in ``m.room.power_levels``, where both +a default and specific users' power levels can be set. By default all users +have a power level of 0. + +State events may contain a ``required_power_level`` key, which indicates the +minimum power a user must have before they can update that state key. The only +exception to this is when a user leaves a room. + +To perform certain actions there are additional power level requirements +defined in the following state events: + +- ``m.room.send_event_level`` defines the minimum level for sending non-state + events. Defaults to 5. +- ``m.room.add_state_level`` defines the minimum level for adding new state, + rather than updating existing state. Defaults to 5. +- ``m.room.ops_level`` defines the minimum levels to ban and kick other users. + This defaults to a kick and ban levels of 5 each. Joining rooms ------------- -- TODO: What does the home server have to do to join a user to a room? +.. TODO kegan + - TODO: What does the home server have to do to join a user to a room? Users need to join a room in order to send and receive events in that room. A user can join a -room by making a request to ``/join/<room alias or id>`` with:: +room by making a request to |/join/<room_alias_or_id>|_ with:: {} -Alternatively, a user can make a request to ``/rooms/<room id>/join`` with the same request content. +Alternatively, a user can make a request to |/rooms/<room_id>/join|_ with the same request content. This is only provided for symmetry with the other membership APIs: ``/rooms/<room id>/invite`` and ``/rooms/<room id>/leave``. If a room alias was specified, it will be automatically resolved to a room ID, which will then be joined. The room ID that was joined will be returned in response:: @@ -345,19 +517,21 @@ by sending the following request to "membership": "join" } -See the "Room events" section for more information on ``m.room.member``. +See the `Room events`_ section for more information on ``m.room.member``. After the user has joined a room, they will receive subsequent events in that room. This room -will now appear as an entry in the ``/initialSync`` API. +will now appear as an entry in the |initialSync|_ API. Some rooms enforce that a user is *invited* to a room before they can join that room. Other rooms will allow anyone to join the room even if they have not received an invite. Inviting users -------------- -- Can invite users to a room if the room config key TODO is set to TODO. Must have required power level. -- Outline invite join dance. What is it? Why is it required? How does it work? -- What does the home server have to do? +.. TODO kegan + - Can invite users to a room if the room config key TODO is set to TODO. Must have required power level. + - Outline invite join dance. What is it? Why is it required? How does it work? + - What does the home server have to do? + - TODO: In what circumstances will direct member editing NOT be equivalent to ``/invite``? The purpose of inviting users to a room is to notify them that the room exists so they can choose to become a member of that room. Some rooms require that all @@ -372,7 +546,7 @@ Only users who have a membership state of ``join`` in a room can invite new users to said room. The person being invited must not be in the ``join`` state in the room. The fully-qualified user ID must be specified when inviting a user, as the user may reside on a different home server. To invite a user, send the -following request to ``/rooms/<room id>/invite``, which will manage the +following request to |/rooms/<room_id>/invite|_, which will manage the entire invitation process:: { @@ -387,16 +561,19 @@ directly by sending the following request to "membership": "invite" } -See the "Room events" section for more information on ``m.room.member``. - -- TODO: In what circumstances will this NOT be equivalent to ``/invite``? +See the `Room events`_ section for more information on ``m.room.member``. Leaving rooms ------------- +.. TODO kegan + - TODO: Grace period before deletion? + - TODO: Under what conditions should a room NOT be purged? + + A user can leave a room to stop receiving events for that room. A user must have joined the room before they are eligible to leave the room. If the room is an "invite-only" room, they will need to be re-invited before they can re-join the room. -To leave a room, a request should be made to ``/rooms/<room id>/leave`` with:: +To leave a room, a request should be made to |/rooms/<room_id>/leave|_ with:: {} @@ -408,9 +585,9 @@ directly by sending the following request to "membership": "leave" } -See the "Room events" section for more information on ``m.room.member``. +See the `Room events`_ section for more information on ``m.room.member``. -Once a user has left a room, that room will no longer appear on the ``/initialSync`` +Once a user has left a room, that room will no longer appear on the |initialSync|_ API. Be aware that leaving a room is not equivalent to have never been in that room. A user who has previously left a room still maintains some residual state in that room. Their membership state will be marked as ``leave``. This contrasts with @@ -418,18 +595,15 @@ a user who has *never been invited or joined to that room* who will not have any membership state for that room. If all members in a room leave, that room becomes eligible for deletion. - - TODO: Grace period before deletion? - - TODO: Under what conditions should a room NOT be purged? Banning users in a room ----------------------- - A user may decide to ban another user in a room. 'Banning' forces the target user to leave the room and prevents them from re-joining the room. A banned user will not be treated as a joined user, and so will not be able to send or receive events in the room. In order to ban someone, the user performing the ban MUST have the required power level. To ban a user, a request should be made to -``/rooms/<room id>/ban`` with:: +|/rooms/<room_id>/ban|_ with:: { "user_id": "<user id to ban" @@ -469,7 +643,7 @@ risk of clashes. State events ------------ -State events can be sent by ``PUT`` ing to ``/rooms/<room id>/state/<event type>/<state key>``. +State events can be sent by ``PUT`` ing to |/rooms/<room_id>/state/<event_type>/<state_key>|_. These events will be overwritten if ``<room id>``, ``<event type>`` and ``<state key>`` all match. If the state event has no ``state_key``, it can be omitted from the path. These requests **cannot use transaction IDs** like other ``PUT`` paths because they cannot be differentiated @@ -506,11 +680,11 @@ In some cases, there may be no need for a ``state_key``, so it can be omitted:: PUT /rooms/!roomid:domain/state/m.room.bgd.color { "color": "red", "hex": "#ff0000" } -See "Room Events" for the ``m.`` event specification. +See `Room Events`_ for the ``m.`` event specification. Non-state events ---------------- -Non-state events can be sent by sending a request to ``/rooms/<room id>/send/<event type>``. +Non-state events can be sent by sending a request to |/rooms/<room_id>/send/<event_type>|_. These requests *can* use transaction IDs and ``PUT``/``POST`` methods. Non-state events allow access to historical events and pagination, making it best suited for sending messages. For example:: @@ -521,23 +695,27 @@ For example:: PUT /rooms/!roomid:domain/send/m.custom.example.message/11 { "text": "Goodbye world!" } -See "Room Events" for the ``m.`` event specification. +See `Room Events`_ for the ``m.`` event specification. Syncing rooms ------------- +.. NOTE:: + This section is a work in progress. + When a client logs in, they may have a list of rooms which they have already joined. These rooms may also have a list of events associated with them. The purpose of 'syncing' is to present the current room and event information in a convenient, compact manner. The events returned are not limited to room events; presence events will also be returned. There are two APIs provided: - - ``/initialSync`` : A global sync which will present room and event information for all rooms + - |initialSync|_ : A global sync which will present room and event information for all rooms the user has joined. - - ``/rooms/<room id>/initialSync`` : A sync scoped to a single room. Presents room and event + - |/rooms/<room_id>/initialSync|_ : A sync scoped to a single room. Presents room and event information for this room only. -- TODO: JSON response format for both types -- TODO: when would you use global? when would you use scoped? +.. TODO kegan + - TODO: JSON response format for both types + - TODO: when would you use global? when would you use scoped? Getting events for a room ------------------------- @@ -551,7 +729,7 @@ There are several APIs provided to ``GET`` events for a room: Example: ``/rooms/!room:domain.com/state/m.room.name`` returns ``{ "name": "Room name" }`` -``/rooms/<room id>/state`` +|/rooms/<room_id>/state|_ Description: Get all state events for a room. Response format: @@ -560,7 +738,7 @@ There are several APIs provided to ``GET`` events for a room: TODO -``/rooms/<room id>/members`` +|/rooms/<room_id>/members|_ Description: Get all ``m.room.member`` state events. Response format: @@ -568,7 +746,7 @@ There are several APIs provided to ``GET`` events for a room: Example: TODO -``/rooms/<room id>/messages`` +|/rooms/<room_id>/messages|_ Description: Get all ``m.room.message`` events. Response format: @@ -576,7 +754,7 @@ There are several APIs provided to ``GET`` events for a room: Example: TODO -``/rooms/<room id>/initialSync`` +|/rooms/<room_id>/initialSync|_ Description: Get all relevant events for a room. This includes state events, paginated non-state events and presence events. @@ -588,7 +766,11 @@ There are several APIs provided to ``GET`` events for a room: Room Events =========== -- voip events? +.. NOTE:: + This section is a work in progress. + +.. TODO dave? + - voip events? This specification outlines several standard event types, all of which are prefixed with ``m.`` @@ -607,7 +789,7 @@ prefixed with ``m.`` human-friendly, but not all rooms have room aliases. The room name is a human-friendly string designed to be displayed to the end-user. The room name is not *unique*, as multiple rooms can have the same room name set. The room name can also be set when - creating a room using ``/createRoom`` with the ``name`` key. + creating a room using |createRoom|_ with the ``name`` key. ``m.room.topic`` Summary: @@ -621,7 +803,8 @@ prefixed with ``m.`` Description: A topic is a short message detailing what is currently being discussed in the room. It can also be used as a way to display extra information about the room, which may - not be suitable for the room name. + not be suitable for the room name. The room topic can also be set when creating a + room using |createRoom|_ with the ``topic`` key. ``m.room.member`` Summary: @@ -637,32 +820,91 @@ prefixed with ``m.`` membership APIs (``/rooms/<room id>/invite`` etc) when performing membership actions rather than adjusting the state directly as there are a restricted set of valid transformations. For example, user A cannot force user B to join a room, and trying - to force this state change directly will fail. See the "Rooms" section for how to + to force this state change directly will fail. See the `Rooms`_ section for how to use the membership APIs. -``m.room.config`` +``m.room.create`` Summary: - The room config. + The first event in the room. Type: State event JSON format: - TODO + ``{ "creator": "string"}`` Example: - TODO + ``{ "creator": "@user:example.com" }`` Description: - TODO + This is the first event in a room and cannot be changed. It acts as the + root of all other events. -``m.room.invite_join`` +``m.room.join_rules`` Summary: - TODO. + Descripes how/if people are allowed to join. Type: State event JSON format: - TODO + ``{ "join_rule": "enum [ public|knock|invite|private ]" }`` Example: - TODO + ``{ "join_rule": "public" }`` Description: - TODO + TODO : Use docs/models/rooms.rst + +``m.room.power_levels`` + Summary: + Defines the power levels of users in the room. + Type: + State event + JSON format: + ``{ "<user_id>": <int>, ..., "default": <int>}`` + Example: + ``{ "@user:example.com": 5, "@user2:example.com": 10, "default": 0 }`` + Description: + If a user is in the list, then they have the associated power level. + Otherwise they have the default level. If not ``default`` key is supplied, + it is assumed to be 0. + +``m.room.add_state_level`` + Summary: + Defines the minimum power level a user needs to add state. + Type: + State event + JSON format: + ``{ "level": <int> }`` + Example: + ``{ "level": 5 }`` + Description: + To add a new piece of state to the room a user must have the given power + level. This does not apply to updating current state, which is goverened + by the ``required_power_level`` event key. + +``m.room.send_event_level`` + Summary: + Defines the minimum power level a user needs to send an event. + Type: + State event + JSON format: + ``{ "level": <int> }`` + Example: + ``{ "level": 0 }`` + Description: + To send a new event into the room a user must have at least this power + level. This allows ops to make the room read only by increasing this level, + or muting individual users by lowering their power level below this + threshold. + +``m.room.ops_levels`` + Summary: + Defines the minimum power levels that a user must have before they can + kick and/or ban other users. + Type: + State event + JSON format: + ``{ "ban_level": <int>, "kick_level": <int> }`` + Example: + ``{ "ban_level": 5, "kick_level": 5 }`` + Description: + This defines who can ban and/or kick people in the room. Most of the time + ``ban_level`` will be greater than or equal to ``kick_level`` since + banning is more severe than kicking. ``m.room.message`` Summary: @@ -678,7 +920,7 @@ prefixed with ``m.`` The ``msgtype`` key outlines the type of message, e.g. text, audio, image, video, etc. Whilst not required, the ``body`` key SHOULD be used with every kind of ``msgtype`` as a fallback mechanism when a client cannot render the message. For more information on - the types of messages which can be sent, see "m.room.message msgtypes". + the types of messages which can be sent, see `m.room.message msgtypes`_. ``m.room.message.feedback`` Summary: @@ -799,6 +1041,8 @@ The following keys can be attached to any ``m.room.message``: Presence ======== +.. NOTE:: + This section is a work in progress. Each user has the concept of presence information. This encodes the "availability" of that user, suitable for display on other user's clients. This @@ -837,8 +1081,12 @@ user was last seen online. Transmission ------------ -- Transmitted as an EDU. -- Presence lists determine who to send to. +.. NOTE:: + This section is a work in progress. + +.. TODO: + - Transmitted as an EDU. + - Presence lists determine who to send to. Presence List ------------- @@ -863,28 +1111,43 @@ presence information in a user list for a room. Typing notifications ==================== -- what is the event type. Are they bundled with other event types? If so, which. -- what are the valid keys / values. What do they represent. Any gotchas? -- Timeouts. How do they work, who sets them and how do they expire. Does one - have priority over another? Give examples. +.. NOTE:: + This section is a work in progress. -TODO : Leo +.. TODO Leo + - what is the event type. Are they bundled with other event types? If so, which. + - what are the valid keys / values. What do they represent. Any gotchas? + - Timeouts. How do they work, who sets them and how do they expire. Does one + have priority over another? Give examples. Voice over IP ============= -- what are the event types. -- what are the valid keys/values. What do they represent. Any gotchas? -- In what sequence should the events be sent? -- How do you accept / decline inbound calls? How do you make outbound calls? - Give examples. -- How does negotiation work? Give examples. -- How do you hang up? -- What does call log information look like e.g. duration of call? - -TODO : Dave +.. NOTE:: + This section is a work in progress. + +.. TODO Dave + - what are the event types. + - what are the valid keys/values. What do they represent. Any gotchas? + - In what sequence should the events be sent? + - How do you accept / decline inbound calls? How do you make outbound calls? + Give examples. + - How does negotiation work? Give examples. + - How do you hang up? + - What does call log information look like e.g. duration of call? Profiles ======== +.. NOTE:: + This section is a work in progress. + +.. TODO + - Metadata extensibility + - Changing profile info generates m.presence events ("presencelike") + - keys on m.presence are optional, except presence which is required + - m.room.member is populated with the current displayname at that point in time. + - That is added by the HS, not you. + - Display name changes also generates m.room.member with displayname key f.e. room + the user is in. Internally within Matrix users are referred to by their user ID, which is not a human-friendly string. Profiles grant users the ability to see human-readable @@ -896,31 +1159,29 @@ metadata fields that the user may wish to publish (email address, phone numbers, website URLs, etc...). This specification puts no requirements on the display name other than it being a valid unicode string. -- Metadata extensibility -- Changing profile info generates m.presence events ("presencelike") -- keys on m.presence are optional, except presence which is required -- m.room.member is populated with the current displayname at that point in time. -- That is added by the HS, not you. -- Display name changes also generates m.room.member with displayname key f.e. room - the user is in. + Registration and login ====================== +.. WARNING:: + The registration API is likely to change. + +.. TODO + - TODO Kegan : Make registration like login (just omit the "user" key on the + initial request?) Clients must register with a home server in order to use Matrix. After registering, the client will be given an access token which must be used in ALL requests to that home server as a query parameter 'access_token'. -- TODO Kegan : Make registration like login (just omit the "user" key on the - initial request?) - If the client has already registered, they need to be able to login to their account. The home server may provide many different ways of logging in, such as user/password auth, login via a social network (OAuth2), login by confirming a token sent to their email address, etc. This specification does not define how home servers should authorise their users who want to login to their existing accounts, but instead defines the standard interface which implementations -should follow so that ANY client can login to ANY home server. +should follow so that ANY client can login to ANY home server. Clients login +using the |login|_ API. The login process breaks down into the following: 1. Determine the requirements for logging in. @@ -985,7 +1246,7 @@ This specification defines the following login types: Password-based -------------- :Type: - m.login.password + ``m.login.password`` :Description: Login is supported via a username and password. @@ -1003,7 +1264,7 @@ process, or a standard error response. OAuth2-based ------------ :Type: - m.login.oauth2 + ``m.login.oauth2`` :Description: Login is supported via OAuth2 URLs. This login consists of multiple requests. @@ -1056,7 +1317,7 @@ visits the REDIRECT_URI with the auth code= query parameter which returns:: Email-based (code) ------------------ :Type: - m.login.email.code + ``m.login.email.code`` :Description: Login is supported by typing in a code which is sent in an email. This login consists of multiple requests. @@ -1091,7 +1352,7 @@ the login process, or a standard error response. Email-based (url) ----------------- :Type: - m.login.email.url + ``m.login.email.url`` :Description: Login is supported by clicking on a URL in an email. This login consists of multiple requests. @@ -1190,9 +1451,11 @@ This MUST return an HTML page which can perform the entire login process. Identity ======== +.. NOTE:: + This section is a work in progress. -TODO : Dave -- 3PIDs and identity server, functions +.. TODO Dave + - 3PIDs and identity server, functions Federation ========== @@ -1233,6 +1496,9 @@ transferred from the origin to the destination home server using an HTTP PUT req Transactions ------------ +.. WARNING:: + This section may be misleading or inaccurate. + The transfer of EDUs and PDUs between home servers is performed by an exchange of Transaction messages, which are encoded as JSON objects, passed over an HTTP PUT request. A Transaction is meaningful only to the pair of home servers that @@ -1244,6 +1510,31 @@ Each transaction has: - An origin and destination server name. - A list of "previous IDs". - A list of PDUs and EDUs - the actual message payload that the Transaction carries. + +``origin`` + Type: + String + Description: + DNS name of homeserver making this transaction. + +``ts`` + Type: + Integer + Description: + Timestamp in milliseconds on originating homeserver when this transaction + started. + +``previous_ids`` + Type: + List of strings + Description: + List of transactions that were sent immediately prior to this transaction. + +``pdus`` + Type: + List of Objects. + Description: + List of updates contained in this transaction. :: @@ -1275,6 +1566,8 @@ mechanism to encourage peers to continue to replicate content.) PDUs and EDUs ------------- +.. WARNING:: + This section may be misleading or inaccurate. All PDUs have: - An ID @@ -1283,8 +1576,98 @@ All PDUs have: - A list of other PDU IDs that have been seen recently on that context (regardless of which origin sent them) -[[TODO(paul): Update this structure so that 'pdu_id' is a two-element -[origin,ref] pair like the prev_pdus are]] +``context`` + Type: + String + Description: + Event context identifier + +``origin`` + Type: + String + Description: + DNS name of homeserver that created this PDU. + +``pdu_id`` + Type: + String + Description: + Unique identifier for PDU within the context for the originating homeserver + +``ts`` + Type: + Integer + Description: + Timestamp in milliseconds on originating homeserver when this PDU was created. + +``pdu_type`` + Type: + String + Description: + PDU event type. + +``prev_pdus`` + Type: + List of pairs of strings + Description: + The originating homeserver and PDU ids of the most recent PDUs the + homeserver was aware of for this context when it made this PDU. + +``depth`` + Type: + Integer + Description: + The maximum depth of the previous PDUs plus one. + + +.. TODO paul + [[TODO(paul): Update this structure so that 'pdu_id' is a two-element + [origin,ref] pair like the prev_pdus are]] + + +For state updates: + +``is_state`` + Type: + Boolean + Description: + True if this PDU is updating state. + +``state_key`` + Type: + String + Description: + Optional key identifying the updated state within the context. + +``power_level`` + Type: + Integer + Description: + The asserted power level of the user performing the update. + +``min_update`` + Type: + Integer + Description: + The required power level needed to replace this update. + +``prev_state_id`` + Type: + String + Description: + PDU event type. + +``prev_state_origin`` + Type: + String + Description: + The PDU id of the update this replaces. + +``user`` + Type: + String + Description: + The user updating the state. :: @@ -1325,12 +1708,13 @@ keys exist to support this: "prev_state_id":TODO "prev_state_origin":TODO} -[[TODO(paul): At this point we should probably have a long description of how -State management works, with descriptions of clobbering rules, power levels, etc -etc... But some of that detail is rather up-in-the-air, on the whiteboard, and -so on. This part needs refining. And writing in its own document as the details -relate to the server/system as a whole, not specifically to server-server -federation.]] +.. TODO paul + [[TODO(paul): At this point we should probably have a long description of how + State management works, with descriptions of clobbering rules, power levels, etc + etc... But some of that detail is rather up-in-the-air, on the whiteboard, and + so on. This part needs refining. And writing in its own document as the details + relate to the server/system as a whole, not specifically to server-server + federation.]] EDUs, by comparison to PDUs, do not have an ID, a context, or a list of "previous" IDs. The only mandatory fields for these are the type, origin and @@ -1342,46 +1726,393 @@ destination home server names, and the actual nested content. "origin":"blue", "destination":"orange", "content":...} + + +Protocol URLs +============= +.. WARNING:: + This section may be misleading or inaccurate. + +All these URLs are namespaced within a prefix of:: + + /_matrix/federation/v1/... + +For active pushing of messages representing live activity "as it happens":: + + PUT .../send/:transaction_id/ + Body: JSON encoding of a single Transaction + Response: TODO + +The transaction_id path argument will override any ID given in the JSON body. +The destination name will be set to that of the receiving server itself. Each +embedded PDU in the transaction body will be processed. + + +To fetch a particular PDU:: + + GET .../pdu/:origin/:pdu_id/ + Response: JSON encoding of a single Transaction containing one PDU + +Retrieves a given PDU from the server. The response will contain a single new +Transaction, inside which will be the requested PDU. + + +To fetch all the state of a given context:: + + GET .../state/:context/ + Response: JSON encoding of a single Transaction containing multiple PDUs + +Retrieves a snapshot of the entire current state of the given context. The +response will contain a single Transaction, inside which will be a list of +PDUs that encode the state. + +To backfill events on a given context:: + + GET .../backfill/:context/ + Query args: v, limit + Response: JSON encoding of a single Transaction containing multiple PDUs + +Retrieves a sliding-window history of previous PDUs that occurred on the +given context. Starting from the PDU ID(s) given in the "v" argument, the +PDUs that preceeded it are retrieved, up to a total number given by the +"limit" argument. These are then returned in a new Transaction containing all +off the PDUs. + + +To stream events all the events:: + + GET .../pull/ + Query args: origin, v + Response: JSON encoding of a single Transaction consisting of multiple PDUs + +Retrieves all of the transactions later than any version given by the "v" +arguments. + + +To make a query:: + + GET .../query/:query_type + Query args: as specified by the individual query types + Response: JSON encoding of a response object + +Performs a single query request on the receiving home server. The Query Type +part of the path specifies the kind of query being made, and its query +arguments have a meaning specific to that kind of query. The response is a +JSON-encoded object whose meaning also depends on the kind of query. Backfilling ----------- -- What it is, when is it used, how is it done +.. NOTE:: + This section is a work in progress. + +.. TODO + - What it is, when is it used, how is it done SRV Records ----------- -- Why it is needed +.. NOTE:: + This section is a work in progress. + +.. TODO + - Why it is needed Security ======== -- rate limiting -- crypto (s-s auth) -- E2E -- Lawful intercept + Key Escrow -TODO Mark +.. NOTE:: + This section is a work in progress. + +Threat Model +------------ + +Denial of Service +~~~~~~~~~~~~~~~~~ + +The attacker could attempt to prevent delivery of messages to or from the +victim in order to: + +* Disrupt service or marketing campaign of a commercial competitor. +* Censor a discussion or censor a participant in a discussion. +* Perform general vandalism. + +Threat: Resource Exhaustion ++++++++++++++++++++++++++++ + +An attacker could cause the victims server to exhaust a particular resource +(e.g. open TCP connections, CPU, memory, disk storage) + +Threat: Unrecoverable Consistency Violations +++++++++++++++++++++++++++++++++++++++++++++ + +An attacker could send messages which created an unrecoverable "split-brain" +state in the cluster such that the victim's servers could no longer dervive a +consistent view of the chatroom state. + +Threat: Bad History ++++++++++++++++++++ + +An attacker could convince the victim to accept invalid messages which the +victim would then include in their view of the chatroom history. Other servers +in the chatroom would reject the invalid messages and potentially reject the +victims messages as well since they depended on the invalid messages. + +Threat: Block Network Traffic ++++++++++++++++++++++++++++++ + +An attacker could try to firewall traffic between the victim's server and some +or all of the other servers in the chatroom. + +Threat: High Volume of Messages ++++++++++++++++++++++++++++++++ + +An attacker could send large volumes of messages to a chatroom with the victim +making the chatroom unusable. + +Threat: Banning users without necessary authorisation ++++++++++++++++++++++++++++++++++++++++++++++++++++++ + +An attacker could attempt to ban a user from a chatroom with the necessary +authorisation. + +Spoofing +~~~~~~~~ + +An attacker could try to send a message claiming to be from the victim without +the victim having sent the message in order to: + +* Impersonate the victim while performing illict activity. +* Obtain privileges of the victim. + +Threat: Altering Message Contents ++++++++++++++++++++++++++++++++++ + +An attacker could try to alter the contents of an existing message from the +victim. + +Threat: Fake Message "origin" Field ++++++++++++++++++++++++++++++++++++ + +An attacker could try to send a new message purporting to be from the victim +with a phony "origin" field. + +Spamming +~~~~~~~~ + +The attacker could try to send a high volume of solicicted or unsolicted +messages to the victim in order to: + +* Find victims for scams. +* Market unwanted products. + +Threat: Unsoliticted Messages ++++++++++++++++++++++++++++++ + +An attacker could try to send messages to victims who do not wish to receive +them. + +Threat: Abusive Messages +++++++++++++++++++++++++ + +An attacker could send abusive or threatening messages to the victim + +Spying +~~~~~~ + +The attacker could try to access message contents or metadata for messages sent +by the victim or to the victim that were not intended to reach the attacker in +order to: + +* Gain sensitive personal or commercial information. +* Impersonate the victim using credentials contained in the messages. + (e.g. password reset messages) +* Discover who the victim was talking to and when. + +Threat: Disclosure during Transmission +++++++++++++++++++++++++++++++++++++++ + +An attacker could try to expose the message contents or metadata during +transmission between the servers. + +Threat: Disclosure to Servers Outside Chatroom +++++++++++++++++++++++++++++++++++++++++++++++ + +An attacker could try to convince servers within a chatroom to send messages to +a server it controls that was not authorised to be within the chatroom. + +Threat: Disclosure to Servers Within Chatroom +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +An attacker could take control of a server within a chatroom to expose message +contents or metadata for messages in that room. + +Rate limiting +------------- +Home servers SHOULD implement rate limiting to reduce the risk of being overloaded. If a +request is refused due to rate limiting, it should return a standard error response of +the form:: + + { + "errcode": "M_LIMIT_EXCEEDED", + "error": "string", + "retry_after_ms": integer (optional) + } + +The ``retry_after_ms`` key SHOULD be included to tell the client how long they have to wait +in milliseconds before they can try again. + +.. TODO + - crypto (s-s auth) + - E2E + - Lawful intercept + Key Escrow + TODO Mark Policy Servers ============== -TODO +.. NOTE:: + This section is a work in progress. Content repository ================== -- path to upload -- format for thumbnail paths, mention what it is protecting against. -- content size limit and associated M_ERROR. +.. NOTE:: + This section is a work in progress. + +.. TODO + - path to upload + - format for thumbnail paths, mention what it is protecting against. + - content size limit and associated M_ERROR. Address book repository ======================= -- format: POST(?) wodges of json, some possible processing, then return wodges of json on GET. -- processing may remove dupes, merge contacts, pepper with extra info (e.g. matrix-ability of - contacts), etc. -- Standard json format for contacts? Piggy back off vcards? +.. NOTE:: + This section is a work in progress. + +.. TODO + - format: POST(?) wodges of json, some possible processing, then return wodges of json on GET. + - processing may remove dupes, merge contacts, pepper with extra info (e.g. matrix-ability of + contacts), etc. + - Standard json format for contacts? Piggy back off vcards? Glossary ======== -- domain specific words/acronyms with definitions +.. NOTE:: + This section is a work in progress. + +Backfilling: + The process of synchronising historic state from one home server to another, + to backfill the event storage so that scrollback can be presented to the + client(s). Not to be confused with pagination. + +Context: + A single human-level entity of interest (currently, a chat room) + +EDU (Ephemeral Data Unit): + A message that relates directly to a given pair of home servers that are + exchanging it. EDUs are short-lived messages that related only to one single + pair of servers; they are not persisted for a long time and are not forwarded + on to other servers. Because of this, they have no internal ID nor previous + EDUs reference chain. + +Event: + A record of activity that records a single thing that happened on to a context + (currently, a chat room). These are the "chat messages" that Synapse makes + available. + +PDU (Persistent Data Unit): + A message that relates to a single context, irrespective of the server that + is communicating it. PDUs either encode a single Event, or a single State + change. A PDU is referred to by its PDU ID; the pair of its origin server + and local reference from that server. + +PDU ID: + The pair of PDU Origin and PDU Reference, that together globally uniquely + refers to a specific PDU. + +PDU Origin: + The name of the origin server that generated a given PDU. This may not be the + server from which it has been received, due to the way they are copied around + from server to server. The origin always records the original server that + created it. + +PDU Reference: + A local ID used to refer to a specific PDU from a given origin server. These + references are opaque at the protocol level, but may optionally have some + structured meaning within a given origin server or implementation. + +Presence: + The concept of whether a user is currently online, how available they declare + they are, and so on. See also: doc/model/presence + +Profile: + A set of metadata about a user, such as a display name, provided for the + benefit of other users. See also: doc/model/profiles + +Room ID: + An opaque string (of as-yet undecided format) that identifies a particular + room and used in PDUs referring to it. + +Room Alias: + A human-readable string of the form #name:some.domain that users can use as a + pointer to identify a room; a Directory Server will map this to its Room ID + +State: + A set of metadata maintained about a Context, which is replicated among the + servers in addition to the history of Events. User ID: - An opaque ID which identifies an end-user, which consists of some opaque - localpart combined with the domain name of their home server. + A string of the form @localpart:domain.name that identifies a user for + wire-protocol purposes. The localpart is meaningless outside of a particular + home server. This takes a human-readable form that end-users can use directly + if they so wish, avoiding the 3PIDs. + +Transaction: + A message which relates to the communication between a given pair of servers. + A transaction contains possibly-empty lists of PDUs and EDUs. + + +.. Links through the external API docs are below +.. ============================================= + +.. |createRoom| replace:: ``/createRoom`` +.. _createRoom: /docs/api/client-server/#!/-rooms/create_room + +.. |initialSync| replace:: ``/initialSync`` +.. _initialSync: /docs/api/client-server/#!/-events/initial_sync + +.. |/rooms/<room_id>/initialSync| replace:: ``/rooms/<room_id>/initialSync`` +.. _/rooms/<room_id>/initialSync: /docs/api/client-server/#!/-rooms/get_room_sync_data + +.. |login| replace:: ``/login`` +.. _login: /docs/api/client-server/#!/-login + +.. |/rooms/<room_id>/messages| replace:: ``/rooms/<room_id>/messages`` +.. _/rooms/<room_id>/messages: /docs/api/client-server/#!/-rooms/get_messages + +.. |/rooms/<room_id>/members| replace:: ``/rooms/<room_id>/members`` +.. _/rooms/<room_id>/members: /docs/api/client-server/#!/-rooms/get_members + +.. |/rooms/<room_id>/state| replace:: ``/rooms/<room_id>/state`` +.. _/rooms/<room_id>/state: /docs/api/client-server/#!/-rooms/get_state_events + +.. |/rooms/<room_id>/send/<event_type>| replace:: ``/rooms/<room_id>/send/<event_type>`` +.. _/rooms/<room_id>/send/<event_type>: /docs/api/client-server/#!/-rooms/send_non_state_event + +.. |/rooms/<room_id>/state/<event_type>/<state_key>| replace:: ``/rooms/<room_id>/state/<event_type>/<state_key>`` +.. _/rooms/<room_id>/state/<event_type>/<state_key>: /docs/api/client-server/#!/-rooms/send_state_event + +.. |/rooms/<room_id>/invite| replace:: ``/rooms/<room_id>/invite`` +.. _/rooms/<room_id>/invite: /docs/api/client-server/#!/-rooms/invite + +.. |/rooms/<room_id>/join| replace:: ``/rooms/<room_id>/join`` +.. _/rooms/<room_id>/join: /docs/api/client-server/#!/-rooms/join_room + +.. |/rooms/<room_id>/leave| replace:: ``/rooms/<room_id>/leave`` +.. _/rooms/<room_id>/leave: /docs/api/client-server/#!/-rooms/leave + +.. |/rooms/<room_id>/ban| replace:: ``/rooms/<room_id>/ban`` +.. _/rooms/<room_id>/ban: /docs/api/client-server/#!/-rooms/ban + +.. |/join/<room_alias_or_id>| replace:: ``/join/<room_alias_or_id>`` +.. _/join/<room_alias_or_id>: /docs/api/client-server/#!/-rooms/join + +.. _`Event Stream`: /docs/api/client-server/#!/-events/get_event_stream diff --git a/experiments/cursesio.py b/experiments/cursesio.py index 31fbda5504..95d87a1fda 100644 --- a/experiments/cursesio.py +++ b/experiments/cursesio.py @@ -1,4 +1,4 @@ -# Copyright 2014 matrix.org +# Copyright 2014 OpenMarket Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. diff --git a/experiments/test_messaging.py b/experiments/test_messaging.py index 3ff7ab820f..fedf786cec 100644 --- a/experiments/test_messaging.py +++ b/experiments/test_messaging.py @@ -1,5 +1,5 @@ # -*- coding: utf-8 -*- -# Copyright 2014 matrix.org +# Copyright 2014 OpenMarket Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. diff --git a/graph/graph.py b/graph/graph.py index ac06d979e1..b2acadcf5e 100644 --- a/graph/graph.py +++ b/graph/graph.py @@ -1,4 +1,4 @@ -# Copyright 2014 matrix.org +# Copyright 2014 OpenMarket Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. diff --git a/jsfiddles/create_room_send_msg/demo.html b/jsfiddles/create_room_send_msg/demo.html index 31c26c7631..088ff7ac0f 100644 --- a/jsfiddles/create_room_send_msg/demo.html +++ b/jsfiddles/create_room_send_msg/demo.html @@ -1,5 +1,5 @@ <div> - <p>This room creation / message sending demo requires a home server to be running on http://localhost:8080</p> + <p>This room creation / message sending demo requires a home server to be running on http://localhost:8008</p> </div> <form class="loginForm"> <input type="text" id="userLogin" placeholder="Username"></input> diff --git a/jsfiddles/create_room_send_msg/demo.js b/jsfiddles/create_room_send_msg/demo.js index 61044da743..3dc7263830 100644 --- a/jsfiddles/create_room_send_msg/demo.js +++ b/jsfiddles/create_room_send_msg/demo.js @@ -10,7 +10,7 @@ $('.login').live('click', function() { var user = $("#userLogin").val(); var password = $("#passwordLogin").val(); $.ajax({ - url: "http://localhost:8080/_matrix/client/api/v1/login", + url: "http://localhost:8008/_matrix/client/api/v1/login", type: "POST", contentType: "application/json; charset=utf-8", data: JSON.stringify({ user: user, password: password, type: "m.login.password" }), @@ -25,7 +25,7 @@ $('.login').live('click', function() { }); var getCurrentRoomList = function() { - var url = "http://localhost:8080/_matrix/client/api/v1/initialSync?access_token=" + accountInfo.access_token + "&limit=1"; + var url = "http://localhost:8008/_matrix/client/api/v1/initialSync?access_token=" + accountInfo.access_token + "&limit=1"; $.getJSON(url, function(data) { var rooms = data.rooms; for (var i=0; i<rooms.length; ++i) { @@ -44,7 +44,7 @@ $('.createRoom').live('click', function() { data.room_alias_name = roomAlias; } $.ajax({ - url: "http://localhost:8080/_matrix/client/api/v1/createRoom?access_token="+accountInfo.access_token, + url: "http://localhost:8008/_matrix/client/api/v1/createRoom?access_token="+accountInfo.access_token, type: "POST", contentType: "application/json; charset=utf-8", data: JSON.stringify(data), @@ -79,7 +79,7 @@ $('.sendMessage').live('click', function() { return; } - var url = "http://localhost:8080/_matrix/client/api/v1/rooms/$roomid/send/m.room.message?access_token=$token"; + var url = "http://localhost:8008/_matrix/client/api/v1/rooms/$roomid/send/m.room.message?access_token=$token"; url = url.replace("$token", accountInfo.access_token); url = url.replace("$roomid", encodeURIComponent(roomId)); diff --git a/jsfiddles/event_stream/demo.html b/jsfiddles/event_stream/demo.html index ee4fc3ea68..7657780d28 100644 --- a/jsfiddles/event_stream/demo.html +++ b/jsfiddles/event_stream/demo.html @@ -1,5 +1,5 @@ <div> - <p>This event stream demo requires a home server to be running on http://localhost:8080</p> + <p>This event stream demo requires a home server to be running on http://localhost:8008</p> </div> <form class="loginForm"> <input type="text" id="userLogin" placeholder="Username"></input> diff --git a/jsfiddles/event_stream/demo.js b/jsfiddles/event_stream/demo.js index 997d1a2240..5c81e08caa 100644 --- a/jsfiddles/event_stream/demo.js +++ b/jsfiddles/event_stream/demo.js @@ -7,7 +7,7 @@ var eventStreamInfo = { var roomInfo = []; var longpollEventStream = function() { - var url = "http://localhost:8080/_matrix/client/api/v1/events?access_token=$token&from=$from"; + var url = "http://localhost:8008/_matrix/client/api/v1/events?access_token=$token&from=$from"; url = url.replace("$token", accountInfo.access_token); url = url.replace("$from", eventStreamInfo.from); @@ -48,7 +48,7 @@ $('.login').live('click', function() { var user = $("#userLogin").val(); var password = $("#passwordLogin").val(); $.ajax({ - url: "http://localhost:8080/_matrix/client/api/v1/login", + url: "http://localhost:8008/_matrix/client/api/v1/login", type: "POST", contentType: "application/json; charset=utf-8", data: JSON.stringify({ user: user, password: password, type: "m.login.password" }), @@ -65,7 +65,7 @@ $('.login').live('click', function() { var getCurrentRoomList = function() { $("#roomId").val(""); - var url = "http://localhost:8080/_matrix/client/api/v1/initialSync?access_token=" + accountInfo.access_token + "&limit=1"; + var url = "http://localhost:8008/_matrix/client/api/v1/initialSync?access_token=" + accountInfo.access_token + "&limit=1"; $.getJSON(url, function(data) { var rooms = data.rooms; for (var i=0; i<rooms.length; ++i) { @@ -98,7 +98,7 @@ var sendMessage = function(roomId) { return; } - var url = "http://localhost:8080/_matrix/client/api/v1/rooms/$roomid/send/m.room.message?access_token=$token"; + var url = "http://localhost:8008/_matrix/client/api/v1/rooms/$roomid/send/m.room.message?access_token=$token"; url = url.replace("$token", accountInfo.access_token); url = url.replace("$roomid", encodeURIComponent(roomId)); diff --git a/jsfiddles/example_app/demo.html b/jsfiddles/example_app/demo.html index 0af946f6bc..7a9dffddd0 100644 --- a/jsfiddles/example_app/demo.html +++ b/jsfiddles/example_app/demo.html @@ -1,5 +1,5 @@ <div class="signUp"> - <p>Matrix example application: Requires a local home server running at http://localhost:8080</p> + <p>Matrix example application: Requires a local home server running at http://localhost:8008</p> <form class="registrationForm"> <p>No account? Register:</p> <input type="text" id="userReg" placeholder="Username"></input> diff --git a/jsfiddles/example_app/demo.js b/jsfiddles/example_app/demo.js index 958232047f..ad79fcca26 100644 --- a/jsfiddles/example_app/demo.js +++ b/jsfiddles/example_app/demo.js @@ -10,7 +10,7 @@ var viewingRoomId; // ************** Event Streaming ************** var longpollEventStream = function() { - var url = "http://localhost:8080/_matrix/client/api/v1/events?access_token=$token&from=$from"; + var url = "http://localhost:8008/_matrix/client/api/v1/events?access_token=$token&from=$from"; url = url.replace("$token", accountInfo.access_token); url = url.replace("$from", eventStreamInfo.from); @@ -89,7 +89,7 @@ $('.login').live('click', function() { var user = $("#userLogin").val(); var password = $("#passwordLogin").val(); $.ajax({ - url: "http://localhost:8080/_matrix/client/api/v1/login", + url: "http://localhost:8008/_matrix/client/api/v1/login", type: "POST", contentType: "application/json; charset=utf-8", data: JSON.stringify({ user: user, password: password, type: "m.login.password" }), @@ -107,7 +107,7 @@ $('.register').live('click', function() { var user = $("#userReg").val(); var password = $("#passwordReg").val(); $.ajax({ - url: "http://localhost:8080/_matrix/client/api/v1/register", + url: "http://localhost:8008/_matrix/client/api/v1/register", type: "POST", contentType: "application/json; charset=utf-8", data: JSON.stringify({ user_id: user, password: password }), @@ -134,7 +134,7 @@ $('.createRoom').live('click', function() { data.room_alias_name = roomAlias; } $.ajax({ - url: "http://localhost:8080/_matrix/client/api/v1/createRoom?access_token="+accountInfo.access_token, + url: "http://localhost:8008/_matrix/client/api/v1/createRoom?access_token="+accountInfo.access_token, type: "POST", contentType: "application/json; charset=utf-8", data: JSON.stringify(data), @@ -155,7 +155,7 @@ $('.createRoom').live('click', function() { // ************** Getting current state ************** var getCurrentRoomList = function() { - var url = "http://localhost:8080/_matrix/client/api/v1/initialSync?access_token=" + accountInfo.access_token + "&limit=1"; + var url = "http://localhost:8008/_matrix/client/api/v1/initialSync?access_token=" + accountInfo.access_token + "&limit=1"; $.getJSON(url, function(data) { var rooms = data.rooms; for (var i=0; i<rooms.length; ++i) { @@ -181,7 +181,7 @@ var loadRoomContent = function(roomId) { var getMessages = function(roomId) { $("#messages").empty(); - var url = "http://localhost:8080/_matrix/client/api/v1/rooms/" + + var url = "http://localhost:8008/_matrix/client/api/v1/rooms/" + encodeURIComponent(roomId) + "/messages?access_token=" + accountInfo.access_token + "&from=END&dir=b&limit=10"; $.getJSON(url, function(data) { for (var i=data.chunk.length-1; i>=0; --i) { @@ -193,7 +193,7 @@ var getMessages = function(roomId) { var getMemberList = function(roomId) { $("#members").empty(); memberInfo = []; - var url = "http://localhost:8080/_matrix/client/api/v1/rooms/" + + var url = "http://localhost:8008/_matrix/client/api/v1/rooms/" + encodeURIComponent(roomId) + "/members?access_token=" + accountInfo.access_token; $.getJSON(url, function(data) { for (var i=0; i<data.chunk.length; ++i) { @@ -216,7 +216,7 @@ $('.sendMessage').live('click', function() { var sendMessage = function(roomId, body) { var msgId = $.now(); - var url = "http://localhost:8080/_matrix/client/api/v1/rooms/$roomid/send/m.room.message?access_token=$token"; + var url = "http://localhost:8008/_matrix/client/api/v1/rooms/$roomid/send/m.room.message?access_token=$token"; url = url.replace("$token", accountInfo.access_token); url = url.replace("$roomid", encodeURIComponent(roomId)); @@ -262,7 +262,7 @@ var setRooms = function(roomList) { var membership = $(this).find('td:eq(1)').text(); if (membership !== "join") { console.log("Joining room " + roomId); - var url = "http://localhost:8080/_matrix/client/api/v1/rooms/$roomid/join?access_token=$token"; + var url = "http://localhost:8008/_matrix/client/api/v1/rooms/$roomid/join?access_token=$token"; url = url.replace("$token", accountInfo.access_token); url = url.replace("$roomid", encodeURIComponent(roomId)); $.ajax({ @@ -290,6 +290,9 @@ var addMessage = function(data) { var msg = data.content.body; if (data.type === "m.room.member") { + if (data.content.membership === undefined) { + return; + } if (data.content.membership === "invite") { msg = "<em>invited " + data.state_key + " to the room</em>"; } @@ -299,10 +302,13 @@ var addMessage = function(data) { else if (data.content.membership === "leave") { msg = "<em>left the room</em>"; } - else { - msg = "<em>" + data.content.membership + "</em>"; + else if (data.content.membership === "ban") { + msg = "<em>was banned from the room</em>"; } } + if (msg === undefined) { + return; + } var row = "<tr>" + "<td>"+data.user_id+"</td>" + diff --git a/jsfiddles/register_login/demo.html b/jsfiddles/register_login/demo.html index 9cdb161306..fcac453ac2 100644 --- a/jsfiddles/register_login/demo.html +++ b/jsfiddles/register_login/demo.html @@ -1,5 +1,5 @@ <div> - <p>This registration/login demo requires a home server to be running on http://localhost:8080</p> + <p>This registration/login demo requires a home server to be running on http://localhost:8008</p> </div> <form class="registrationForm"> <input type="text" id="user" placeholder="Username"></input> diff --git a/jsfiddles/register_login/demo.js b/jsfiddles/register_login/demo.js index 1e68cb91bd..9595039173 100644 --- a/jsfiddles/register_login/demo.js +++ b/jsfiddles/register_login/demo.js @@ -11,7 +11,7 @@ $('.register').live('click', function() { var user = $("#user").val(); var password = $("#password").val(); $.ajax({ - url: "http://localhost:8080/_matrix/client/api/v1/register", + url: "http://localhost:8008/_matrix/client/api/v1/register", type: "POST", contentType: "application/json; charset=utf-8", data: JSON.stringify({ user_id: user, password: password }), @@ -27,7 +27,7 @@ $('.register').live('click', function() { var login = function(user, password) { $.ajax({ - url: "http://localhost:8080/_matrix/client/api/v1/login", + url: "http://localhost:8008/_matrix/client/api/v1/login", type: "POST", contentType: "application/json; charset=utf-8", data: JSON.stringify({ user: user, password: password, type: "m.login.password" }), @@ -44,7 +44,7 @@ var login = function(user, password) { $('.login').live('click', function() { var user = $("#userLogin").val(); var password = $("#passwordLogin").val(); - $.getJSON("http://localhost:8080/_matrix/client/api/v1/login", function(data) { + $.getJSON("http://localhost:8008/_matrix/client/api/v1/login", function(data) { if (data.flows[0].type !== "m.login.password") { alert("I don't know how to login with this type: " + data.type); return; @@ -60,7 +60,7 @@ $('.logout').live('click', function() { }); $('.testToken').live('click', function() { - var url = "http://localhost:8080/_matrix/client/api/v1/initialSync?access_token=" + accountInfo.access_token + "&limit=1"; + var url = "http://localhost:8008/_matrix/client/api/v1/initialSync?access_token=" + accountInfo.access_token + "&limit=1"; $.getJSON(url, function(data) { $("#imSyncText").text(JSON.stringify(data, undefined, 2)); }).fail(function(err) { diff --git a/jsfiddles/room_memberships/demo.html b/jsfiddles/room_memberships/demo.html index 4c1bf6b4bb..e6f39df5aa 100644 --- a/jsfiddles/room_memberships/demo.html +++ b/jsfiddles/room_memberships/demo.html @@ -1,5 +1,5 @@ <div> - <p>This room membership demo requires a home server to be running on http://localhost:8080</p> + <p>This room membership demo requires a home server to be running on http://localhost:8008</p> </div> <form class="loginForm"> <input type="text" id="userLogin" placeholder="Username"></input> diff --git a/jsfiddles/room_memberships/demo.js b/jsfiddles/room_memberships/demo.js index 7e499049ab..64ba767138 100644 --- a/jsfiddles/room_memberships/demo.js +++ b/jsfiddles/room_memberships/demo.js @@ -18,7 +18,7 @@ $('.login').live('click', function() { var user = $("#userLogin").val(); var password = $("#passwordLogin").val(); $.ajax({ - url: "http://localhost:8080/_matrix/client/api/v1/login", + url: "http://localhost:8008/_matrix/client/api/v1/login", type: "POST", contentType: "application/json; charset=utf-8", data: JSON.stringify({ user: user, password: password, type: "m.login.password" }), @@ -39,7 +39,7 @@ var getCurrentRoomList = function() { // solution but that is out of scope of this fiddle. $("#rooms").find("tr:gt(0)").remove(); - var url = "http://localhost:8080/_matrix/client/api/v1/initialSync?access_token=" + accountInfo.access_token + "&limit=1"; + var url = "http://localhost:8008/_matrix/client/api/v1/initialSync?access_token=" + accountInfo.access_token + "&limit=1"; $.getJSON(url, function(data) { var rooms = data.rooms; for (var i=0; i<rooms.length; ++i) { @@ -53,7 +53,7 @@ var getCurrentRoomList = function() { $('.createRoom').live('click', function() { var data = {}; $.ajax({ - url: "http://localhost:8080/_matrix/client/api/v1/createRoom?access_token="+accountInfo.access_token, + url: "http://localhost:8008/_matrix/client/api/v1/createRoom?access_token="+accountInfo.access_token, type: "POST", contentType: "application/json; charset=utf-8", data: JSON.stringify(data), @@ -87,7 +87,7 @@ $('.changeMembership').live('click', function() { return; } - var url = "http://localhost:8080/_matrix/client/api/v1/rooms/$roomid/$membership?access_token=$token"; + var url = "http://localhost:8008/_matrix/client/api/v1/rooms/$roomid/$membership?access_token=$token"; url = url.replace("$token", accountInfo.access_token); url = url.replace("$roomid", encodeURIComponent(roomId)); url = url.replace("$membership", membership); @@ -117,7 +117,7 @@ $('.changeMembership').live('click', function() { $('.joinAlias').live('click', function() { var roomAlias = $("#roomAlias").val(); - var url = "http://localhost:8080/_matrix/client/api/v1/join/$roomalias?access_token=$token"; + var url = "http://localhost:8008/_matrix/client/api/v1/join/$roomalias?access_token=$token"; url = url.replace("$token", accountInfo.access_token); url = url.replace("$roomalias", encodeURIComponent(roomAlias)); $.ajax({ diff --git a/scripts/basic.css b/scripts/basic.css new file mode 100644 index 0000000000..6411570ee6 --- /dev/null +++ b/scripts/basic.css @@ -0,0 +1,510 @@ +/* + * basic.css + * ~~~~~~~~~ + * + * Sphinx stylesheet -- basic theme. + * + * :copyright: Copyright 2007-2010 by the Sphinx team, see AUTHORS. + * :license: BSD, see LICENSE for details. + * + */ + +/* -- main layout ----------------------------------------------------------- */ + +div.clearer { + clear: both; +} + +/* -- relbar ---------------------------------------------------------------- */ + +div.related { + width: 100%; + font-size: 90%; +} + +div.related h3 { + display: none; +} + +div.related ul { + margin: 0; + padding: 0 0 0 10px; + list-style: none; +} + +div.related li { + display: inline; +} + +div.related li.right { + float: right; + margin-right: 5px; +} + +/* -- sidebar --------------------------------------------------------------- */ + +div.sphinxsidebarwrapper { + padding: 10px 5px 0 10px; +} + +div.sphinxsidebar { + float: left; + width: 230px; + margin-left: -100%; + font-size: 90%; +} + +div.sphinxsidebar ul { + list-style: none; +} + +div.sphinxsidebar ul ul, +div.sphinxsidebar ul.want-points { + margin-left: 20px; + list-style: square; +} + +div.sphinxsidebar ul ul { + margin-top: 0; + margin-bottom: 0; +} + +div.sphinxsidebar form { + margin-top: 10px; +} + +div.sphinxsidebar input { + border: 1px solid #98dbcc; + font-family: sans-serif; + font-size: 1em; +} + +img { + border: 0; +} + +/* -- search page ----------------------------------------------------------- */ + +ul.search { + margin: 10px 0 0 20px; + padding: 0; +} + +ul.search li { + padding: 5px 0 5px 20px; + background-image: url(file.png); + background-repeat: no-repeat; + background-position: 0 7px; +} + +ul.search li a { + font-weight: bold; +} + +ul.search li div.context { + color: #888; + margin: 2px 0 0 30px; + text-align: left; +} + +ul.keywordmatches li.goodmatch a { + font-weight: bold; +} + +/* -- index page ------------------------------------------------------------ */ + +table.contentstable { + width: 90%; +} + +table.contentstable p.biglink { + line-height: 150%; +} + +a.biglink { + font-size: 1.3em; +} + +span.linkdescr { + font-style: italic; + padding-top: 5px; + font-size: 90%; +} + +/* -- general index --------------------------------------------------------- */ + +table.indextable { + width: 100%; +} + +table.indextable td { + text-align: left; + vertical-align: top; +} + +table.indextable dl, table.indextable dd { + margin-top: 0; + margin-bottom: 0; +} + +table.indextable tr.pcap { + height: 10px; +} + +table.indextable tr.cap { + margin-top: 10px; + background-color: #f2f2f2; +} + +img.toggler { + margin-right: 3px; + margin-top: 3px; + cursor: pointer; +} + +div.modindex-jumpbox { + border-top: 1px solid #ddd; + border-bottom: 1px solid #ddd; + margin: 1em 0 1em 0; + padding: 0.4em; +} + +div.genindex-jumpbox { + border-top: 1px solid #ddd; + border-bottom: 1px solid #ddd; + margin: 1em 0 1em 0; + padding: 0.4em; +} + +/* -- general body styles --------------------------------------------------- */ + +a.headerlink { + visibility: hidden; +} + +h1:hover > a.headerlink, +h2:hover > a.headerlink, +h3:hover > a.headerlink, +h4:hover > a.headerlink, +h5:hover > a.headerlink, +h6:hover > a.headerlink, +dt:hover > a.headerlink { + visibility: visible; +} + +div.document p.caption { + text-align: inherit; +} + +div.document td { + text-align: left; +} + +.field-list ul { + padding-left: 1em; +} + +.first { + margin-top: 0 !important; +} + +p.rubric { + margin-top: 30px; + font-weight: bold; +} + +.align-left { + text-align: left; +} + +.align-center { + clear: both; + text-align: center; +} + +.align-right { + text-align: right; +} + +/* -- sidebars -------------------------------------------------------------- */ + +div.sidebar { + margin: 0 0 0.5em 1em; + border: 1px solid #ddb; + padding: 7px 7px 0 7px; + background-color: #ffe; + width: 40%; + float: right; +} + +p.sidebar-title { + font-weight: bold; +} + +/* -- topics ---------------------------------------------------------------- */ + +div.topic { + border: 1px solid #ccc; + padding: 7px 7px 0 7px; + margin: 10px 0 10px 0; +} + +p.topic-title { + font-size: 1.1em; + font-weight: bold; + margin-top: 10px; +} + +/* -- admonitions ----------------------------------------------------------- */ + +div.admonition { + margin-top: 10px; + margin-bottom: 10px; + padding: 7px; +} + +div.admonition dt { + font-weight: bold; +} + +div.admonition dl { + margin-bottom: 0; +} + +p.admonition-title { + margin: 0px 10px 5px 0px; + font-weight: bold; +} + +div.document p.centered { + text-align: center; + margin-top: 25px; +} + +/* -- tables ---------------------------------------------------------------- */ + +table.docutils { + border: 0; + border-collapse: collapse; +} + +table.docutils td, table.docutils th { + padding: 1px 8px 1px 5px; + border-top: 0; + border-left: 0; + border-right: 0; + border-bottom: 1px solid #aaa; +} + +table.field-list td, table.field-list th { + border: 0 !important; +} + +table.footnote td, table.footnote th { + border: 0 !important; +} + +th { + text-align: left; + padding-right: 5px; +} + +table.citation { + border-left: solid 1px gray; + margin-left: 1px; +} + +table.citation td { + border-bottom: none; +} + +/* -- other body styles ----------------------------------------------------- */ + +ol.arabic { + list-style: decimal; +} + +ol.loweralpha { + list-style: lower-alpha; +} + +ol.upperalpha { + list-style: upper-alpha; +} + +ol.lowerroman { + list-style: lower-roman; +} + +ol.upperroman { + list-style: upper-roman; +} + +dl { + margin-bottom: 15px; +} + +dd p { + margin-top: 0px; +} + +dd ul, dd table { + margin-bottom: 10px; +} + +dd { + margin-top: 3px; + margin-bottom: 10px; + margin-left: 30px; +} + +dt:target, .highlighted { + background-color: #fbe54e; +} + +dl.glossary dt { + font-weight: bold; + font-size: 1.1em; +} + +.field-list ul { + margin: 0; + padding-left: 1em; +} + +.field-list p { + margin: 0; +} + +.refcount { + color: #060; +} + +.optional { + font-size: 1.3em; +} + +.versionmodified { + font-style: italic; +} + +.system-message { + background-color: #fda; + padding: 5px; + border: 3px solid red; +} + +.footnote:target { + background-color: #ffa +} + +.line-block { + display: block; + margin-top: 1em; + margin-bottom: 1em; +} + +.line-block .line-block { + margin-top: 0; + margin-bottom: 0; + margin-left: 1.5em; +} + +.guilabel, .menuselection { + font-family: sans-serif; +} + +.accelerator { + text-decoration: underline; +} + +.classifier { + font-style: oblique; +} + +/* -- code displays --------------------------------------------------------- */ + +pre { + overflow: auto; +} + +td.linenos pre { + padding: 5px 0px; + border: 0; + background-color: transparent; + color: #aaa; +} + +table.highlighttable { + margin-left: 0.5em; +} + +table.highlighttable td { + padding: 0 0.5em 0 0.5em; +} + +tt.descname { + background-color: transparent; + font-weight: bold; + font-size: 1.2em; +} + +tt.descclassname { + background-color: transparent; +} + +tt.xref, a tt { + background-color: transparent; + font-weight: bold; +} + +h1 tt, h2 tt, h3 tt, h4 tt, h5 tt, h6 tt { + background-color: transparent; +} + +.viewcode-link { + float: right; +} + +.viewcode-back { + float: right; + font-family: sans-serif; +} + +div.viewcode-block:target { + margin: -1px -10px; + padding: 0 10px; +} + +/* -- math display ---------------------------------------------------------- */ + +img.math { + vertical-align: middle; +} + +div.document div.math p { + text-align: center; +} + +span.eqno { + float: right; +} + +/* -- printout stylesheet --------------------------------------------------- */ + +@media print { + div.document, + div.documentwrapper, + div.bodywrapper { + margin: 0 !important; + width: 100%; + } + + div.sphinxsidebar, + div.related, + div.footer, + #top-link { + display: none; + } +} + diff --git a/scripts/copyrighter.pl b/scripts/copyrighter.pl index e476c9cc85..7c03ef21fc 100755 --- a/scripts/copyrighter.pl +++ b/scripts/copyrighter.pl @@ -1,5 +1,5 @@ #!/usr/bin/perl -pi -# Copyright 2014 matrix.org +# Copyright 2014 OpenMarket Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. @@ -14,7 +14,7 @@ # limitations under the License. $copyright = <<EOT; -# Copyright 2014 matrix.org +# Copyright 2014 OpenMarket Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. diff --git a/scripts/gendoc.sh b/scripts/gendoc.sh new file mode 100755 index 0000000000..3c849e52e1 --- /dev/null +++ b/scripts/gendoc.sh @@ -0,0 +1,14 @@ +#!/bin/bash + +MATRIXDOTORG=$HOME/workspace/matrix.org + +rst2html-2.7.py --stylesheet=basic.css,nature.css ../docs/specification.rst > $MATRIXDOTORG/docs/spec/index.html +rst2html-2.7.py --stylesheet=basic.css,nature.css ../docs/client-server/howto.rst > $MATRIXDOTORG/docs/howtos/client-server.html + +perl -pi -e 's#<head>#<head><link rel="stylesheet" href="/site.css">#' $MATRIXDOTORG/docs/spec/index.html $MATRIXDOTORG/docs/howtos/client-server.html + +perl -pi -e 's#<body>#<body><div id="header"><div id="headerContent"> </div></div><div id="page"><div id="wrapper"><div style="text-align: center; padding: 40px;"><img src="/matrix.png" width="305" height="130" alt="[matrix]"/></div>#' $MATRIXDOTORG/docs/spec/index.html $MATRIXDOTORG/docs/howtos/client-server.html + +perl -pi -e 's#</body>#</div></div><div id="footer"><div id="footerContent">© 2014 Matrix.org</div></div></body>#' $MATRIXDOTORG/docs/spec/index.html $MATRIXDOTORG/docs/howtos/client-server.html + +scp -r $MATRIXDOTORG/docs matrix@ldc-prd-matrix-001:/sites/matrix-beta \ No newline at end of file diff --git a/scripts/nature.css b/scripts/nature.css new file mode 100644 index 0000000000..b8147f10ee --- /dev/null +++ b/scripts/nature.css @@ -0,0 +1,270 @@ +/* + * nature.css_t + * ~~~~~~~~~~~~ + * + * Sphinx stylesheet -- nature theme. + * + * :copyright: Copyright 2007-2010 by the Sphinx team, see AUTHORS. + * :license: BSD, see LICENSE for details. + * + */ + +/* -- page layout ----------------------------------------------------------- */ + +body { + font-family: Arial, sans-serif; + font-size: 100%; + /*background-color: #111;*/ + color: #555; + margin: 0; + padding: 0; +} + +div.documentwrapper { + float: left; + width: 100%; +} + +div.bodywrapper { + margin: 0 0 0 230px; +} + +hr { + border: 1px solid #B1B4B6; +} + +/* +div.document { + background-color: #eee; +} +*/ + +div.document { + background-color: #ffffff; + color: #3E4349; + padding: 0 30px 30px 30px; + font-size: 0.9em; +} + +div.footer { + color: #555; + width: 100%; + padding: 13px 0; + text-align: center; + font-size: 75%; +} + +div.footer a { + color: #444; + text-decoration: underline; +} + +div.related { + background-color: #6BA81E; + line-height: 32px; + color: #fff; + text-shadow: 0px 1px 0 #444; + font-size: 0.9em; +} + +div.related a { + color: #E2F3CC; +} + +div.sphinxsidebar { + font-size: 0.75em; + line-height: 1.5em; +} + +div.sphinxsidebarwrapper{ + padding: 20px 0; +} + +div.sphinxsidebar h3, +div.sphinxsidebar h4 { + font-family: Arial, sans-serif; + color: #222; + font-size: 1.2em; + font-weight: normal; + margin: 0; + padding: 5px 10px; + background-color: #ddd; + text-shadow: 1px 1px 0 white +} + +div.sphinxsidebar h4{ + font-size: 1.1em; +} + +div.sphinxsidebar h3 a { + color: #444; +} + + +div.sphinxsidebar p { + color: #888; + padding: 5px 20px; +} + +div.sphinxsidebar p.topless { +} + +div.sphinxsidebar ul { + margin: 10px 20px; + padding: 0; + color: #000; +} + +div.sphinxsidebar a { + color: #444; +} + +div.sphinxsidebar input { + border: 1px solid #ccc; + font-family: sans-serif; + font-size: 1em; +} + +div.sphinxsidebar input[type=text]{ + margin-left: 20px; +} + +/* -- body styles ----------------------------------------------------------- */ + +a { + color: #005B81; + text-decoration: none; +} + +a:hover { + color: #E32E00; + text-decoration: underline; +} + +div.document h1, +div.document h2, +div.document h3, +div.document h4, +div.document h5, +div.document h6 { + font-family: Arial, sans-serif; + background-color: #BED4EB; + font-weight: normal; + color: #212224; + margin: 30px 0px 10px 0px; + padding: 5px 0 5px 10px; + text-shadow: 0px 1px 0 white +} + +div.document h1 { border-top: 20px solid white; margin-top: 0; font-size: 200%; } +div.document h2 { font-size: 150%; background-color: #C8D5E3; } +div.document h3 { font-size: 120%; background-color: #D8DEE3; } +div.document h4 { font-size: 110%; background-color: #D8DEE3; } +div.document h5 { font-size: 100%; background-color: #D8DEE3; } +div.document h6 { font-size: 100%; background-color: #D8DEE3; } + +a.headerlink { + color: #c60f0f; + font-size: 0.8em; + padding: 0 4px 0 4px; + text-decoration: none; +} + +a.headerlink:hover { + background-color: #c60f0f; + color: white; +} + +div.document p, div.document dd, div.document li { + line-height: 1.5em; +} + +div.admonition p.admonition-title + p { + display: inline; +} + +div.highlight{ + background-color: white; +} + +div.note { + background-color: #eee; + border: 1px solid #ccc; +} + +div.seealso { + background-color: #ffc; + border: 1px solid #ff6; +} + +div.topic { + background-color: #eee; +} + +div.warning { + background-color: #ffe4e4; + border: 1px solid #f66; +} + +p.admonition-title { + display: inline; +} + +p.admonition-title:after { + content: ":"; +} + +pre { + padding: 10px; + background-color: White; + color: #222; + line-height: 1.2em; + border: 1px solid #C6C9CB; + font-size: 1.1em; + margin: 1.5em 0 1.5em 0; + -webkit-box-shadow: 1px 1px 1px #d8d8d8; + -moz-box-shadow: 1px 1px 1px #d8d8d8; +} + +tt { + background-color: #ecf0f3; + color: #222; + /* padding: 1px 2px; */ + font-size: 1.1em; + font-family: monospace; +} + +.viewcode-back { + font-family: Arial, sans-serif; +} + +div.viewcode-block:target { + background-color: #f4debf; + border-top: 1px solid #ac9; + border-bottom: 1px solid #ac9; +} + +p { + margin: 0; +} + +ul li dd { + margin-top: 0; +} + +ul li dl { + margin-bottom: 0; +} + +li dl dd { + margin-bottom: 0; +} + +dd ul { + padding-left: 0; +} + +li dd ul { + margin-bottom: 0; +} + diff --git a/setup.py b/setup.py index 0eec5c9354..a9470b4c9e 100755 --- a/setup.py +++ b/setup.py @@ -1,6 +1,6 @@ #!/usr/bin/env python -# Copyright 2014 matrix.org +# Copyright 2014 OpenMarket Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. diff --git a/synapse/__init__.py b/synapse/__init__.py index b45cf47b56..440e633966 100644 --- a/synapse/__init__.py +++ b/synapse/__init__.py @@ -1,5 +1,5 @@ # -*- coding: utf-8 -*- -# Copyright 2014 matrix.org +# Copyright 2014 OpenMarket Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. @@ -16,4 +16,4 @@ """ This is a reference implementation of a synapse home server. """ -__version__ = "0.2.0" +__version__ = "0.2.1" diff --git a/synapse/api/__init__.py b/synapse/api/__init__.py index 2216c0f1ca..9bff9ec169 100644 --- a/synapse/api/__init__.py +++ b/synapse/api/__init__.py @@ -1,5 +1,5 @@ # -*- coding: utf-8 -*- -# Copyright 2014 matrix.org +# Copyright 2014 OpenMarket Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. diff --git a/synapse/api/auth.py b/synapse/api/auth.py index 54ecbe5b3a..b4eda3df01 100644 --- a/synapse/api/auth.py +++ b/synapse/api/auth.py @@ -1,5 +1,5 @@ # -*- coding: utf-8 -*- -# Copyright 2014 matrix.org +# Copyright 2014 OpenMarket Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. diff --git a/synapse/api/constants.py b/synapse/api/constants.py index 668ffa07ca..fcef062fc9 100644 --- a/synapse/api/constants.py +++ b/synapse/api/constants.py @@ -1,5 +1,5 @@ # -*- coding: utf-8 -*- -# Copyright 2014 matrix.org +# Copyright 2014 OpenMarket Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. diff --git a/synapse/api/errors.py b/synapse/api/errors.py index 21ededc5ae..84afe4fa37 100644 --- a/synapse/api/errors.py +++ b/synapse/api/errors.py @@ -1,5 +1,5 @@ # -*- coding: utf-8 -*- -# Copyright 2014 matrix.org +# Copyright 2014 OpenMarket Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. @@ -28,6 +28,7 @@ class Codes(object): UNKNOWN = "M_UNKNOWN" NOT_FOUND = "M_NOT_FOUND" UNKNOWN_TOKEN = "M_UNKNOWN_TOKEN" + LIMIT_EXCEEDED = "M_LIMIT_EXCEEDED" class CodeMessageException(Exception): @@ -38,11 +39,15 @@ class CodeMessageException(Exception): super(CodeMessageException, self).__init__("%d: %s" % (code, msg)) self.code = code self.msg = msg + self.response_code_message = None + + def error_dict(self): + return cs_error(self.msg) class SynapseError(CodeMessageException): """A base error which can be caught for all synapse events.""" - def __init__(self, code, msg, errcode=""): + def __init__(self, code, msg, errcode=Codes.UNKNOWN): """Constructs a synapse error. Args: @@ -53,6 +58,11 @@ class SynapseError(CodeMessageException): super(SynapseError, self).__init__(code, msg) self.errcode = errcode + def error_dict(self): + return cs_error( + self.msg, + self.errcode, + ) class RoomError(SynapseError): """An error raised when a room event fails.""" @@ -91,13 +101,26 @@ class StoreError(SynapseError): pass -def cs_exception(exception): - if isinstance(exception, SynapseError): +class LimitExceededError(SynapseError): + """A client has sent too many requests and is being throttled. + """ + def __init__(self, code=429, msg="Too Many Requests", retry_after_ms=None, + errcode=Codes.LIMIT_EXCEEDED): + super(LimitExceededError, self).__init__(code, msg, errcode) + self.retry_after_ms = retry_after_ms + self.response_code_message = "Too Many Requests" + + def error_dict(self): return cs_error( - exception.msg, - Codes.UNKNOWN if not exception.errcode else exception.errcode) - elif isinstance(exception, CodeMessageException): - return cs_error(exception.msg) + self.msg, + self.errcode, + retry_after_ms=self.retry_after_ms, + ) + + +def cs_exception(exception): + if isinstance(exception, CodeMessageException): + return exception.error_dict() else: logging.error("Unknown exception type: %s", type(exception)) diff --git a/synapse/api/events/__init__.py b/synapse/api/events/__init__.py index 9502f5df8f..f95468fc65 100644 --- a/synapse/api/events/__init__.py +++ b/synapse/api/events/__init__.py @@ -1,5 +1,5 @@ # -*- coding: utf-8 -*- -# Copyright 2014 matrix.org +# Copyright 2014 OpenMarket Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. diff --git a/synapse/api/events/factory.py b/synapse/api/events/factory.py index 159728b2d2..a3b293e024 100644 --- a/synapse/api/events/factory.py +++ b/synapse/api/events/factory.py @@ -1,5 +1,5 @@ # -*- coding: utf-8 -*- -# Copyright 2014 matrix.org +# Copyright 2014 OpenMarket Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. diff --git a/synapse/api/events/room.py b/synapse/api/events/room.py index f6d3c59a9a..33f0f0cb99 100644 --- a/synapse/api/events/room.py +++ b/synapse/api/events/room.py @@ -1,5 +1,5 @@ # -*- coding: utf-8 -*- -# Copyright 2014 matrix.org +# Copyright 2014 OpenMarket Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. @@ -103,8 +103,7 @@ class FeedbackEvent(SynapseEvent): def get_content_template(self): return { "type": u"string", - "target_event_id": u"string", - "msg_sender_id": u"string" + "target_event_id": u"string" } diff --git a/synapse/api/ratelimiting.py b/synapse/api/ratelimiting.py new file mode 100644 index 0000000000..b25358090f --- /dev/null +++ b/synapse/api/ratelimiting.py @@ -0,0 +1,79 @@ +# Copyright 2014 OpenMarket Ltd +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import collections + + +class Ratelimiter(object): + """ + Ratelimit message sending by user. + """ + + def __init__(self): + self.message_counts = collections.OrderedDict() + + def send_message(self, user_id, time_now_s, msg_rate_hz, burst_count): + """Can the user send a message? + Args: + user_id: The user sending a message. + time_now_s: The time now. + msg_rate_hz: The long term number of messages a user can send in a + second. + burst_count: How many messages the user can send before being + limited. + Returns: + A pair of a bool indicating if they can send a message now and a + time in seconds of when they can next send a message. + """ + self.prune_message_counts(time_now_s) + message_count, time_start, _ignored = self.message_counts.pop( + user_id, (0., time_now_s, None), + ) + time_delta = time_now_s - time_start + sent_count = message_count - time_delta * msg_rate_hz + if sent_count < 0: + allowed = True + time_start = time_now_s + message_count = 1. + elif sent_count > burst_count - 1.: + allowed = False + else: + allowed = True + message_count += 1 + + self.message_counts[user_id] = ( + message_count, time_start, msg_rate_hz + ) + + if msg_rate_hz > 0: + time_allowed = ( + time_start + (message_count - burst_count + 1) / msg_rate_hz + ) + if time_allowed < time_now_s: + time_allowed = time_now_s + else: + time_allowed = -1 + + return allowed, time_allowed + + def prune_message_counts(self, time_now_s): + for user_id in self.message_counts.keys(): + message_count, time_start, msg_rate_hz = ( + self.message_counts[user_id] + ) + time_delta = time_now_s - time_start + if message_count - time_delta * msg_rate_hz > 0: + break + else: + del self.message_counts[user_id] diff --git a/synapse/api/urls.py b/synapse/api/urls.py index 3d0b5de965..6314f31f7a 100644 --- a/synapse/api/urls.py +++ b/synapse/api/urls.py @@ -1,5 +1,5 @@ # -*- coding: utf-8 -*- -# Copyright 2014 matrix.org +# Copyright 2014 OpenMarket Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. diff --git a/synapse/app/__init__.py b/synapse/app/__init__.py index 2216c0f1ca..9bff9ec169 100644 --- a/synapse/app/__init__.py +++ b/synapse/app/__init__.py @@ -1,5 +1,5 @@ # -*- coding: utf-8 -*- -# Copyright 2014 matrix.org +# Copyright 2014 OpenMarket Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. diff --git a/synapse/app/homeserver.py b/synapse/app/homeserver.py index 606c9c650d..49cf928cc1 100755 --- a/synapse/app/homeserver.py +++ b/synapse/app/homeserver.py @@ -1,6 +1,6 @@ #!/usr/bin/env python # -*- coding: utf-8 -*- -# Copyright 2014 matrix.org +# Copyright 2014 OpenMarket Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. @@ -23,7 +23,8 @@ from twisted.enterprise import adbapi from twisted.web.resource import Resource from twisted.web.static import File from twisted.web.server import Site -from synapse.http.server import JsonResource, RootRedirect, ContentRepoResource +from synapse.http.server import JsonResource, RootRedirect +from synapse.http.content_repository import ContentRepoResource from synapse.http.client import TwistedHttpClient from synapse.api.urls import ( CLIENT_PREFIX, FEDERATION_PREFIX, WEB_CLIENT_PREFIX, CONTENT_REPO_PREFIX @@ -74,7 +75,9 @@ class SynapseHomeServer(HomeServer): return File("webclient") # TODO configurable? def build_resource_for_content_repo(self): - return ContentRepoResource(self, self.upload_dir, self.auth) + return ContentRepoResource( + self, self.upload_dir, self.auth, self.content_addr + ) def build_db_pool(self): """ Set up all the dbs. Since all the *.sql have IF NOT EXISTS, so we @@ -90,20 +93,28 @@ class SynapseHomeServer(HomeServer): if row and row[0]: user_version = row[0] - if user_version < SCHEMA_VERSION: - # TODO(paul): add some kind of intelligent fixup here - raise ValueError("Cannot use this database as the " + - "schema version (%d) does not match (%d)" % - (user_version, SCHEMA_VERSION) + if user_version > SCHEMA_VERSION: + raise ValueError("Cannot use this database as it is too " + + "new for the server to understand" + ) + elif user_version < SCHEMA_VERSION: + logging.info("Upgrading database from version %d", + user_version ) + # Run every version since after the current version. + for v in range(user_version + 1, SCHEMA_VERSION + 1): + sql_script = read_schema("delta/v%d" % (v)) + c.executescript(sql_script) + + db_conn.commit() + else: for sql_loc in SCHEMAS: sql_script = read_schema(sql_loc) c.executescript(sql_script) - db_conn.commit() - + db_conn.commit() c.execute("PRAGMA user_version = %d" % SCHEMA_VERSION) c.close() @@ -247,6 +258,8 @@ def setup(): upload_dir=os.path.abspath("uploads"), db_name=config.database_path, tls_context_factory=tls_context_factory, + config=config, + content_addr=config.content_addr, ) hs.register_servlets() diff --git a/synapse/config/__init__.py b/synapse/config/__init__.py index fe8a073cd3..f9811bfa04 100644 --- a/synapse/config/__init__.py +++ b/synapse/config/__init__.py @@ -1,5 +1,5 @@ # -*- coding: utf-8 -*- -# Copyright 2014 matrix.org +# Copyright 2014 OpenMarket Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. diff --git a/synapse/config/_base.py b/synapse/config/_base.py index 1913179c3a..35bcece2c0 100644 --- a/synapse/config/_base.py +++ b/synapse/config/_base.py @@ -1,5 +1,5 @@ # -*- coding: utf-8 -*- -# Copyright 2014 matrix.org +# Copyright 2014 OpenMarket Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. @@ -13,8 +13,6 @@ # See the License for the specific language governing permissions and # limitations under the License. - -import ConfigParser as configparser import argparse import sys import os @@ -121,6 +119,9 @@ class Config(object): and value is not None): config[key] = value with open(config_args.config_path, "w") as config_file: + # TODO(paul) it would be lovely if we wrote out vim- and emacs- + # style mode markers into the file, to hint to people that + # this is a YAML file. yaml.dump(config, config_file, default_flow_style=False) sys.exit(0) diff --git a/synapse/config/database.py b/synapse/config/database.py index edf2361914..460445f15d 100644 --- a/synapse/config/database.py +++ b/synapse/config/database.py @@ -1,5 +1,5 @@ # -*- coding: utf-8 -*- -# Copyright 2014 matrix.org +# Copyright 2014 OpenMarket Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. diff --git a/synapse/config/homeserver.py b/synapse/config/homeserver.py index 18072e3196..76e2cdeddd 100644 --- a/synapse/config/homeserver.py +++ b/synapse/config/homeserver.py @@ -1,5 +1,5 @@ # -*- coding: utf-8 -*- -# Copyright 2014 matrix.org +# Copyright 2014 OpenMarket Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. @@ -17,8 +17,11 @@ from .tls import TlsConfig from .server import ServerConfig from .logger import LoggingConfig from .database import DatabaseConfig +from .ratelimiting import RatelimitConfig +from .repository import ContentRepositoryConfig -class HomeServerConfig(TlsConfig, ServerConfig, DatabaseConfig, LoggingConfig): +class HomeServerConfig(TlsConfig, ServerConfig, DatabaseConfig, LoggingConfig, + RatelimitConfig, ContentRepositoryConfig): pass if __name__=='__main__': diff --git a/synapse/config/logger.py b/synapse/config/logger.py index 8db6621ae8..56cd095433 100644 --- a/synapse/config/logger.py +++ b/synapse/config/logger.py @@ -1,5 +1,5 @@ # -*- coding: utf-8 -*- -# Copyright 2014 matrix.org +# Copyright 2014 OpenMarket Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. diff --git a/synapse/config/ratelimiting.py b/synapse/config/ratelimiting.py new file mode 100644 index 0000000000..f126782b8d --- /dev/null +++ b/synapse/config/ratelimiting.py @@ -0,0 +1,35 @@ +# Copyright 2014 OpenMarket Ltd +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from ._base import Config + +class RatelimitConfig(Config): + + def __init__(self, args): + super(RatelimitConfig, self).__init__(args) + self.rc_messages_per_second = args.rc_messages_per_second + self.rc_message_burst_count = args.rc_message_burst_count + + @classmethod + def add_arguments(cls, parser): + super(RatelimitConfig, cls).add_arguments(parser) + rc_group = parser.add_argument_group("ratelimiting") + rc_group.add_argument( + "--rc-messages-per-second", type=float, default=0.2, + help="number of messages a client can send per second" + ) + rc_group.add_argument( + "--rc-message-burst-count", type=float, default=10, + help="number of message a client can send before being throttled" + ) diff --git a/synapse/config/repository.py b/synapse/config/repository.py new file mode 100644 index 0000000000..407c8d6c24 --- /dev/null +++ b/synapse/config/repository.py @@ -0,0 +1,39 @@ +# -*- coding: utf-8 -*- +# Copyright 2014 matrix.org +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from ._base import Config +import os + +class ContentRepositoryConfig(Config): + def __init__(self, args): + super(ContentRepositoryConfig, self).__init__(args) + self.max_upload_size = self.parse_size(args.max_upload_size) + + def parse_size(self, string): + sizes = {"K": 1024, "M": 1024 * 1024} + size = 1 + suffix = string[-1] + if suffix in sizes: + string = string[:-1] + size = sizes[suffix] + return int(string) * size + + @classmethod + def add_arguments(cls, parser): + super(ContentRepositoryConfig, cls).add_arguments(parser) + db_group = parser.add_argument_group("content_repository") + db_group.add_argument( + "--max-upload-size", default="1M" + ) diff --git a/synapse/config/server.py b/synapse/config/server.py index 36143e3c9c..516e4cf882 100644 --- a/synapse/config/server.py +++ b/synapse/config/server.py @@ -1,5 +1,5 @@ # -*- coding: utf-8 -*- -# Copyright 2014 matrix.org +# Copyright 2014 OpenMarket Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. @@ -32,6 +32,14 @@ class ServerConfig(Config): self.webclient = True self.manhole = args.manhole + if not args.content_addr: + host = args.server_name + if ':' not in host: + host = "%s:%d" % (host, args.bind_port) + args.content_addr = "https://%s" % (host,) + + self.content_addr = args.content_addr + @classmethod def add_arguments(cls, parser): super(ServerConfig, cls).add_arguments(parser) @@ -50,13 +58,16 @@ class ServerConfig(Config): help="Local interface to listen on") server_group.add_argument("-D", "--daemonize", action='store_true', help="Daemonize the home server") - server_group.add_argument('--pid-file', default="hs.pid", + server_group.add_argument('--pid-file', default="homeserver.pid", help="When running as a daemon, the file to" " store the pid in") server_group.add_argument("--manhole", metavar="PORT", dest="manhole", type=int, help="Turn on the twisted telnet manhole" " service on the given port.") + server_group.add_argument("--content-addr", default=None, + help="The host and scheme to use for the " + "content repository") def read_signing_key(self, signing_key_path): signing_key_base64 = self.read_file(signing_key_path, "signing_key") @@ -77,3 +88,4 @@ class ServerConfig(Config): with open(args.signing_key_path, "w") as signing_key_file: key = nacl.signing.SigningKey.generate() signing_key_file.write(encode_base64(key.encode())) + diff --git a/synapse/config/tls.py b/synapse/config/tls.py index 16f6f3aba6..72d5518a89 100644 --- a/synapse/config/tls.py +++ b/synapse/config/tls.py @@ -1,5 +1,5 @@ # -*- coding: utf-8 -*- -# Copyright 2014 matrix.org +# Copyright 2014 OpenMarket Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. diff --git a/synapse/crypto/__init__.py b/synapse/crypto/__init__.py index 2216c0f1ca..9bff9ec169 100644 --- a/synapse/crypto/__init__.py +++ b/synapse/crypto/__init__.py @@ -1,5 +1,5 @@ # -*- coding: utf-8 -*- -# Copyright 2014 matrix.org +# Copyright 2014 OpenMarket Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. diff --git a/synapse/crypto/context_factory.py b/synapse/crypto/context_factory.py index 45958abbf5..f86bd19255 100644 --- a/synapse/crypto/context_factory.py +++ b/synapse/crypto/context_factory.py @@ -1,4 +1,18 @@ -from twisted.internet import reactor, ssl +# Copyright 2014 OpenMarket Ltd +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from twisted.internet import ssl from OpenSSL import SSL from twisted.internet._sslverify import _OpenSSLECCurve, _defaultCurveName diff --git a/synapse/crypto/keyclient.py b/synapse/crypto/keyclient.py index e615866b68..c11df5c529 100644 --- a/synapse/crypto/keyclient.py +++ b/synapse/crypto/keyclient.py @@ -1,5 +1,5 @@ # -*- coding: utf-8 -*- -# Copyright 2014 matrix.org +# Copyright 2014 OpenMarket Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. diff --git a/synapse/crypto/keyserver.py b/synapse/crypto/keyserver.py index 3d80a0e660..a23484dbae 100644 --- a/synapse/crypto/keyserver.py +++ b/synapse/crypto/keyserver.py @@ -1,5 +1,5 @@ # -*- coding: utf-8 -*- -# Copyright 2014 matrix.org +# Copyright 2014 OpenMarket Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. diff --git a/synapse/crypto/resource/__init__.py b/synapse/crypto/resource/__init__.py index 2216c0f1ca..9bff9ec169 100644 --- a/synapse/crypto/resource/__init__.py +++ b/synapse/crypto/resource/__init__.py @@ -1,5 +1,5 @@ # -*- coding: utf-8 -*- -# Copyright 2014 matrix.org +# Copyright 2014 OpenMarket Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. diff --git a/synapse/crypto/resource/key.py b/synapse/crypto/resource/key.py index 6aecd2b95f..48d14b9f4a 100644 --- a/synapse/crypto/resource/key.py +++ b/synapse/crypto/resource/key.py @@ -1,5 +1,5 @@ # -*- coding: utf-8 -*- -# Copyright 2014 matrix.org +# Copyright 2014 OpenMarket Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. diff --git a/synapse/federation/__init__.py b/synapse/federation/__init__.py index b15e7cf941..1351b68fd6 100644 --- a/synapse/federation/__init__.py +++ b/synapse/federation/__init__.py @@ -1,5 +1,5 @@ # -*- coding: utf-8 -*- -# Copyright 2014 matrix.org +# Copyright 2014 OpenMarket Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. diff --git a/synapse/federation/pdu_codec.py b/synapse/federation/pdu_codec.py index adc166c564..cef61108dd 100644 --- a/synapse/federation/pdu_codec.py +++ b/synapse/federation/pdu_codec.py @@ -1,5 +1,5 @@ # -*- coding: utf-8 -*- -# Copyright 2014 matrix.org +# Copyright 2014 OpenMarket Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. diff --git a/synapse/federation/persistence.py b/synapse/federation/persistence.py index 4cf72b2e42..de36a80e41 100644 --- a/synapse/federation/persistence.py +++ b/synapse/federation/persistence.py @@ -1,5 +1,5 @@ # -*- coding: utf-8 -*- -# Copyright 2014 matrix.org +# Copyright 2014 OpenMarket Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. diff --git a/synapse/federation/replication.py b/synapse/federation/replication.py index cadf574b3b..e12510017f 100644 --- a/synapse/federation/replication.py +++ b/synapse/federation/replication.py @@ -1,5 +1,5 @@ # -*- coding: utf-8 -*- -# Copyright 2014 matrix.org +# Copyright 2014 OpenMarket Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. @@ -628,7 +628,6 @@ class _TransactionQueue(object): for deferred in deferreds: deferred.errback(e) - yield deferred finally: # We want to be *very* sure we delete this after we stop processing diff --git a/synapse/federation/transport.py b/synapse/federation/transport.py index 50c3df4a5d..6e62ae7c74 100644 --- a/synapse/federation/transport.py +++ b/synapse/federation/transport.py @@ -1,5 +1,5 @@ # -*- coding: utf-8 -*- -# Copyright 2014 matrix.org +# Copyright 2014 OpenMarket Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. diff --git a/synapse/federation/units.py b/synapse/federation/units.py index b468f70546..9740431279 100644 --- a/synapse/federation/units.py +++ b/synapse/federation/units.py @@ -1,5 +1,5 @@ # -*- coding: utf-8 -*- -# Copyright 2014 matrix.org +# Copyright 2014 OpenMarket Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. diff --git a/synapse/handlers/__init__.py b/synapse/handlers/__init__.py index b2208b26c3..5308e2c8e1 100644 --- a/synapse/handlers/__init__.py +++ b/synapse/handlers/__init__.py @@ -1,5 +1,5 @@ # -*- coding: utf-8 -*- -# Copyright 2014 matrix.org +# Copyright 2014 OpenMarket Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. diff --git a/synapse/handlers/_base.py b/synapse/handlers/_base.py index b37c8be964..9989fe8670 100644 --- a/synapse/handlers/_base.py +++ b/synapse/handlers/_base.py @@ -1,5 +1,5 @@ # -*- coding: utf-8 -*- -# Copyright 2014 matrix.org +# Copyright 2014 OpenMarket Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. @@ -14,6 +14,7 @@ # limitations under the License. from twisted.internet import defer +from synapse.api.errors import LimitExceededError class BaseHandler(object): @@ -25,8 +26,22 @@ class BaseHandler(object): self.room_lock = hs.get_room_lock_manager() self.state_handler = hs.get_state_handler() self.distributor = hs.get_distributor() + self.ratelimiter = hs.get_ratelimiter() + self.clock = hs.get_clock() self.hs = hs + def ratelimit(self, user_id): + time_now = self.clock.time() + allowed, time_allowed = self.ratelimiter.send_message( + user_id, time_now, + msg_rate_hz=self.hs.config.rc_messages_per_second, + burst_count=self.hs.config.rc_message_burst_count, + ) + if not allowed: + raise LimitExceededError( + retry_after_ms=int(1000*(time_allowed - time_now)), + ) + class BaseRoomHandler(BaseHandler): diff --git a/synapse/handlers/directory.py b/synapse/handlers/directory.py index 7c89150d99..1b9e831fc0 100644 --- a/synapse/handlers/directory.py +++ b/synapse/handlers/directory.py @@ -1,5 +1,5 @@ # -*- coding: utf-8 -*- -# Copyright 2014 matrix.org +# Copyright 2014 OpenMarket Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. @@ -18,6 +18,7 @@ from twisted.internet import defer from ._base import BaseHandler from synapse.api.errors import SynapseError +from synapse.http.client import HttpClient import logging @@ -36,7 +37,7 @@ class DirectoryHandler(BaseHandler): ) @defer.inlineCallbacks - def create_association(self, room_alias, room_id, servers): + def create_association(self, room_alias, room_id, servers=None): # TODO(erikj): Do auth. if not room_alias.is_mine: @@ -47,6 +48,12 @@ class DirectoryHandler(BaseHandler): # TODO(erikj): Check if there is a current association. + if not servers: + servers = yield self.store.get_joined_hosts_for_room(room_id) + + if not servers: + raise SynapseError(400, "Failed to get server list") + yield self.store.create_room_alias_association( room_alias, room_id, @@ -68,7 +75,10 @@ class DirectoryHandler(BaseHandler): result = yield self.federation.make_query( destination=room_alias.domain, query_type="directory", - args={"room_alias": room_alias.to_string()}, + args={ + "room_alias": room_alias.to_string(), + HttpClient.RETRY_DNS_LOOKUP_FAILURES: False + } ) if result and "room_id" in result and "servers" in result: @@ -79,6 +89,9 @@ class DirectoryHandler(BaseHandler): defer.returnValue({}) return + extra_servers = yield self.store.get_joined_hosts_for_room(room_id) + servers = list(set(extra_servers) | set(servers)) + defer.returnValue({ "room_id": room_id, "servers": servers, diff --git a/synapse/handlers/events.py b/synapse/handlers/events.py index 980a169b25..fd24a11fb8 100644 --- a/synapse/handlers/events.py +++ b/synapse/handlers/events.py @@ -1,5 +1,5 @@ # -*- coding: utf-8 -*- -# Copyright 2014 matrix.org +# Copyright 2014 OpenMarket Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. @@ -126,5 +126,7 @@ class EventHandler(BaseHandler): defer.returnValue(None) return - yield self.auth.check(event, raises=True) + if hasattr(event, "room_id"): + yield self.auth.check_joined_room(event.room_id, user.to_string()) + defer.returnValue(event) diff --git a/synapse/handlers/federation.py b/synapse/handlers/federation.py index eac110419c..59cbf71d78 100644 --- a/synapse/handlers/federation.py +++ b/synapse/handlers/federation.py @@ -1,5 +1,5 @@ # -*- coding: utf-8 -*- -# Copyright 2014 matrix.org +# Copyright 2014 OpenMarket Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. @@ -21,8 +21,9 @@ from synapse.api.events.room import InviteJoinEvent, RoomMemberEvent from synapse.api.constants import Membership from synapse.util.logutils import log_function from synapse.federation.pdu_codec import PduCodec +from synapse.api.errors import SynapseError -from twisted.internet import defer +from twisted.internet import defer, reactor import logging @@ -133,7 +134,7 @@ class FederationHandler(BaseHandler): yield self.hs.get_handlers().room_member_handler.change_membership( new_event, - do_auth=True + do_auth=False, ) else: @@ -231,7 +232,12 @@ class FederationHandler(BaseHandler): # TODO (erikj): Time out here. d = defer.Deferred() self.waiting_for_join_list.setdefault((joinee, room_id), []).append(d) - yield d + reactor.callLater(10, d.cancel) + + try: + yield d + except defer.CancelledError: + raise SynapseError(500, "Unable to join remote room") try: yield self.store.store_room( diff --git a/synapse/handlers/login.py b/synapse/handlers/login.py index 0220fa0604..6ee7ce5a2d 100644 --- a/synapse/handlers/login.py +++ b/synapse/handlers/login.py @@ -1,5 +1,5 @@ # -*- coding: utf-8 -*- -# Copyright 2014 matrix.org +# Copyright 2014 OpenMarket Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. diff --git a/synapse/handlers/message.py b/synapse/handlers/message.py index 4aeb2089f5..dad2bbd1a4 100644 --- a/synapse/handlers/message.py +++ b/synapse/handlers/message.py @@ -1,5 +1,5 @@ # -*- coding: utf-8 -*- -# Copyright 2014 matrix.org +# Copyright 2014 OpenMarket Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. @@ -76,6 +76,8 @@ class MessageHandler(BaseRoomHandler): Raises: SynapseError if something went wrong. """ + + self.ratelimit(event.user_id) # TODO(paul): Why does 'event' not have a 'user' object? user = self.hs.parse_userid(event.user_id) assert user.is_mine, "User must be our own: %s" % (user,) @@ -140,7 +142,12 @@ class MessageHandler(BaseRoomHandler): SynapseError if something went wrong. """ - snapshot = yield self.store.snapshot_room(event.room_id, event.user_id) + snapshot = yield self.store.snapshot_room( + event.room_id, + event.user_id, + state_type=event.type, + state_key=event.state_key, + ) yield self.auth.check(event, snapshot, raises=True) diff --git a/synapse/handlers/presence.py b/synapse/handlers/presence.py index 9bfceda88a..c79bb6ff76 100644 --- a/synapse/handlers/presence.py +++ b/synapse/handlers/presence.py @@ -1,5 +1,5 @@ # -*- coding: utf-8 -*- -# Copyright 2014 matrix.org +# Copyright 2014 OpenMarket Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. @@ -155,19 +155,18 @@ class PresenceHandler(BaseHandler): if observer_user == observed_user: defer.returnValue(True) - allowed_by_subscription = yield self.store.is_presence_visible( + if (yield self.store.user_rooms_intersect( + [u.to_string() for u in observer_user, observed_user] + )): + defer.returnValue(True) + + if (yield self.store.is_presence_visible( observed_localpart=observed_user.localpart, observer_userid=observer_user.to_string(), - ) - - if allowed_by_subscription: + )): defer.returnValue(True) - share_room = yield self.store.do_users_share_a_room( - [observer_user, observed_user] - ) - - defer.returnValue(share_room) + defer.returnValue(False) @defer.inlineCallbacks def get_state(self, target_user, auth_user): @@ -181,7 +180,7 @@ class PresenceHandler(BaseHandler): state = yield self.store.get_presence_state(target_user.localpart) if "mtime" in state: del state["mtime"] - state["presence"] = state["state"] + state["presence"] = state.pop("state") if target_user in self._user_cachemap: state["last_active"] = ( @@ -208,21 +207,17 @@ class PresenceHandler(BaseHandler): raise SynapseError(400, "User is not hosted on this Home Server") if target_user != auth_user: - raise AuthError(400, "Cannot set another user's displayname") + raise AuthError(400, "Cannot set another user's presence") if "status_msg" not in state: state["status_msg"] = None for k in state.keys(): - if k not in ("presence", "state", "status_msg"): + if k not in ("presence", "status_msg"): raise SynapseError( 400, "Unexpected presence state key '%s'" % (k,) ) - # Handle legacy "state" key for now - if "state" in state: - state["presence"] = state.pop("state") - if state["presence"] not in self.STATE_LEVELS: raise SynapseError(400, "'%s' is not a valid presence state" % state["presence"] @@ -601,7 +596,7 @@ class PresenceHandler(BaseHandler): if state is None: state = yield self.store.get_presence_state(user.localpart) del state["mtime"] - state["presence"] = state["state"] + state["presence"] = state.pop("state") if user in self._user_cachemap: state["last_active"] = ( @@ -622,8 +617,6 @@ class PresenceHandler(BaseHandler): "user_id": user.to_string(), } user_state.update(**state) - if "state" in user_state and "presence" not in user_state: - user_state["presence"] = user_state["state"] yield self.federation.send_edu( destination=destination, @@ -655,21 +648,12 @@ class PresenceHandler(BaseHandler): state = dict(push) del state["user_id"] - if "presence" in state: - # all is OK - pass - elif "state" in state: - # Legacy handling - state["presence"] = state["state"] - else: + if "presence" not in state: logger.warning("Received a presence 'push' EDU from %s without" - + " either a 'presence' or 'state' key", origin + + " a 'presence' key", origin ) continue - if "state" in state: - del state["state"] - if "last_active_ago" in state: state["last_active"] = int( self.clock.time_msec() - state.pop("last_active_ago") @@ -773,15 +757,52 @@ class PresenceEventSource(object): self.hs = hs self.clock = hs.get_clock() + @defer.inlineCallbacks + def is_visible(self, observer_user, observed_user): + if observer_user == observed_user: + defer.returnValue(True) + + presence = self.hs.get_handlers().presence_handler + + if (yield presence.store.user_rooms_intersect( + [u.to_string() for u in observer_user, observed_user] + )): + defer.returnValue(True) + + if observed_user.is_mine: + pushmap = presence._local_pushmap + + defer.returnValue( + observed_user.localpart in pushmap and + observer_user in pushmap[observed_user.localpart] + ) + else: + recvmap = presence._remote_recvmap + + defer.returnValue( + observed_user in recvmap and + observer_user in recvmap[observed_user] + ) + + @defer.inlineCallbacks def get_new_events_for_user(self, user, from_key, limit): from_key = int(from_key) + observer_user = user + presence = self.hs.get_handlers().presence_handler cachemap = presence._user_cachemap - # TODO(paul): limit, and filter by visibility - updates = [(k, cachemap[k]) for k in cachemap - if from_key < cachemap[k].serial] + updates = [] + # TODO(paul): use a DeferredList ? How to limit concurrency. + for observed_user in cachemap.keys(): + if not (from_key < cachemap[observed_user].serial): + continue + + if (yield self.is_visible(observer_user, observed_user)): + updates.append((observed_user, cachemap[observed_user])) + + # TODO(paul): limit if updates: clock = self.clock @@ -789,20 +810,23 @@ class PresenceEventSource(object): latest_serial = max([x[1].serial for x in updates]) data = [x[1].make_event(user=x[0], clock=clock) for x in updates] - return ((data, latest_serial)) + defer.returnValue((data, latest_serial)) else: - return (([], presence._user_cachemap_latest_serial)) + defer.returnValue(([], presence._user_cachemap_latest_serial)) def get_current_key(self): presence = self.hs.get_handlers().presence_handler return presence._user_cachemap_latest_serial + @defer.inlineCallbacks def get_pagination_rows(self, user, pagination_config, key): # TODO (erikj): Does this make sense? Ordering? from_token = pagination_config.from_token to_token = pagination_config.to_token + observer_user = user + from_key = int(from_token.presence_key) if to_token: @@ -813,7 +837,17 @@ class PresenceEventSource(object): presence = self.hs.get_handlers().presence_handler cachemap = presence._user_cachemap - # TODO(paul): limit, and filter by visibility + updates = [] + # TODO(paul): use a DeferredList ? How to limit concurrency. + for observed_user in cachemap.keys(): + if not (to_key < cachemap[observed_user].serial < from_key): + continue + + if (yield self.is_visible(observer_user, observed_user)): + updates.append((observed_user, cachemap[observed_user])) + + # TODO(paul): limit + updates = [(k, cachemap[k]) for k in cachemap if to_key < cachemap[k].serial < from_key] @@ -831,13 +865,13 @@ class PresenceEventSource(object): next_token = next_token.copy_and_replace( "presence_key", earliest_serial ) - return ((data, next_token)) + defer.returnValue((data, next_token)) else: if not to_token: to_token = from_token.copy_and_replace( "presence_key", 0 ) - return (([], to_token)) + defer.returnValue(([], to_token)) class UserPresenceCache(object): @@ -851,7 +885,6 @@ class UserPresenceCache(object): def update(self, state, serial): assert("mtime_age" not in state) - assert("state" not in state) self.state.update(state) # Delete keys that are now 'None' @@ -869,11 +902,6 @@ class UserPresenceCache(object): def get_state(self): # clone it so caller can't break our cache state = dict(self.state) - - # Legacy handling - if "presence" in state: - state["state"] = state["presence"] - return state def make_event(self, user, clock): diff --git a/synapse/handlers/profile.py b/synapse/handlers/profile.py index 6799132054..023d8c0cf2 100644 --- a/synapse/handlers/profile.py +++ b/synapse/handlers/profile.py @@ -1,5 +1,5 @@ # -*- coding: utf-8 -*- -# Copyright 2014 matrix.org +# Copyright 2014 OpenMarket Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. diff --git a/synapse/handlers/register.py b/synapse/handlers/register.py index 593c603346..bee052274f 100644 --- a/synapse/handlers/register.py +++ b/synapse/handlers/register.py @@ -1,5 +1,5 @@ # -*- coding: utf-8 -*- -# Copyright 2014 matrix.org +# Copyright 2014 OpenMarket Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. @@ -20,9 +20,13 @@ from synapse.types import UserID from synapse.api.errors import SynapseError, RegistrationError from ._base import BaseHandler import synapse.util.stringutils as stringutils +from synapse.http.client import PlainHttpClient import base64 import bcrypt +import logging + +logger = logging.getLogger(__name__) class RegistrationHandler(BaseHandler): @@ -34,7 +38,7 @@ class RegistrationHandler(BaseHandler): self.distributor.declare("registered_user") @defer.inlineCallbacks - def register(self, localpart=None, password=None): + def register(self, localpart=None, password=None, threepidCreds=None): """Registers a new client on the server. Args: @@ -47,6 +51,20 @@ class RegistrationHandler(BaseHandler): Raises: RegistrationError if there was a problem registering. """ + + if threepidCreds: + for c in threepidCreds: + logger.info("validating theeepidcred sid %s on id server %s", c['sid'], c['idServer']) + try: + threepid = yield self._threepid_from_creds(c) + except: + logger.err() + raise RegistrationError(400, "Couldn't validate 3pid") + + if not threepid: + raise RegistrationError(400, "Couldn't validate 3pid") + logger.info("got threepid medium %s address %s", threepid['medium'], threepid['address']) + password_hash = None if password: password_hash = bcrypt.hashpw(password, bcrypt.gensalt()) @@ -61,7 +79,6 @@ class RegistrationHandler(BaseHandler): password_hash=password_hash) self.distributor.fire("registered_user", user) - defer.returnValue((user_id, token)) else: # autogen a random user ID attempts = 0 @@ -80,7 +97,6 @@ class RegistrationHandler(BaseHandler): password_hash=password_hash) self.distributor.fire("registered_user", user) - defer.returnValue((user_id, token)) except SynapseError: # if user id is taken, just generate another user_id = None @@ -90,6 +106,15 @@ class RegistrationHandler(BaseHandler): raise RegistrationError( 500, "Cannot generate user ID.") + # Now we have a matrix ID, bind it to the threepids we were given + if threepidCreds: + for c in threepidCreds: + # XXX: This should be a deferred list, shouldn't it? + yield self._bind_threepid(c, user_id) + + + defer.returnValue((user_id, token)) + def _generate_token(self, user_id): # urlsafe variant uses _ and - so use . as the separator and replace # all =s with .s so http clients don't quote =s when it is used as @@ -99,3 +124,34 @@ class RegistrationHandler(BaseHandler): def _generate_user_id(self): return "-" + stringutils.random_string(18) + + @defer.inlineCallbacks + def _threepid_from_creds(self, creds): + httpCli = PlainHttpClient(self.hs) + # XXX: make this configurable! + trustedIdServers = [ 'matrix.org:8090' ] + if not creds['idServer'] in trustedIdServers: + logger.warn('%s is not a trusted ID server: rejecting 3pid credentials', creds['idServer']) + defer.returnValue(None) + data = yield httpCli.get_json( + creds['idServer'], + "/_matrix/identity/api/v1/3pid/getValidated3pid", + { 'sid': creds['sid'], 'clientSecret': creds['clientSecret'] } + ) + + if 'medium' in data: + defer.returnValue(data) + defer.returnValue(None) + + @defer.inlineCallbacks + def _bind_threepid(self, creds, mxid): + httpCli = PlainHttpClient(self.hs) + data = yield httpCli.post_urlencoded_get_json( + creds['idServer'], + "/_matrix/identity/api/v1/3pid/bind", + { 'sid': creds['sid'], 'clientSecret': creds['clientSecret'], 'mxid':mxid } + ) + defer.returnValue(data) + + + diff --git a/synapse/handlers/room.py b/synapse/handlers/room.py index 53aa77405c..8171e9eb45 100644 --- a/synapse/handlers/room.py +++ b/synapse/handlers/room.py @@ -1,5 +1,5 @@ # -*- coding: utf-8 -*- -# Copyright 2014 matrix.org +# Copyright 2014 OpenMarket Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. @@ -49,6 +49,7 @@ class RoomCreationHandler(BaseRoomHandler): SynapseError if the room ID was taken, couldn't be stored, or something went horribly wrong. """ + self.ratelimit(user_id) if "room_alias_name" in config: room_alias = RoomAlias.create_local( @@ -110,8 +111,6 @@ class RoomCreationHandler(BaseRoomHandler): servers=[self.hs.hostname], ) - federation_handler = self.hs.get_handlers().federation_handler - @defer.inlineCallbacks def handle_event(event): snapshot = yield self.store.snapshot_room( @@ -138,6 +137,17 @@ class RoomCreationHandler(BaseRoomHandler): ) yield handle_event(name_event) + elif room_alias: + name = room_alias.to_string() + name_event = self.event_factory.create_event( + etype=RoomNameEvent.TYPE, + room_id=room_id, + user_id=user_id, + required_power_level=5, + content={"name": name}, + ) + + yield handle_event(name_event) if "topic" in config: topic = config["topic"] diff --git a/synapse/handlers/typing.py b/synapse/handlers/typing.py index 3268427ecd..0ca4e5c31e 100644 --- a/synapse/handlers/typing.py +++ b/synapse/handlers/typing.py @@ -1,5 +1,5 @@ # -*- coding: utf-8 -*- -# Copyright 2014 matrix.org +# Copyright 2014 OpenMarket Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. diff --git a/synapse/http/__init__.py b/synapse/http/__init__.py index 2216c0f1ca..9bff9ec169 100644 --- a/synapse/http/__init__.py +++ b/synapse/http/__init__.py @@ -1,5 +1,5 @@ # -*- coding: utf-8 -*- -# Copyright 2014 matrix.org +# Copyright 2014 OpenMarket Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. diff --git a/synapse/http/client.py b/synapse/http/client.py index 093bdf0e3f..ebf1aa47c4 100644 --- a/synapse/http/client.py +++ b/synapse/http/client.py @@ -1,5 +1,5 @@ # -*- coding: utf-8 -*- -# Copyright 2014 matrix.org +# Copyright 2014 OpenMarket Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. @@ -15,7 +15,8 @@ from twisted.internet import defer, reactor -from twisted.web.client import _AgentBase, _URI, readBody +from twisted.internet.error import DNSLookupError +from twisted.web.client import _AgentBase, _URI, readBody, FileBodyProducer from twisted.web.http_headers import Headers from synapse.http.endpoint import matrix_endpoint @@ -23,7 +24,9 @@ from synapse.util.async import sleep from syutil.jsonutil import encode_canonical_json -from synapse.api.errors import CodeMessageException +from synapse.api.errors import CodeMessageException, SynapseError + +from StringIO import StringIO import json import logging @@ -43,6 +46,7 @@ _destination_mappings = { class HttpClient(object): """ Interface for talking json over http """ + RETRY_DNS_LOOKUP_FAILURES = "__retry_dns" def put_json(self, destination, path, data): """ Sends the specifed json data using PUT @@ -142,13 +146,43 @@ class TwistedHttpClient(HttpClient): destination = _destination_mappings[destination] logger.debug("get_json args: %s", args) + + retry_on_dns_fail = True + if HttpClient.RETRY_DNS_LOOKUP_FAILURES in args: + # FIXME: This isn't ideal, but the interface exposed in get_json + # isn't comprehensive enough to give caller's any control over + # their connection mechanics. + retry_on_dns_fail = args.pop(HttpClient.RETRY_DNS_LOOKUP_FAILURES) + query_bytes = urllib.urlencode(args, True) + logger.debug("Query bytes: %s Retry DNS: %s", args, retry_on_dns_fail) response = yield self._create_request( destination.encode("ascii"), "GET", path.encode("ascii"), - query_bytes=query_bytes + query_bytes=query_bytes, + retry_on_dns_fail=retry_on_dns_fail + ) + + body = yield readBody(response) + + defer.returnValue(json.loads(body)) + + @defer.inlineCallbacks + def post_urlencoded_get_json(self, destination, path, args={}): + if destination in _destination_mappings: + destination = _destination_mappings[destination] + + logger.debug("post_urlencoded_get_json args: %s", args) + query_bytes = urllib.urlencode(args, True) + + response = yield self._create_request( + destination.encode("ascii"), + "POST", + path.encode("ascii"), + producer=FileBodyProducer(StringIO(urllib.urlencode(args))), + headers_dict={"Content-Type": ["application/x-www-form-urlencoded"]} ) body = yield readBody(response) @@ -157,7 +191,8 @@ class TwistedHttpClient(HttpClient): @defer.inlineCallbacks def _create_request(self, destination, method, path_bytes, param_bytes=b"", - query_bytes=b"", producer=None, headers_dict={}): + query_bytes=b"", producer=None, headers_dict={}, + retry_on_dns_fail=True): """ Creates and sends a request to the given url """ headers_dict[b"User-Agent"] = [b"Synapse"] @@ -178,10 +213,7 @@ class TwistedHttpClient(HttpClient): retries_left = 5 # TODO: setup and pass in an ssl_context to enable TLS - endpoint = matrix_endpoint( - reactor, destination, timeout=10, - ssl_context_factory=self.hs.tls_context_factory - ) + endpoint = self._getEndpoint(reactor, destination); while True: try: @@ -199,6 +231,11 @@ class TwistedHttpClient(HttpClient): logger.debug("Got response to %s", method) break except Exception as e: + if not retry_on_dns_fail and isinstance(e, DNSLookupError): + logger.warn("DNS Lookup failed to %s with %s", destination, + e) + raise SynapseError(400, "Domain specified not found.") + logger.exception("Got error in _create_request") _print_ex(e) @@ -223,6 +260,17 @@ class TwistedHttpClient(HttpClient): defer.returnValue(response) + def _getEndpoint(self, reactor, destination): + return matrix_endpoint( + reactor, destination, timeout=10, + ssl_context_factory=self.hs.tls_context_factory + ) + + +class PlainHttpClient(TwistedHttpClient): + def _getEndpoint(self, reactor, destination): + return matrix_endpoint(reactor, destination, timeout=10) + def _print_ex(e): if hasattr(e, "reasons") and e.reasons: diff --git a/synapse/http/content_repository.py b/synapse/http/content_repository.py new file mode 100644 index 0000000000..7dd4a859f8 --- /dev/null +++ b/synapse/http/content_repository.py @@ -0,0 +1,206 @@ +# -*- coding: utf-8 -*- +# Copyright 2014 OpenMarket Ltd +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from .server import respond_with_json_bytes + +from synapse.util.stringutils import random_string +from synapse.api.errors import ( + cs_exception, SynapseError, CodeMessageException, Codes, cs_error +) + +from twisted.protocols.basic import FileSender +from twisted.web import server, resource +from twisted.internet import defer + +import base64 +import json +import logging +import os +import re + +logger = logging.getLogger(__name__) + + +class ContentRepoResource(resource.Resource): + """Provides file uploading and downloading. + + Uploads are POSTed to wherever this Resource is linked to. This resource + returns a "content token" which can be used to GET this content again. The + token is typically a path, but it may not be. Tokens can expire, be one-time + uses, etc. + + In this case, the token is a path to the file and contains 3 interesting + sections: + - User ID base64d (for namespacing content to each user) + - random 24 char string + - Content type base64d (so we can return it when clients GET it) + + """ + isLeaf = True + + def __init__(self, hs, directory, auth, external_addr): + resource.Resource.__init__(self) + self.hs = hs + self.directory = directory + self.auth = auth + self.external_addr = external_addr.rstrip('/') + self.max_upload_size = hs.config.max_upload_size + + if not os.path.isdir(self.directory): + os.mkdir(self.directory) + logger.info("ContentRepoResource : Created %s directory.", + self.directory) + + @defer.inlineCallbacks + def map_request_to_name(self, request): + # auth the user + auth_user = yield self.auth.get_user_by_req(request) + + # namespace all file uploads on the user + prefix = base64.urlsafe_b64encode( + auth_user.to_string() + ).replace('=', '') + + # use a random string for the main portion + main_part = random_string(24) + + # suffix with a file extension if we can make one. This is nice to + # provide a hint to clients on the file information. We will also reuse + # this info to spit back the content type to the client. + suffix = "" + if request.requestHeaders.hasHeader("Content-Type"): + content_type = request.requestHeaders.getRawHeaders( + "Content-Type")[0] + suffix = "." + base64.urlsafe_b64encode(content_type) + if (content_type.split("/")[0].lower() in + ["image", "video", "audio"]): + file_ext = content_type.split("/")[-1] + # be a little paranoid and only allow a-z + file_ext = re.sub("[^a-z]", "", file_ext) + suffix += "." + file_ext + + file_name = prefix + main_part + suffix + file_path = os.path.join(self.directory, file_name) + logger.info("User %s is uploading a file to path %s", + auth_user.to_string(), + file_path) + + # keep trying to make a non-clashing file, with a sensible max attempts + attempts = 0 + while os.path.exists(file_path): + main_part = random_string(24) + file_name = prefix + main_part + suffix + file_path = os.path.join(self.directory, file_name) + attempts += 1 + if attempts > 25: # really? Really? + raise SynapseError(500, "Unable to create file.") + + defer.returnValue(file_path) + + def render_GET(self, request): + # no auth here on purpose, to allow anyone to view, even across home + # servers. + + # TODO: A little crude here, we could do this better. + filename = request.path.split('/')[-1] + # be paranoid + filename = re.sub("[^0-9A-z.-_]", "", filename) + + file_path = self.directory + "/" + filename + + logger.debug("Searching for %s", file_path) + + if os.path.isfile(file_path): + # filename has the content type + base64_contentype = filename.split(".")[1] + content_type = base64.urlsafe_b64decode(base64_contentype) + logger.info("Sending file %s", file_path) + f = open(file_path, 'rb') + request.setHeader('Content-Type', content_type) + d = FileSender().beginFileTransfer(f, request) + + # after the file has been sent, clean up and finish the request + def cbFinished(ignored): + f.close() + request.finish() + d.addCallback(cbFinished) + else: + respond_with_json_bytes( + request, + 404, + json.dumps(cs_error("Not found", code=Codes.NOT_FOUND)), + send_cors=True) + + return server.NOT_DONE_YET + + def render_POST(self, request): + self._async_render(request) + return server.NOT_DONE_YET + + def render_OPTIONS(self, request): + respond_with_json_bytes(request, 200, {}, send_cors=True) + return server.NOT_DONE_YET + + @defer.inlineCallbacks + def _async_render(self, request): + try: + # TODO: The checks here are a bit late. The content will have + # already been uploaded to a tmp file at this point + content_length = request.getHeader("Content-Length") + if content_length is None: + raise SynapseError( + msg="Request must specify a Content-Length", code=400 + ) + if int(content_length) > self.max_upload_size: + raise SynapseError( + msg="Upload request body is too large", + code=413, + ) + + fname = yield self.map_request_to_name(request) + + # TODO I have a suspcious feeling this is just going to block + with open(fname, "wb") as f: + f.write(request.content.read()) + + + # FIXME (erikj): These should use constants. + file_name = os.path.basename(fname) + # FIXME: we can't assume what the public mounted path of the repo is + # ...plus self-signed SSL won't work to remote clients anyway + # ...and we can't assume that it's SSL anyway, as we might want to + # server it via the non-SSL listener... + url = "%s/_matrix/content/%s" % ( + self.external_addr, file_name + ) + + respond_with_json_bytes(request, 200, + json.dumps({"content_token": url}), + send_cors=True) + + except CodeMessageException as e: + logger.exception(e) + respond_with_json_bytes(request, e.code, + json.dumps(cs_exception(e))) + except Exception as e: + logger.error("Failed to store file: %s" % e) + respond_with_json_bytes( + request, + 500, + json.dumps({"error": "Internal server error"}), + send_cors=True) + + + diff --git a/synapse/http/endpoint.py b/synapse/http/endpoint.py index 6c1fdcb853..7018ee3458 100644 --- a/synapse/http/endpoint.py +++ b/synapse/http/endpoint.py @@ -1,5 +1,5 @@ # -*- coding: utf-8 -*- -# Copyright 2014 matrix.org +# Copyright 2014 OpenMarket Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. diff --git a/synapse/http/server.py b/synapse/http/server.py index 0b87718bfa..8d419c02dd 100644 --- a/synapse/http/server.py +++ b/synapse/http/server.py @@ -1,5 +1,5 @@ # -*- coding: utf-8 -*- -# Copyright 2014 matrix.org +# Copyright 2014 OpenMarket Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. @@ -18,22 +18,16 @@ from syutil.jsonutil import ( encode_canonical_json, encode_pretty_printed_json ) from synapse.api.errors import ( - cs_exception, SynapseError, CodeMessageException, Codes, cs_error + cs_exception, SynapseError, CodeMessageException ) -from synapse.util.stringutils import random_string from twisted.internet import defer, reactor -from twisted.protocols.basic import FileSender from twisted.web import server, resource from twisted.web.server import NOT_DONE_YET from twisted.web.util import redirectTo -import base64 import collections -import json import logging -import os -import re logger = logging.getLogger(__name__) @@ -140,7 +134,8 @@ class JsonResource(HttpServer, resource.Resource): self._send_response( request, e.code, - cs_exception(e) + cs_exception(e), + response_code_message=e.response_code_message ) except Exception as e: logger.exception(e) @@ -150,7 +145,8 @@ class JsonResource(HttpServer, resource.Resource): {"error": "Internal server error"} ) - def _send_response(self, request, code, response_json_object): + def _send_response(self, request, code, response_json_object, + response_code_message=None): # could alternatively use request.notifyFinish() and flip a flag when # the Deferred fires, but since the flag is RIGHT THERE it seems like # a waste. @@ -166,7 +162,8 @@ class JsonResource(HttpServer, resource.Resource): json_bytes = encode_pretty_printed_json(response_json_object) # TODO: Only enable CORS for the requests that need it. - respond_with_json_bytes(request, code, json_bytes, send_cors=True) + respond_with_json_bytes(request, code, json_bytes, send_cors=True, + response_code_message=response_code_message) @staticmethod def _request_user_agent_is_curl(request): @@ -195,162 +192,8 @@ class RootRedirect(resource.Resource): return resource.Resource.getChild(self, name, request) -class ContentRepoResource(resource.Resource): - """Provides file uploading and downloading. - - Uploads are POSTed to wherever this Resource is linked to. This resource - returns a "content token" which can be used to GET this content again. The - token is typically a path, but it may not be. Tokens can expire, be one-time - uses, etc. - - In this case, the token is a path to the file and contains 3 interesting - sections: - - User ID base64d (for namespacing content to each user) - - random 24 char string - - Content type base64d (so we can return it when clients GET it) - - """ - isLeaf = True - - def __init__(self, hs, directory, auth): - resource.Resource.__init__(self) - self.hs = hs - self.directory = directory - self.auth = auth - - if not os.path.isdir(self.directory): - os.mkdir(self.directory) - logger.info("ContentRepoResource : Created %s directory.", - self.directory) - - @defer.inlineCallbacks - def map_request_to_name(self, request): - # auth the user - auth_user = yield self.auth.get_user_by_req(request) - - # namespace all file uploads on the user - prefix = base64.urlsafe_b64encode( - auth_user.to_string() - ).replace('=', '') - - # use a random string for the main portion - main_part = random_string(24) - - # suffix with a file extension if we can make one. This is nice to - # provide a hint to clients on the file information. We will also reuse - # this info to spit back the content type to the client. - suffix = "" - if request.requestHeaders.hasHeader("Content-Type"): - content_type = request.requestHeaders.getRawHeaders( - "Content-Type")[0] - suffix = "." + base64.urlsafe_b64encode(content_type) - if (content_type.split("/")[0].lower() in - ["image", "video", "audio"]): - file_ext = content_type.split("/")[-1] - # be a little paranoid and only allow a-z - file_ext = re.sub("[^a-z]", "", file_ext) - suffix += "." + file_ext - - file_name = prefix + main_part + suffix - file_path = os.path.join(self.directory, file_name) - logger.info("User %s is uploading a file to path %s", - auth_user.to_string(), - file_path) - - # keep trying to make a non-clashing file, with a sensible max attempts - attempts = 0 - while os.path.exists(file_path): - main_part = random_string(24) - file_name = prefix + main_part + suffix - file_path = os.path.join(self.directory, file_name) - attempts += 1 - if attempts > 25: # really? Really? - raise SynapseError(500, "Unable to create file.") - - defer.returnValue(file_path) - - def render_GET(self, request): - # no auth here on purpose, to allow anyone to view, even across home - # servers. - - # TODO: A little crude here, we could do this better. - filename = request.path.split('/')[-1] - # be paranoid - filename = re.sub("[^0-9A-z.-_]", "", filename) - - file_path = self.directory + "/" + filename - - logger.debug("Searching for %s", file_path) - - if os.path.isfile(file_path): - # filename has the content type - base64_contentype = filename.split(".")[1] - content_type = base64.urlsafe_b64decode(base64_contentype) - logger.info("Sending file %s", file_path) - f = open(file_path, 'rb') - request.setHeader('Content-Type', content_type) - d = FileSender().beginFileTransfer(f, request) - - # after the file has been sent, clean up and finish the request - def cbFinished(ignored): - f.close() - request.finish() - d.addCallback(cbFinished) - else: - respond_with_json_bytes( - request, - 404, - json.dumps(cs_error("Not found", code=Codes.NOT_FOUND)), - send_cors=True) - - return server.NOT_DONE_YET - - def render_POST(self, request): - self._async_render(request) - return server.NOT_DONE_YET - - def render_OPTIONS(self, request): - respond_with_json_bytes(request, 200, {}, send_cors=True) - return server.NOT_DONE_YET - - @defer.inlineCallbacks - def _async_render(self, request): - try: - fname = yield self.map_request_to_name(request) - - # TODO I have a suspcious feeling this is just going to block - with open(fname, "wb") as f: - f.write(request.content.read()) - - - # FIXME (erikj): These should use constants. - file_name = os.path.basename(fname) - # FIXME: we can't assume what the public mounted path of the repo is - # ...plus self-signed SSL won't work to remote clients anyway - # ...and we can't assume that it's SSL anyway, as we might want to - # server it via the non-SSL listener... - url = "https://%s/_matrix/content/%s" % ( - self.hs.domain_with_port, file_name - ) - - respond_with_json_bytes(request, 200, - json.dumps({"content_token": url}), - send_cors=True) - - except CodeMessageException as e: - logger.exception(e) - respond_with_json_bytes(request, e.code, - json.dumps(cs_exception(e))) - except Exception as e: - logger.error("Failed to store file: %s" % e) - respond_with_json_bytes( - request, - 500, - json.dumps({"error": "Internal server error"}), - send_cors=True) - - -def respond_with_json_bytes(request, code, json_bytes, send_cors=False): +def respond_with_json_bytes(request, code, json_bytes, send_cors=False, + response_code_message=None): """Sends encoded JSON in response to the given request. Args: @@ -362,7 +205,7 @@ def respond_with_json_bytes(request, code, json_bytes, send_cors=False): Returns: twisted.web.server.NOT_DONE_YET""" - request.setResponseCode(code) + request.setResponseCode(code, message=response_code_message) request.setHeader(b"Content-Type", b"application/json") if send_cors: diff --git a/synapse/notifier.py b/synapse/notifier.py index 3260aa744f..5b02c71d1e 100644 --- a/synapse/notifier.py +++ b/synapse/notifier.py @@ -1,5 +1,5 @@ # -*- coding: utf-8 -*- -# Copyright 2014 matrix.org +# Copyright 2014 OpenMarket Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. @@ -167,7 +167,12 @@ class Notifier(object): ) def eb(failure): - logger.exception("Failed to notify listener", failure) + logger.error("Failed to notify listener", + exc_info=( + failure.type, + failure.value, + failure.getTracebackObject()) + ) yield defer.DeferredList( [notify(l).addErrback(eb) for l in listeners] diff --git a/synapse/rest/__init__.py b/synapse/rest/__init__.py index f33024e72a..ed785cfbd5 100644 --- a/synapse/rest/__init__.py +++ b/synapse/rest/__init__.py @@ -1,5 +1,5 @@ # -*- coding: utf-8 -*- -# Copyright 2014 matrix.org +# Copyright 2014 OpenMarket Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. diff --git a/synapse/rest/base.py b/synapse/rest/base.py index e855d293e5..2e8e3fa7d4 100644 --- a/synapse/rest/base.py +++ b/synapse/rest/base.py @@ -1,5 +1,5 @@ # -*- coding: utf-8 -*- -# Copyright 2014 matrix.org +# Copyright 2014 OpenMarket Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. diff --git a/synapse/rest/directory.py b/synapse/rest/directory.py index dc347652a0..18df7c8d8b 100644 --- a/synapse/rest/directory.py +++ b/synapse/rest/directory.py @@ -1,5 +1,5 @@ # -*- coding: utf-8 -*- -# Copyright 2014 matrix.org +# Copyright 2014 OpenMarket Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. @@ -16,6 +16,7 @@ from twisted.internet import defer +from synapse.api.errors import SynapseError, Codes from base import RestServlet, client_path_pattern import json @@ -44,8 +45,10 @@ class ClientDirectoryServer(RestServlet): @defer.inlineCallbacks def on_PUT(self, request, room_alias): - # TODO(erikj): Exceptions - content = json.loads(request.content.read()) + content = _parse_json(request) + if not "room_id" in content: + raise SynapseError(400, "Missing room_id key", + errcode=Codes.BAD_JSON) logger.debug("Got content: %s", content) @@ -54,7 +57,7 @@ class ClientDirectoryServer(RestServlet): logger.debug("Got room name: %s", room_alias.to_string()) room_id = content["room_id"] - servers = content["servers"] + servers = content["servers"] if "servers" in content else None logger.debug("Got room_id: %s", room_id) logger.debug("Got servers: %s", servers) @@ -68,7 +71,20 @@ class ClientDirectoryServer(RestServlet): yield dir_handler.create_association( room_alias, room_id, servers ) + except SynapseError as e: + raise e except: logger.exception("Failed to create association") defer.returnValue((200, {})) + + +def _parse_json(request): + try: + content = json.loads(request.content.read()) + if type(content) != dict: + raise SynapseError(400, "Content must be a JSON object.", + errcode=Codes.NOT_JSON) + return content + except ValueError: + raise SynapseError(400, "Content not JSON.", errcode=Codes.NOT_JSON) diff --git a/synapse/rest/events.py b/synapse/rest/events.py index 2e7563d14b..7fde143200 100644 --- a/synapse/rest/events.py +++ b/synapse/rest/events.py @@ -1,5 +1,5 @@ # -*- coding: utf-8 -*- -# Copyright 2014 matrix.org +# Copyright 2014 OpenMarket Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. diff --git a/synapse/rest/initial_sync.py b/synapse/rest/initial_sync.py index d18c4c0f60..a1cb442256 100644 --- a/synapse/rest/initial_sync.py +++ b/synapse/rest/initial_sync.py @@ -1,5 +1,5 @@ # -*- coding: utf-8 -*- -# Copyright 2014 matrix.org +# Copyright 2014 OpenMarket Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. diff --git a/synapse/rest/login.py b/synapse/rest/login.py index 99e4f10aac..c7bf901c8e 100644 --- a/synapse/rest/login.py +++ b/synapse/rest/login.py @@ -1,5 +1,5 @@ # -*- coding: utf-8 -*- -# Copyright 2014 matrix.org +# Copyright 2014 OpenMarket Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. diff --git a/synapse/rest/presence.py b/synapse/rest/presence.py index bce3943542..7fc8ce4404 100644 --- a/synapse/rest/presence.py +++ b/synapse/rest/presence.py @@ -1,5 +1,5 @@ # -*- coding: utf-8 -*- -# Copyright 2014 matrix.org +# Copyright 2014 OpenMarket Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. @@ -17,11 +17,12 @@ """ from twisted.internet import defer +from synapse.api.errors import SynapseError from base import RestServlet, client_path_pattern import json import logging - +import urllib logger = logging.getLogger(__name__) @@ -32,6 +33,7 @@ class PresenceStatusRestServlet(RestServlet): @defer.inlineCallbacks def on_GET(self, request, user_id): auth_user = yield self.auth.get_user_by_req(request) + user_id = urllib.unquote(user_id) user = self.hs.parse_userid(user_id) state = yield self.handlers.presence_handler.get_state( @@ -42,25 +44,26 @@ class PresenceStatusRestServlet(RestServlet): @defer.inlineCallbacks def on_PUT(self, request, user_id): auth_user = yield self.auth.get_user_by_req(request) + user_id = urllib.unquote(user_id) user = self.hs.parse_userid(user_id) state = {} try: content = json.loads(request.content.read()) - # Legacy handling - if "state" in content: - state["presence"] = content.pop("state") - else: - state["presence"] = content.pop("presence") + state["presence"] = content.pop("presence") if "status_msg" in content: state["status_msg"] = content.pop("status_msg") + if not isinstance(state["status_msg"], basestring): + raise SynapseError(400, "status_msg must be a string.") if content: raise KeyError() + except SynapseError as e: + raise e except: - defer.returnValue((400, "Unable to parse state")) + raise SynapseError(400, "Unable to parse state") yield self.handlers.presence_handler.set_state( target_user=user, auth_user=auth_user, state=state) @@ -77,13 +80,14 @@ class PresenceListRestServlet(RestServlet): @defer.inlineCallbacks def on_GET(self, request, user_id): auth_user = yield self.auth.get_user_by_req(request) + user_id = urllib.unquote(user_id) user = self.hs.parse_userid(user_id) if not user.is_mine: - defer.returnValue((400, "User not hosted on this Home Server")) + raise SynapseError(400, "User not hosted on this Home Server") if auth_user != user: - defer.returnValue((400, "Cannot get another user's presence list")) + raise SynapseError(400, "Cannot get another user's presence list") presence = yield self.handlers.presence_handler.get_presence_list( observer_user=user, accepted=True) @@ -97,31 +101,40 @@ class PresenceListRestServlet(RestServlet): @defer.inlineCallbacks def on_POST(self, request, user_id): auth_user = yield self.auth.get_user_by_req(request) + user_id = urllib.unquote(user_id) user = self.hs.parse_userid(user_id) if not user.is_mine: - defer.returnValue((400, "User not hosted on this Home Server")) + raise SynapseError(400, "User not hosted on this Home Server") if auth_user != user: - defer.returnValue(( - 400, "Cannot modify another user's presence list")) + raise SynapseError( + 400, "Cannot modify another user's presence list") try: content = json.loads(request.content.read()) except: logger.exception("JSON parse error") - defer.returnValue((400, "Unable to parse content")) + raise SynapseError(400, "Unable to parse content") deferreds = [] if "invite" in content: for u in content["invite"]: + if not isinstance(u, basestring): + raise SynapseError(400, "Bad invite value.") + if len(u) == 0: + continue invited_user = self.hs.parse_userid(u) deferreds.append(self.handlers.presence_handler.send_invite( observer_user=user, observed_user=invited_user)) if "drop" in content: for u in content["drop"]: + if not isinstance(u, basestring): + raise SynapseError(400, "Bad drop value.") + if len(u) == 0: + continue dropped_user = self.hs.parse_userid(u) deferreds.append(self.handlers.presence_handler.drop( observer_user=user, observed_user=dropped_user)) diff --git a/synapse/rest/profile.py b/synapse/rest/profile.py index 06076667c7..2e17f87fa1 100644 --- a/synapse/rest/profile.py +++ b/synapse/rest/profile.py @@ -1,5 +1,5 @@ # -*- coding: utf-8 -*- -# Copyright 2014 matrix.org +# Copyright 2014 OpenMarket Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. @@ -19,6 +19,7 @@ from twisted.internet import defer from base import RestServlet, client_path_pattern import json +import urllib class ProfileDisplaynameRestServlet(RestServlet): @@ -26,6 +27,7 @@ class ProfileDisplaynameRestServlet(RestServlet): @defer.inlineCallbacks def on_GET(self, request, user_id): + user_id = urllib.unquote(user_id) user = self.hs.parse_userid(user_id) displayname = yield self.handlers.profile_handler.get_displayname( @@ -37,6 +39,7 @@ class ProfileDisplaynameRestServlet(RestServlet): @defer.inlineCallbacks def on_PUT(self, request, user_id): auth_user = yield self.auth.get_user_by_req(request) + user_id = urllib.unquote(user_id) user = self.hs.parse_userid(user_id) try: @@ -59,6 +62,7 @@ class ProfileAvatarURLRestServlet(RestServlet): @defer.inlineCallbacks def on_GET(self, request, user_id): + user_id = urllib.unquote(user_id) user = self.hs.parse_userid(user_id) avatar_url = yield self.handlers.profile_handler.get_avatar_url( @@ -70,6 +74,7 @@ class ProfileAvatarURLRestServlet(RestServlet): @defer.inlineCallbacks def on_PUT(self, request, user_id): auth_user = yield self.auth.get_user_by_req(request) + user_id = urllib.unquote(user_id) user = self.hs.parse_userid(user_id) try: @@ -92,6 +97,7 @@ class ProfileRestServlet(RestServlet): @defer.inlineCallbacks def on_GET(self, request, user_id): + user_id = urllib.unquote(user_id) user = self.hs.parse_userid(user_id) displayname = yield self.handlers.profile_handler.get_displayname( diff --git a/synapse/rest/register.py b/synapse/rest/register.py index f17ec11cf4..b8de3b250d 100644 --- a/synapse/rest/register.py +++ b/synapse/rest/register.py @@ -1,5 +1,5 @@ # -*- coding: utf-8 -*- -# Copyright 2014 matrix.org +# Copyright 2014 OpenMarket Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. @@ -47,10 +47,15 @@ class RegisterRestServlet(RestServlet): except KeyError: pass # user_id is optional + threepidCreds = None + if 'threepidCreds' in register_json: + threepidCreds = register_json['threepidCreds'] + handler = self.handlers.registration_handler (user_id, token) = yield handler.register( localpart=desired_user_id, - password=password) + password=password, + threepidCreds=threepidCreds) result = { "user_id": user_id, diff --git a/synapse/rest/room.py b/synapse/rest/room.py index a10b3b54f9..308b447090 100644 --- a/synapse/rest/room.py +++ b/synapse/rest/room.py @@ -1,5 +1,5 @@ # -*- coding: utf-8 -*- -# Copyright 2014 matrix.org +# Copyright 2014 OpenMarket Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. @@ -388,7 +388,7 @@ class RoomMembershipRestServlet(RestServlet): def register(self, http_server): # /rooms/$roomid/[invite|join|leave] PATTERN = ("/rooms/(?P<room_id>[^/]*)/" + - "(?P<membership_action>join|invite|leave)") + "(?P<membership_action>join|invite|leave|ban|kick)") register_txn_path(self, PATTERN, http_server) @defer.inlineCallbacks @@ -399,11 +399,14 @@ class RoomMembershipRestServlet(RestServlet): # target user is you unless it is an invite state_key = user.to_string() - if membership_action == "invite": + if membership_action in ["invite", "ban", "kick"]: if "user_id" not in content: raise SynapseError(400, "Missing user_id key.") state_key = content["user_id"] + if membership_action == "kick": + membership_action = "leave" + event = self.event_factory.create_event( etype=RoomMemberEvent.TYPE, content={"membership": unicode(membership_action)}, diff --git a/synapse/rest/transactions.py b/synapse/rest/transactions.py index b8aa1ef11c..e06dcc8c57 100644 --- a/synapse/rest/transactions.py +++ b/synapse/rest/transactions.py @@ -1,5 +1,5 @@ # -*- coding: utf-8 -*- -# Copyright 2014 matrix.org +# Copyright 2014 OpenMarket Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. diff --git a/synapse/server.py b/synapse/server.py index 3e72b2bcd5..83368ea5a7 100644 --- a/synapse/server.py +++ b/synapse/server.py @@ -1,5 +1,5 @@ # -*- coding: utf-8 -*- -# Copyright 2014 matrix.org +# Copyright 2014 OpenMarket Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. @@ -32,6 +32,7 @@ from synapse.util import Clock from synapse.util.distributor import Distributor from synapse.util.lockutils import LockManager from synapse.streams.events import EventSources +from synapse.api.ratelimiting import Ratelimiter class BaseHomeServer(object): @@ -73,6 +74,7 @@ class BaseHomeServer(object): 'resource_for_web_client', 'resource_for_content_repo', 'event_sources', + 'ratelimiter', ] def __init__(self, hostname, **kwargs): @@ -190,6 +192,9 @@ class HomeServer(BaseHomeServer): def build_event_sources(self): return EventSources(self) + def build_ratelimiter(self): + return Ratelimiter() + def register_servlets(self): """ Register all servlets associated with this HomeServer. """ diff --git a/synapse/state.py b/synapse/state.py index e1a1a159bb..36d8210eb5 100644 --- a/synapse/state.py +++ b/synapse/state.py @@ -1,5 +1,5 @@ # -*- coding: utf-8 -*- -# Copyright 2014 matrix.org +# Copyright 2014 OpenMarket Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. @@ -179,6 +179,18 @@ class StateHandler(object): key=lambda x: x.depth ) + if not hasattr(missing_prev, "prev_state_id"): + # FIXME Hmm + # temporary fallback + for algo in conflict_res: + new_res, curr_res = algo(new_branch, current_branch) + + if new_res < curr_res: + defer.returnValue(False) + elif new_res > curr_res: + defer.returnValue(True) + return + pdu_id = missing_prev.prev_state_id origin = missing_prev.prev_state_origin diff --git a/synapse/storage/__init__.py b/synapse/storage/__init__.py index aadaab06e7..d97014f4da 100644 --- a/synapse/storage/__init__.py +++ b/synapse/storage/__init__.py @@ -1,5 +1,5 @@ # -*- coding: utf-8 -*- -# Copyright 2014 matrix.org +# Copyright 2014 OpenMarket Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. diff --git a/synapse/storage/_base.py b/synapse/storage/_base.py index 33d56f47ce..bae50e7d1f 100644 --- a/synapse/storage/_base.py +++ b/synapse/storage/_base.py @@ -1,5 +1,5 @@ # -*- coding: utf-8 -*- -# Copyright 2014 matrix.org +# Copyright 2014 OpenMarket Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. @@ -79,19 +79,21 @@ class SQLBaseStore(object): # "Simple" SQL API methods that operate on a single table with no JOINs, # no complex WHERE clauses, just a dict of values for columns. - def _simple_insert(self, table, values): + def _simple_insert(self, table, values, or_replace=False): """Executes an INSERT query on the named table. Args: table : string giving the table name values : dict of new column names and values for them + or_replace : bool; if True performs an INSERT OR REPLACE """ return self._db_pool.runInteraction( - self._simple_insert_txn, table, values, + self._simple_insert_txn, table, values, or_replace=or_replace ) - def _simple_insert_txn(self, txn, table, values): - sql = "INSERT INTO %s (%s) VALUES(%s)" % ( + def _simple_insert_txn(self, txn, table, values, or_replace=False): + sql = "%s INTO %s (%s) VALUES(%s)" % ( + ("INSERT OR REPLACE" if or_replace else "INSERT"), table, ", ".join(k for k in values), ", ".join("?" for k in values) diff --git a/synapse/storage/directory.py b/synapse/storage/directory.py index b22ce02f3f..bf55449253 100644 --- a/synapse/storage/directory.py +++ b/synapse/storage/directory.py @@ -1,5 +1,5 @@ # -*- coding: utf-8 -*- -# Copyright 2014 matrix.org +# Copyright 2014 OpenMarket Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. diff --git a/synapse/storage/feedback.py b/synapse/storage/feedback.py index bac3dea955..8a18617188 100644 --- a/synapse/storage/feedback.py +++ b/synapse/storage/feedback.py @@ -1,5 +1,5 @@ # -*- coding: utf-8 -*- -# Copyright 2014 matrix.org +# Copyright 2014 OpenMarket Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. diff --git a/synapse/storage/keys.py b/synapse/storage/keys.py index 4d19b9f641..5a38c3e8f2 100644 --- a/synapse/storage/keys.py +++ b/synapse/storage/keys.py @@ -1,5 +1,5 @@ # -*- coding: utf-8 -*- -# Copyright 2014 matrix.org +# Copyright 2014 OpenMarket Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. diff --git a/synapse/storage/pdu.py b/synapse/storage/pdu.py index 9fd44f2454..0bf97e37ee 100644 --- a/synapse/storage/pdu.py +++ b/synapse/storage/pdu.py @@ -1,5 +1,5 @@ # -*- coding: utf-8 -*- -# Copyright 2014 matrix.org +# Copyright 2014 OpenMarket Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. diff --git a/synapse/storage/presence.py b/synapse/storage/presence.py index a529104f4d..71b2bb084d 100644 --- a/synapse/storage/presence.py +++ b/synapse/storage/presence.py @@ -1,5 +1,5 @@ # -*- coding: utf-8 -*- -# Copyright 2014 matrix.org +# Copyright 2014 OpenMarket Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. diff --git a/synapse/storage/profile.py b/synapse/storage/profile.py index 91dd565033..7e1fdd9d88 100644 --- a/synapse/storage/profile.py +++ b/synapse/storage/profile.py @@ -1,5 +1,5 @@ # -*- coding: utf-8 -*- -# Copyright 2014 matrix.org +# Copyright 2014 OpenMarket Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. diff --git a/synapse/storage/registration.py b/synapse/storage/registration.py index b1e4196435..fd762bc643 100644 --- a/synapse/storage/registration.py +++ b/synapse/storage/registration.py @@ -1,5 +1,5 @@ # -*- coding: utf-8 -*- -# Copyright 2014 matrix.org +# Copyright 2014 OpenMarket Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. diff --git a/synapse/storage/room.py b/synapse/storage/room.py index 01ae190316..017169ce00 100644 --- a/synapse/storage/room.py +++ b/synapse/storage/room.py @@ -1,5 +1,5 @@ # -*- coding: utf-8 -*- -# Copyright 2014 matrix.org +# Copyright 2014 OpenMarket Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. diff --git a/synapse/storage/roommember.py b/synapse/storage/roommember.py index 2746126e85..75c9a60101 100644 --- a/synapse/storage/roommember.py +++ b/synapse/storage/roommember.py @@ -1,5 +1,5 @@ # -*- coding: utf-8 -*- -# Copyright 2014 matrix.org +# Copyright 2014 OpenMarket Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. @@ -165,7 +165,7 @@ class RoomMemberStore(SQLBaseStore): defer.returnValue(results) @defer.inlineCallbacks - def do_users_share_a_room(self, user_list): + def user_rooms_intersect(self, user_list): """ Checks whether a list of users share a room. """ user_list_clause = " OR ".join(["m.user_id = ?"] * len(user_list)) diff --git a/synapse/storage/schema/delta/v2.sql b/synapse/storage/schema/delta/v2.sql new file mode 100644 index 0000000000..73b140465e --- /dev/null +++ b/synapse/storage/schema/delta/v2.sql @@ -0,0 +1,168 @@ +/* Copyright 2014 OpenMarket Ltd + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +CREATE TABLE IF NOT EXISTS events( + stream_ordering INTEGER PRIMARY KEY AUTOINCREMENT, + topological_ordering INTEGER NOT NULL, + event_id TEXT NOT NULL, + type TEXT NOT NULL, + room_id TEXT NOT NULL, + content TEXT NOT NULL, + unrecognized_keys TEXT, + processed BOOL NOT NULL, + outlier BOOL NOT NULL, + CONSTRAINT ev_uniq UNIQUE (event_id) +); + +CREATE INDEX IF NOT EXISTS events_event_id ON events (event_id); +CREATE INDEX IF NOT EXISTS events_stream_ordering ON events (stream_ordering); +CREATE INDEX IF NOT EXISTS events_topological_ordering ON events (topological_ordering); +CREATE INDEX IF NOT EXISTS events_room_id ON events (room_id); + +CREATE TABLE IF NOT EXISTS state_events( + event_id TEXT NOT NULL, + room_id TEXT NOT NULL, + type TEXT NOT NULL, + state_key TEXT NOT NULL, + prev_state TEXT +); + +CREATE UNIQUE INDEX IF NOT EXISTS state_events_event_id ON state_events (event_id); +CREATE INDEX IF NOT EXISTS state_events_room_id ON state_events (room_id); +CREATE INDEX IF NOT EXISTS state_events_type ON state_events (type); +CREATE INDEX IF NOT EXISTS state_events_state_key ON state_events (state_key); + + +CREATE TABLE IF NOT EXISTS current_state_events( + event_id TEXT NOT NULL, + room_id TEXT NOT NULL, + type TEXT NOT NULL, + state_key TEXT NOT NULL, + CONSTRAINT curr_uniq UNIQUE (room_id, type, state_key) ON CONFLICT REPLACE +); + +CREATE INDEX IF NOT EXISTS curr_events_event_id ON current_state_events (event_id); +CREATE INDEX IF NOT EXISTS current_state_events_room_id ON current_state_events (room_id); +CREATE INDEX IF NOT EXISTS current_state_events_type ON current_state_events (type); +CREATE INDEX IF NOT EXISTS current_state_events_state_key ON current_state_events (state_key); + +CREATE TABLE IF NOT EXISTS room_memberships( + event_id TEXT NOT NULL, + user_id TEXT NOT NULL, + sender TEXT NOT NULL, + room_id TEXT NOT NULL, + membership TEXT NOT NULL +); + +CREATE INDEX IF NOT EXISTS room_memberships_event_id ON room_memberships (event_id); +CREATE INDEX IF NOT EXISTS room_memberships_room_id ON room_memberships (room_id); +CREATE INDEX IF NOT EXISTS room_memberships_user_id ON room_memberships (user_id); + +CREATE TABLE IF NOT EXISTS feedback( + event_id TEXT NOT NULL, + feedback_type TEXT, + target_event_id TEXT, + sender TEXT, + room_id TEXT +); + +CREATE TABLE IF NOT EXISTS topics( + event_id TEXT NOT NULL, + room_id TEXT NOT NULL, + topic TEXT NOT NULL +); + +CREATE TABLE IF NOT EXISTS room_names( + event_id TEXT NOT NULL, + room_id TEXT NOT NULL, + name TEXT NOT NULL +); + +CREATE TABLE IF NOT EXISTS rooms( + room_id TEXT PRIMARY KEY NOT NULL, + is_public INTEGER, + creator TEXT +); + +CREATE TABLE IF NOT EXISTS room_join_rules( + event_id TEXT NOT NULL, + room_id TEXT NOT NULL, + join_rule TEXT NOT NULL +); +CREATE INDEX IF NOT EXISTS room_join_rules_event_id ON room_join_rules(event_id); +CREATE INDEX IF NOT EXISTS room_join_rules_room_id ON room_join_rules(room_id); + + +CREATE TABLE IF NOT EXISTS room_power_levels( + event_id TEXT NOT NULL, + room_id TEXT NOT NULL, + user_id TEXT NOT NULL, + level INTEGER NOT NULL +); +CREATE INDEX IF NOT EXISTS room_power_levels_event_id ON room_power_levels(event_id); +CREATE INDEX IF NOT EXISTS room_power_levels_room_id ON room_power_levels(room_id); +CREATE INDEX IF NOT EXISTS room_power_levels_room_user ON room_power_levels(room_id, user_id); + + +CREATE TABLE IF NOT EXISTS room_default_levels( + event_id TEXT NOT NULL, + room_id TEXT NOT NULL, + level INTEGER NOT NULL +); + +CREATE INDEX IF NOT EXISTS room_default_levels_event_id ON room_default_levels(event_id); +CREATE INDEX IF NOT EXISTS room_default_levels_room_id ON room_default_levels(room_id); + + +CREATE TABLE IF NOT EXISTS room_add_state_levels( + event_id TEXT NOT NULL, + room_id TEXT NOT NULL, + level INTEGER NOT NULL +); + +CREATE INDEX IF NOT EXISTS room_add_state_levels_event_id ON room_add_state_levels(event_id); +CREATE INDEX IF NOT EXISTS room_add_state_levels_room_id ON room_add_state_levels(room_id); + + +CREATE TABLE IF NOT EXISTS room_send_event_levels( + event_id TEXT NOT NULL, + room_id TEXT NOT NULL, + level INTEGER NOT NULL +); + +CREATE INDEX IF NOT EXISTS room_send_event_levels_event_id ON room_send_event_levels(event_id); +CREATE INDEX IF NOT EXISTS room_send_event_levels_room_id ON room_send_event_levels(room_id); + + +CREATE TABLE IF NOT EXISTS room_ops_levels( + event_id TEXT NOT NULL, + room_id TEXT NOT NULL, + ban_level INTEGER, + kick_level INTEGER +); + +CREATE INDEX IF NOT EXISTS room_ops_levels_event_id ON room_ops_levels(event_id); +CREATE INDEX IF NOT EXISTS room_ops_levels_room_id ON room_ops_levels(room_id); + + +CREATE TABLE IF NOT EXISTS room_hosts( + room_id TEXT NOT NULL, + host TEXT NOT NULL, + CONSTRAINT room_hosts_uniq UNIQUE (room_id, host) ON CONFLICT IGNORE +); + +CREATE INDEX IF NOT EXISTS room_hosts_room_id ON room_hosts (room_id); + +PRAGMA user_version = 2; diff --git a/synapse/storage/schema/edge_pdus.sql b/synapse/storage/schema/edge_pdus.sql index 17b3c52f0d..8a00868065 100644 --- a/synapse/storage/schema/edge_pdus.sql +++ b/synapse/storage/schema/edge_pdus.sql @@ -1,4 +1,4 @@ -/* Copyright 2014 matrix.org +/* Copyright 2014 OpenMarket Ltd * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. diff --git a/synapse/storage/schema/im.sql b/synapse/storage/schema/im.sql index dbefbbda31..6ffea51310 100644 --- a/synapse/storage/schema/im.sql +++ b/synapse/storage/schema/im.sql @@ -1,4 +1,4 @@ -/* Copyright 2014 matrix.org +/* Copyright 2014 OpenMarket Ltd * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. diff --git a/synapse/storage/schema/keys.sql b/synapse/storage/schema/keys.sql index 45cdbcecae..706a1a03ff 100644 --- a/synapse/storage/schema/keys.sql +++ b/synapse/storage/schema/keys.sql @@ -1,4 +1,4 @@ -/* Copyright 2014 matrix.org +/* Copyright 2014 OpenMarket Ltd * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. diff --git a/synapse/storage/schema/pdu.sql b/synapse/storage/schema/pdu.sql index ca3de005e9..16e111a56c 100644 --- a/synapse/storage/schema/pdu.sql +++ b/synapse/storage/schema/pdu.sql @@ -1,4 +1,4 @@ -/* Copyright 2014 matrix.org +/* Copyright 2014 OpenMarket Ltd * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. diff --git a/synapse/storage/schema/presence.sql b/synapse/storage/schema/presence.sql index b1081d3aab..595b3b5a69 100644 --- a/synapse/storage/schema/presence.sql +++ b/synapse/storage/schema/presence.sql @@ -1,4 +1,4 @@ -/* Copyright 2014 matrix.org +/* Copyright 2014 OpenMarket Ltd * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. diff --git a/synapse/storage/schema/profiles.sql b/synapse/storage/schema/profiles.sql index 1092d7672c..58209f1af0 100644 --- a/synapse/storage/schema/profiles.sql +++ b/synapse/storage/schema/profiles.sql @@ -1,4 +1,4 @@ -/* Copyright 2014 matrix.org +/* Copyright 2014 OpenMarket Ltd * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. diff --git a/synapse/storage/schema/room_aliases.sql b/synapse/storage/schema/room_aliases.sql index 71a8b90e4d..9191016814 100644 --- a/synapse/storage/schema/room_aliases.sql +++ b/synapse/storage/schema/room_aliases.sql @@ -1,3 +1,18 @@ +/* Copyright 2014 OpenMarket Ltd + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + CREATE TABLE IF NOT EXISTS room_aliases( room_alias TEXT NOT NULL, room_id TEXT NOT NULL diff --git a/synapse/storage/schema/transactions.sql b/synapse/storage/schema/transactions.sql index 4b1a2368f6..88e3e4e04d 100644 --- a/synapse/storage/schema/transactions.sql +++ b/synapse/storage/schema/transactions.sql @@ -1,4 +1,4 @@ -/* Copyright 2014 matrix.org +/* Copyright 2014 OpenMarket Ltd * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. diff --git a/synapse/storage/schema/users.sql b/synapse/storage/schema/users.sql index 46b60297cb..2519702971 100644 --- a/synapse/storage/schema/users.sql +++ b/synapse/storage/schema/users.sql @@ -1,4 +1,4 @@ -/* Copyright 2014 matrix.org +/* Copyright 2014 OpenMarket Ltd * * Licensed under the Apache License, Version 2.0 (the "License"); * you may not use this file except in compliance with the License. diff --git a/synapse/storage/stream.py b/synapse/storage/stream.py index 0b78222827..2cb0067a67 100644 --- a/synapse/storage/stream.py +++ b/synapse/storage/stream.py @@ -1,5 +1,5 @@ # -*- coding: utf-8 -*- -# Copyright 2014 matrix.org +# Copyright 2014 OpenMarket Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. diff --git a/synapse/storage/transactions.py b/synapse/storage/transactions.py index a277e4971a..7467e1035b 100644 --- a/synapse/storage/transactions.py +++ b/synapse/storage/transactions.py @@ -1,5 +1,5 @@ # -*- coding: utf-8 -*- -# Copyright 2014 matrix.org +# Copyright 2014 OpenMarket Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. diff --git a/synapse/streams/__init__.py b/synapse/streams/__init__.py index fe8a073cd3..f9811bfa04 100644 --- a/synapse/streams/__init__.py +++ b/synapse/streams/__init__.py @@ -1,5 +1,5 @@ # -*- coding: utf-8 -*- -# Copyright 2014 matrix.org +# Copyright 2014 OpenMarket Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. diff --git a/synapse/streams/config.py b/synapse/streams/config.py index 01bab568ff..6483ce2e25 100644 --- a/synapse/streams/config.py +++ b/synapse/streams/config.py @@ -1,5 +1,5 @@ # -*- coding: utf-8 -*- -# Copyright 2014 matrix.org +# Copyright 2014 OpenMarket Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. diff --git a/synapse/streams/events.py b/synapse/streams/events.py index 08d6e6f733..41715436b0 100644 --- a/synapse/streams/events.py +++ b/synapse/streams/events.py @@ -1,5 +1,5 @@ # -*- coding: utf-8 -*- -# Copyright 2014 matrix.org +# Copyright 2014 OpenMarket Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. diff --git a/synapse/types.py b/synapse/types.py index 1a9dceabf5..c51bc8e4f2 100644 --- a/synapse/types.py +++ b/synapse/types.py @@ -1,5 +1,5 @@ # -*- coding: utf-8 -*- -# Copyright 2014 matrix.org +# Copyright 2014 OpenMarket Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. diff --git a/synapse/util/__init__.py b/synapse/util/__init__.py index 3ea431a7f9..c9a73b0413 100644 --- a/synapse/util/__init__.py +++ b/synapse/util/__init__.py @@ -1,5 +1,5 @@ # -*- coding: utf-8 -*- -# Copyright 2014 matrix.org +# Copyright 2014 OpenMarket Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. diff --git a/synapse/util/async.py b/synapse/util/async.py index ebbdc00ae1..647ea6142c 100644 --- a/synapse/util/async.py +++ b/synapse/util/async.py @@ -1,5 +1,5 @@ # -*- coding: utf-8 -*- -# Copyright 2014 matrix.org +# Copyright 2014 OpenMarket Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. diff --git a/synapse/util/distributor.py b/synapse/util/distributor.py index 9605d7d1b9..1de50e049f 100644 --- a/synapse/util/distributor.py +++ b/synapse/util/distributor.py @@ -1,5 +1,5 @@ # -*- coding: utf-8 -*- -# Copyright 2014 matrix.org +# Copyright 2014 OpenMarket Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. @@ -32,7 +32,9 @@ class Distributor(object): model will do for today. """ - def __init__(self): + def __init__(self, suppress_failures=True): + self.suppress_failures = suppress_failures + self.signals = {} self.pre_registration = {} @@ -40,7 +42,9 @@ class Distributor(object): if name in self.signals: raise KeyError("%r already has a signal named %s" % (self, name)) - self.signals[name] = Signal(name) + self.signals[name] = Signal(name, + suppress_failures=self.suppress_failures, + ) if name in self.pre_registration: signal = self.signals[name] @@ -74,8 +78,9 @@ class Signal(object): method into all of the observers. """ - def __init__(self, name): + def __init__(self, name, suppress_failures): self.name = name + self.suppress_failures = suppress_failures self.observers = [] def observe(self, observer): @@ -104,6 +109,10 @@ class Signal(object): failure.type, failure.value, failure.getTracebackObject())) + if not self.suppress_failures: + raise failure deferreds.append(d.addErrback(eb)) - return defer.DeferredList(deferreds) + return defer.DeferredList( + deferreds, fireOnOneErrback=not self.suppress_failures + ) diff --git a/synapse/util/jsonobject.py b/synapse/util/jsonobject.py index e2840b59f9..6c99705747 100644 --- a/synapse/util/jsonobject.py +++ b/synapse/util/jsonobject.py @@ -1,5 +1,5 @@ # -*- coding: utf-8 -*- -# Copyright 2014 matrix.org +# Copyright 2014 OpenMarket Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. diff --git a/synapse/util/lockutils.py b/synapse/util/lockutils.py index d0bb50d035..3a84c09db4 100644 --- a/synapse/util/lockutils.py +++ b/synapse/util/lockutils.py @@ -1,5 +1,5 @@ # -*- coding: utf-8 -*- -# Copyright 2014 matrix.org +# Copyright 2014 OpenMarket Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. diff --git a/synapse/util/logutils.py b/synapse/util/logutils.py index b94a749786..fadf0bd510 100644 --- a/synapse/util/logutils.py +++ b/synapse/util/logutils.py @@ -1,5 +1,5 @@ # -*- coding: utf-8 -*- -# Copyright 2014 matrix.org +# Copyright 2014 OpenMarket Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. @@ -19,7 +19,6 @@ from functools import wraps import logging import inspect -import traceback def log_function(f): diff --git a/synapse/util/stringutils.py b/synapse/util/stringutils.py index e1b0796e56..8767e437dd 100644 --- a/synapse/util/stringutils.py +++ b/synapse/util/stringutils.py @@ -1,5 +1,5 @@ # -*- coding: utf-8 -*- -# Copyright 2014 matrix.org +# Copyright 2014 OpenMarket Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. diff --git a/synctl b/synctl new file mode 100755 index 0000000000..763bcce0a0 --- /dev/null +++ b/synctl @@ -0,0 +1,38 @@ +#!/bin/bash + +SYNAPSE="synapse/app/homeserver.py" + +CONFIGFILE="homeserver.yaml" +PIDFILE="homeserver.pid" + +GREEN=$'\e[1;32m' +NORMAL=$'\e[m' + +set -e + +case "$1" in + start) + if [ ! -f "$CONFIGFILE" ]; then + echo "No config file found - generating a default one..." + $SYNAPSE -c "$CONFIGFILE" --generate-config + echo "Wrote $CONFIGFILE" + echo "You must now edit this file before continuing" + exit 1 + fi + + echo -n "Starting ..." + $SYNAPSE --daemonize -c "$CONFIGFILE" --pid-file "$PIDFILE" --log-file synapse.log + echo "${GREEN}started${NORMAL}" + ;; + stop) + echo -n "Stopping ..." + test -f $PIDFILE && kill `cat $PIDFILE` + echo "${GREEN}stopped${NORMAL}" + ;; + restart) + $0 stop && $0 start + ;; + *) + echo "Usage: $0 [start|stop|restart]" >&2 + exit 1 +esac diff --git a/tests/__init__.py b/tests/__init__.py index 2216c0f1ca..9bff9ec169 100644 --- a/tests/__init__.py +++ b/tests/__init__.py @@ -1,5 +1,5 @@ # -*- coding: utf-8 -*- -# Copyright 2014 matrix.org +# Copyright 2014 OpenMarket Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. diff --git a/tests/api/__init__.py b/tests/api/__init__.py new file mode 100644 index 0000000000..e69de29bb2 --- /dev/null +++ b/tests/api/__init__.py diff --git a/tests/api/test_ratelimiting.py b/tests/api/test_ratelimiting.py new file mode 100644 index 0000000000..dc2f83c7eb --- /dev/null +++ b/tests/api/test_ratelimiting.py @@ -0,0 +1,39 @@ +from synapse.api.ratelimiting import Ratelimiter + +import unittest + +class TestRatelimiter(unittest.TestCase): + + def test_allowed(self): + limiter = Ratelimiter() + allowed, time_allowed = limiter.send_message( + user_id="test_id", time_now_s=0, msg_rate_hz=0.1, burst_count=1, + ) + self.assertTrue(allowed) + self.assertEquals(10., time_allowed) + + allowed, time_allowed = limiter.send_message( + user_id="test_id", time_now_s=5, msg_rate_hz=0.1, burst_count=1, + ) + self.assertFalse(allowed) + self.assertEquals(10., time_allowed) + + allowed, time_allowed = limiter.send_message( + user_id="test_id", time_now_s=10, msg_rate_hz=0.1, burst_count=1 + ) + self.assertTrue(allowed) + self.assertEquals(20., time_allowed) + + def test_pruning(self): + limiter = Ratelimiter() + allowed, time_allowed = limiter.send_message( + user_id="test_id_1", time_now_s=0, msg_rate_hz=0.1, burst_count=1, + ) + + self.assertIn("test_id_1", limiter.message_counts) + + allowed, time_allowed = limiter.send_message( + user_id="test_id_2", time_now_s=10, msg_rate_hz=0.1, burst_count=1 + ) + + self.assertNotIn("test_id_1", limiter.message_counts) diff --git a/tests/events/__init__.py b/tests/events/__init__.py index 2216c0f1ca..9bff9ec169 100644 --- a/tests/events/__init__.py +++ b/tests/events/__init__.py @@ -1,5 +1,5 @@ # -*- coding: utf-8 -*- -# Copyright 2014 matrix.org +# Copyright 2014 OpenMarket Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. diff --git a/tests/events/test_events.py b/tests/events/test_events.py index 35e9c68f5c..93d5c15c6f 100644 --- a/tests/events/test_events.py +++ b/tests/events/test_events.py @@ -1,5 +1,5 @@ # -*- coding: utf-8 -*- -# Copyright 2014 matrix.org +# Copyright 2014 OpenMarket Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. diff --git a/tests/federation/test_federation.py b/tests/federation/test_federation.py index 51308ca358..0b105fe723 100644 --- a/tests/federation/test_federation.py +++ b/tests/federation/test_federation.py @@ -1,4 +1,4 @@ -# Copyright 2014 matrix.org +# Copyright 2014 OpenMarket Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. diff --git a/tests/federation/test_pdu_codec.py b/tests/federation/test_pdu_codec.py index 2c546040b8..9f74ba119f 100644 --- a/tests/federation/test_pdu_codec.py +++ b/tests/federation/test_pdu_codec.py @@ -1,5 +1,5 @@ # -*- coding: utf-8 -*- -# Copyright 2014 matrix.org +# Copyright 2014 OpenMarket Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. diff --git a/tests/handlers/test_directory.py b/tests/handlers/test_directory.py index 88ac8933f8..72a2b1443a 100644 --- a/tests/handlers/test_directory.py +++ b/tests/handlers/test_directory.py @@ -1,5 +1,5 @@ # -*- coding: utf-8 -*- -# Copyright 2014 matrix.org +# Copyright 2014 OpenMarket Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. @@ -21,6 +21,7 @@ from mock import Mock import logging from synapse.server import HomeServer +from synapse.http.client import HttpClient from synapse.handlers.directory import DirectoryHandler from synapse.storage.directory import RoomAliasMapping @@ -49,6 +50,7 @@ class DirectoryTestCase(unittest.TestCase): hs = HomeServer("test", datastore=Mock(spec=[ "get_association_from_room_alias", + "get_joined_hosts_for_room", ]), http_client=None, resource_for_federation=Mock(), @@ -60,6 +62,10 @@ class DirectoryTestCase(unittest.TestCase): self.datastore = hs.get_datastore() + def hosts(room_id): + return defer.succeed([]) + self.datastore.get_joined_hosts_for_room.side_effect = hosts + self.my_room = hs.parse_roomalias("#my-room:test") self.remote_room = hs.parse_roomalias("#another:remote") @@ -92,7 +98,10 @@ class DirectoryTestCase(unittest.TestCase): self.mock_federation.make_query.assert_called_with( destination="remote", query_type="directory", - args={"room_alias": "#another:remote"} + args={ + "room_alias": "#another:remote", + HttpClient.RETRY_DNS_LOOKUP_FAILURES: False + } ) @defer.inlineCallbacks diff --git a/tests/handlers/test_federation.py b/tests/handlers/test_federation.py index fd19442645..6fc3d8f7fd 100644 --- a/tests/handlers/test_federation.py +++ b/tests/handlers/test_federation.py @@ -1,4 +1,4 @@ -# Copyright 2014 matrix.org +# Copyright 2014 OpenMarket Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. @@ -105,7 +105,7 @@ class FederationTestCase(unittest.TestCase): lambda event, do_auth: None, mem_handler.change_membership ) - self.assertEquals(True, call_args["do_auth"]) + self.assertEquals(False, call_args["do_auth"]) new_event = call_args["event"] self.assertEquals(RoomMemberEvent.TYPE, new_event.type) diff --git a/tests/handlers/test_presence.py b/tests/handlers/test_presence.py index b8309bc063..9eb8b6909f 100644 --- a/tests/handlers/test_presence.py +++ b/tests/handlers/test_presence.py @@ -1,5 +1,5 @@ # -*- coding: utf-8 -*- -# Copyright 2014 matrix.org +# Copyright 2014 OpenMarket Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. @@ -117,10 +117,12 @@ class PresenceStateTestCase(unittest.TestCase): return defer.succeed([]) room_member_handler.get_room_members = get_room_members - def do_users_share_a_room(userlist): - shared = all(map(lambda u: u in self.room_members, userlist)) + def user_rooms_intersect(userlist): + room_member_ids = map(lambda u: u.to_string(), self.room_members) + + shared = all(map(lambda i: i in room_member_ids, userlist)) return defer.succeed(shared) - self.datastore.do_users_share_a_room = do_users_share_a_room + self.datastore.user_rooms_intersect = user_rooms_intersect self.mock_start = Mock() self.mock_stop = Mock() @@ -140,7 +142,7 @@ class PresenceStateTestCase(unittest.TestCase): ) self.assertEquals( - {"state": ONLINE, "presence": ONLINE, "status_msg": "Online"}, + {"presence": ONLINE, "status_msg": "Online"}, state ) mocked_get.assert_called_with("apple") @@ -157,7 +159,7 @@ class PresenceStateTestCase(unittest.TestCase): ) self.assertEquals( - {"state": ONLINE, "presence": ONLINE, "status_msg": "Online"}, + {"presence": ONLINE, "status_msg": "Online"}, state ) mocked_get.assert_called_with("apple") @@ -176,7 +178,7 @@ class PresenceStateTestCase(unittest.TestCase): ) self.assertEquals( - {"state": ONLINE, "presence": ONLINE, "status_msg": "Online"}, + {"presence": ONLINE, "status_msg": "Online"}, state ) @@ -206,7 +208,8 @@ class PresenceStateTestCase(unittest.TestCase): state={"presence": UNAVAILABLE, "status_msg": "Away"}) mocked_set.assert_called_with("apple", - {"state": UNAVAILABLE, "status_msg": "Away"}) + {"state": UNAVAILABLE, "status_msg": "Away"} + ) self.mock_start.assert_called_with(self.u_apple, state={ "presence": UNAVAILABLE, @@ -458,8 +461,7 @@ class PresenceInvitesTestCase(unittest.TestCase): self.assertEquals([ {"observed_user": self.u_banana, - "presence": OFFLINE, - "state": OFFLINE}, + "presence": OFFLINE}, ], presence) self.datastore.get_presence_list.assert_called_with("apple", @@ -476,8 +478,7 @@ class PresenceInvitesTestCase(unittest.TestCase): self.assertEquals([ {"observed_user": self.u_banana, - "presence": OFFLINE, - "state": OFFLINE}, + "presence": OFFLINE}, ], presence) self.datastore.get_presence_list.assert_called_with("apple", @@ -562,6 +563,13 @@ class PresencePushTestCase(unittest.TestCase): return defer.succeed([]) self.datastore.get_joined_hosts_for_room = get_room_hosts + def user_rooms_intersect(userlist): + room_member_ids = map(lambda u: u.to_string(), self.room_members) + + shared = all(map(lambda i: i in room_member_ids, userlist)) + return defer.succeed(shared) + self.datastore.user_rooms_intersect = user_rooms_intersect + @defer.inlineCallbacks def fetch_room_distributions_into(room_id, localusers=None, remotedomains=None, ignore_user=None): @@ -604,6 +612,7 @@ class PresencePushTestCase(unittest.TestCase): self.u_apple = hs.parse_userid("@apple:test") self.u_banana = hs.parse_userid("@banana:test") self.u_clementine = hs.parse_userid("@clementine:test") + self.u_durian = hs.parse_userid("@durian:test") self.u_elderberry = hs.parse_userid("@elderberry:test") # Remote user @@ -615,7 +624,8 @@ class PresencePushTestCase(unittest.TestCase): self.room_members = [self.u_apple, self.u_elderberry] self.datastore.set_presence_state.return_value = defer.succeed( - {"state": ONLINE}) + {"state": ONLINE} + ) # TODO(paul): Gut-wrenching self.handler._user_cachemap[self.u_apple] = UserPresenceCache() @@ -632,6 +642,7 @@ class PresencePushTestCase(unittest.TestCase): {"presence": ONLINE} ) + # Apple sees self-reflection (events, _) = yield self.event_source.get_new_events_for_user( self.u_apple, 0, None ) @@ -643,10 +654,56 @@ class PresencePushTestCase(unittest.TestCase): "content": { "user_id": "@apple:test", "presence": ONLINE, - "state": ONLINE, "last_active_ago": 0, }}, ], + msg="Presence event should be visible to self-reflection" + ) + + # Banana sees it because of presence subscription + (events, _) = yield self.event_source.get_new_events_for_user( + self.u_banana, 0, None + ) + + self.assertEquals(self.event_source.get_current_key(), 1) + self.assertEquals(events, + [ + {"type": "m.presence", + "content": { + "user_id": "@apple:test", + "presence": ONLINE, + "last_active_ago": 0, + }}, + ], + msg="Presence event should be visible to explicit subscribers" + ) + + # Elderberry sees it because of same room + (events, _) = yield self.event_source.get_new_events_for_user( + self.u_elderberry, 0, None + ) + + self.assertEquals(self.event_source.get_current_key(), 1) + self.assertEquals(events, + [ + {"type": "m.presence", + "content": { + "user_id": "@apple:test", + "presence": ONLINE, + "last_active_ago": 0, + }}, + ], + msg="Presence event should be visible to other room members" + ) + + # Durian is not in the room, should not see this event + (events, _) = yield self.event_source.get_new_events_for_user( + self.u_durian, 0, None + ) + + self.assertEquals(self.event_source.get_current_key(), 1) + self.assertEquals(events, [], + msg="Presence event should not be visible to others" ) presence = yield self.handler.get_presence_list( @@ -655,15 +712,17 @@ class PresencePushTestCase(unittest.TestCase): self.assertEquals( [ {"observed_user": self.u_banana, - "presence": OFFLINE, - "state": OFFLINE}, + "presence": OFFLINE}, {"observed_user": self.u_clementine, - "presence": OFFLINE, - "state": OFFLINE}, + "presence": OFFLINE}, ], presence ) + # TODO(paul): Gut-wrenching + banana_set = self.handler._local_pushmap.setdefault("banana", set()) + banana_set.add(self.u_apple) + yield self.handler.set_state(self.u_banana, self.u_banana, {"presence": ONLINE} ) @@ -676,11 +735,9 @@ class PresencePushTestCase(unittest.TestCase): self.assertEquals([ {"observed_user": self.u_banana, "presence": ONLINE, - "state": ONLINE, "last_active_ago": 2000}, {"observed_user": self.u_clementine, - "presence": OFFLINE, - "state": OFFLINE}, + "presence": OFFLINE}, ], presence) (events, _) = yield self.event_source.get_new_events_for_user( @@ -694,7 +751,6 @@ class PresencePushTestCase(unittest.TestCase): "content": { "user_id": "@banana:test", "presence": ONLINE, - "state": ONLINE, "last_active_ago": 2000 }}, ] @@ -711,7 +767,21 @@ class PresencePushTestCase(unittest.TestCase): "push": [ {"user_id": "@apple:test", "presence": u"online", - "state": u"online", + "last_active_ago": 0}, + ], + } + ) + ), + defer.succeed((200, "OK")) + ) + put_json.expect_call_and_return( + call("remote", + path=ANY, # Can't guarantee which txn ID will be which + data=_expect_edu("remote", "m.presence", + content={ + "push": [ + {"user_id": "@apple:test", + "presence": u"online", "last_active_ago": 0}, ], } @@ -757,7 +827,7 @@ class PresencePushTestCase(unittest.TestCase): content={ "push": [ {"user_id": "@potato:remote", - "state": "online", + "presence": "online", "last_active_ago": 1000}, ], } @@ -775,7 +845,6 @@ class PresencePushTestCase(unittest.TestCase): "content": { "user_id": "@potato:remote", "presence": ONLINE, - "state": ONLINE, "last_active_ago": 1000, }} ] @@ -786,7 +855,7 @@ class PresencePushTestCase(unittest.TestCase): state = yield self.handler.get_state(self.u_potato, self.u_apple) self.assertEquals( - {"state": ONLINE, "presence": ONLINE, "last_active_ago": 3000}, + {"presence": ONLINE, "last_active_ago": 3000}, state ) @@ -809,6 +878,8 @@ class PresencePushTestCase(unittest.TestCase): "a-room" ) + self.room_members.append(self.u_clementine) + (events, _) = yield self.event_source.get_new_events_for_user( self.u_apple, 0, None ) @@ -820,7 +891,6 @@ class PresencePushTestCase(unittest.TestCase): "content": { "user_id": "@clementine:test", "presence": ONLINE, - "state": ONLINE, "last_active_ago": 0, }} ] @@ -837,8 +907,7 @@ class PresencePushTestCase(unittest.TestCase): content={ "push": [ {"user_id": "@apple:test", - "presence": "online", - "state": "online"}, + "presence": "online"}, ], } ), @@ -852,8 +921,7 @@ class PresencePushTestCase(unittest.TestCase): content={ "push": [ {"user_id": "@banana:test", - "presence": "offline", - "state": "offline"}, + "presence": "offline"}, ], } ), @@ -882,8 +950,7 @@ class PresencePushTestCase(unittest.TestCase): content={ "push": [ {"user_id": "@clementine:test", - "presence": "online", - "state": "online"}, + "presence": "online"}, ], } ), @@ -1182,7 +1249,6 @@ class PresencePollingTestCase(unittest.TestCase): "push": [ {"user_id": "@banana:test", "presence": "offline", - "state": "offline", "status_msg": None}, ], }, diff --git a/tests/handlers/test_presencelike.py b/tests/handlers/test_presencelike.py index 38cc34350b..b35980d948 100644 --- a/tests/handlers/test_presencelike.py +++ b/tests/handlers/test_presencelike.py @@ -1,5 +1,5 @@ # -*- coding: utf-8 -*- -# Copyright 2014 matrix.org +# Copyright 2014 OpenMarket Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. @@ -107,9 +107,9 @@ class PresenceProfilelikeDataTestCase(unittest.TestCase): return defer.succeed(self.presence_list) self.datastore.get_presence_list = get_presence_list - def do_users_share_a_room(userlist): + def user_rooms_intersect(userlist): return defer.succeed(False) - self.datastore.do_users_share_a_room = do_users_share_a_room + self.datastore.user_rooms_intersect = user_rooms_intersect self.handlers = hs.get_handlers() @@ -148,10 +148,11 @@ class PresenceProfilelikeDataTestCase(unittest.TestCase): yield self.handlers.presence_handler.set_state( target_user=self.u_apple, auth_user=self.u_apple, - state={"state": UNAVAILABLE, "status_msg": "Away"}) + state={"presence": UNAVAILABLE, "status_msg": "Away"}) mocked_set.assert_called_with("apple", - {"state": UNAVAILABLE, "status_msg": "Away"}) + {"state": UNAVAILABLE, "status_msg": "Away"} + ) @defer.inlineCallbacks def test_push_local(self): @@ -161,7 +162,8 @@ class PresenceProfilelikeDataTestCase(unittest.TestCase): ] self.datastore.set_presence_state.return_value = defer.succeed( - {"state": ONLINE}) + {"state": ONLINE} + ) # TODO(paul): Gut-wrenching from synapse.handlers.presence import UserPresenceCache @@ -177,9 +179,11 @@ class PresenceProfilelikeDataTestCase(unittest.TestCase): apple_set.add(self.u_clementine) yield self.handlers.presence_handler.set_state(self.u_apple, - self.u_apple, {"state": ONLINE}) + self.u_apple, {"presence": ONLINE} + ) yield self.handlers.presence_handler.set_state(self.u_banana, - self.u_banana, {"state": ONLINE}) + self.u_banana, {"presence": ONLINE} + ) presence = yield self.handlers.presence_handler.get_presence_list( observer_user=self.u_apple, accepted=True) @@ -187,14 +191,12 @@ class PresenceProfilelikeDataTestCase(unittest.TestCase): self.assertEquals([ {"observed_user": self.u_banana, "presence": ONLINE, - "state": ONLINE, "last_active_ago": 0, "displayname": "Frank", "avatar_url": "http://foo"}, {"observed_user": self.u_clementine, - "presence": OFFLINE, - "state": OFFLINE}], - presence) + "presence": OFFLINE} + ], presence) self.mock_update_client.assert_has_calls([ call(users_to_push=set([self.u_apple, self.u_banana, self.u_clementine]), @@ -242,7 +244,8 @@ class PresenceProfilelikeDataTestCase(unittest.TestCase): ] self.datastore.set_presence_state.return_value = defer.succeed( - {"state": ONLINE}) + {"state": ONLINE} + ) # TODO(paul): Gut-wrenching from synapse.handlers.presence import UserPresenceCache @@ -257,7 +260,8 @@ class PresenceProfilelikeDataTestCase(unittest.TestCase): apple_set.add(self.u_potato.domain) yield self.handlers.presence_handler.set_state(self.u_apple, - self.u_apple, {"state": ONLINE}) + self.u_apple, {"presence": ONLINE} + ) self.replication.send_edu.assert_called_with( destination="remote", @@ -266,7 +270,6 @@ class PresenceProfilelikeDataTestCase(unittest.TestCase): "push": [ {"user_id": "@apple:test", "presence": "online", - "state": "online", "last_active_ago": 0, "displayname": "Frank", "avatar_url": "http://foo"}, @@ -283,18 +286,19 @@ class PresenceProfilelikeDataTestCase(unittest.TestCase): # TODO(paul): Gut-wrenching potato_set = self.handlers.presence_handler._remote_recvmap.setdefault( - self.u_potato, set()) + self.u_potato, set() + ) potato_set.add(self.u_apple) yield self.replication.received_edu( - "remote", "m.presence", { - "push": [ - {"user_id": "@potato:remote", - "state": "online", - "displayname": "Frank", - "avatar_url": "http://foo"}, - ], - } + "remote", "m.presence", { + "push": [ + {"user_id": "@potato:remote", + "presence": "online", + "displayname": "Frank", + "avatar_url": "http://foo"}, + ], + } ) self.mock_update_client.assert_called_with( @@ -313,7 +317,6 @@ class PresenceProfilelikeDataTestCase(unittest.TestCase): self.assertEquals( {"presence": ONLINE, - "state": ONLINE, "displayname": "Frank", "avatar_url": "http://foo"}, state) diff --git a/tests/handlers/test_profile.py b/tests/handlers/test_profile.py index 87a8139920..8e7a89b479 100644 --- a/tests/handlers/test_profile.py +++ b/tests/handlers/test_profile.py @@ -1,5 +1,5 @@ # -*- coding: utf-8 -*- -# Copyright 2014 matrix.org +# Copyright 2014 OpenMarket Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. diff --git a/tests/handlers/test_room.py b/tests/handlers/test_room.py index 219a53c426..5687bbea0b 100644 --- a/tests/handlers/test_room.py +++ b/tests/handlers/test_room.py @@ -1,5 +1,5 @@ # -*- coding: utf-8 -*- -# Copyright 2014 matrix.org +# Copyright 2014 OpenMarket Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. @@ -39,6 +39,10 @@ class RoomMemberHandlerTestCase(unittest.TestCase): hs = HomeServer( self.hostname, db_pool=None, + ratelimiter=NonCallableMock(spec_set=[ + "send_message", + ]), + config=NonCallableMock(), datastore=NonCallableMock(spec_set=[ "persist_event", "get_joined_hosts_for_room", @@ -82,6 +86,8 @@ class RoomMemberHandlerTestCase(unittest.TestCase): self.snapshot = Mock() self.datastore.snapshot_room.return_value = self.snapshot + self.ratelimiter = hs.get_ratelimiter() + self.ratelimiter.send_message.return_value = (True, 0) @defer.inlineCallbacks def test_invite(self): @@ -342,6 +348,10 @@ class RoomCreationTest(unittest.TestCase): ]), auth=NonCallableMock(spec_set=["check"]), state_handler=NonCallableMock(spec_set=["handle_new_event"]), + ratelimiter=NonCallableMock(spec_set=[ + "send_message", + ]), + config=NonCallableMock(), ) self.federation = NonCallableMock(spec_set=[ @@ -368,6 +378,9 @@ class RoomCreationTest(unittest.TestCase): return defer.succeed([]) self.datastore.get_joined_hosts_for_room.side_effect = hosts + self.ratelimiter = hs.get_ratelimiter() + self.ratelimiter.send_message.return_value = (True, 0) + @defer.inlineCallbacks def test_room_creation(self): user_id = "@foo:red" diff --git a/tests/handlers/test_typing.py b/tests/handlers/test_typing.py index c3c98074cc..6532ac94a3 100644 --- a/tests/handlers/test_typing.py +++ b/tests/handlers/test_typing.py @@ -1,5 +1,5 @@ # -*- coding: utf-8 -*- -# Copyright 2014 matrix.org +# Copyright 2014 OpenMarket Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. diff --git a/tests/rest/__init__.py b/tests/rest/__init__.py index 2216c0f1ca..9bff9ec169 100644 --- a/tests/rest/__init__.py +++ b/tests/rest/__init__.py @@ -1,5 +1,5 @@ # -*- coding: utf-8 -*- -# Copyright 2014 matrix.org +# Copyright 2014 OpenMarket Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. diff --git a/tests/rest/test_events.py b/tests/rest/test_events.py index 1d1336d12d..1dccf4c503 100644 --- a/tests/rest/test_events.py +++ b/tests/rest/test_events.py @@ -1,5 +1,5 @@ # -*- coding: utf-8 -*- -# Copyright 2014 matrix.org +# Copyright 2014 OpenMarket Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. @@ -32,7 +32,7 @@ import logging from ..utils import MockHttpResource, MemoryDataStore from .utils import RestTestCase -from mock import Mock +from mock import Mock, NonCallableMock logging.getLogger().addHandler(logging.NullHandler()) @@ -136,8 +136,15 @@ class EventStreamPermissionsTestCase(RestTestCase): "call_later", "cancel_call_later", "time_msec", + "time" ]), + ratelimiter=NonCallableMock(spec_set=[ + "send_message", + ]), + config=NonCallableMock(), ) + self.ratelimiter = hs.get_ratelimiter() + self.ratelimiter.send_message.return_value = (True, 0) hs.get_handlers().federation_handler = Mock() diff --git a/tests/rest/test_presence.py b/tests/rest/test_presence.py index e2cdd80e07..a1db0fbcf3 100644 --- a/tests/rest/test_presence.py +++ b/tests/rest/test_presence.py @@ -1,5 +1,5 @@ # -*- coding: utf-8 -*- -# Copyright 2014 matrix.org +# Copyright 2014 OpenMarket Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. @@ -99,7 +99,7 @@ class PresenceStateTestCase(unittest.TestCase): self.assertEquals(200, code) self.assertEquals( - {"presence": ONLINE, "state": ONLINE, "status_msg": "Available"}, + {"presence": ONLINE, "status_msg": "Available"}, response ) mocked_get.assert_called_with("apple") @@ -115,7 +115,8 @@ class PresenceStateTestCase(unittest.TestCase): self.assertEquals(200, code) mocked_set.assert_called_with("apple", - {"state": UNAVAILABLE, "status_msg": "Away"}) + {"state": UNAVAILABLE, "status_msg": "Away"} + ) class PresenceListTestCase(unittest.TestCase): @@ -176,7 +177,7 @@ class PresenceListTestCase(unittest.TestCase): self.assertEquals(200, code) self.assertEquals([ - {"user_id": "@banana:test", "presence": OFFLINE, "state": OFFLINE}, + {"user_id": "@banana:test", "presence": OFFLINE}, ], response) self.datastore.get_presence_list.assert_called_with( @@ -269,11 +270,16 @@ class PresenceEventStreamTestCase(unittest.TestCase): hs.register_servlets() - hs.handlers.room_member_handler = Mock(spec=[ - "get_rooms_for_user", - ]) - hs.handlers.room_member_handler.get_rooms_for_user = ( - lambda u: defer.succeed([])) + hs.handlers.room_member_handler = Mock(spec=[]) + + self.room_members = [] + + def get_rooms_for_user(user): + if user in self.room_members: + return ["a-room"] + else: + return [] + hs.handlers.room_member_handler.get_rooms_for_user = get_rooms_for_user self.mock_datastore = hs.get_datastore() @@ -285,6 +291,17 @@ class PresenceEventStreamTestCase(unittest.TestCase): return defer.succeed(None) self.mock_datastore.get_profile_avatar_url = get_profile_avatar_url + def user_rooms_intersect(user_list): + room_member_ids = map(lambda u: u.to_string(), self.room_members) + + shared = all(map(lambda i: i in room_member_ids, user_list)) + return defer.succeed(shared) + self.mock_datastore.user_rooms_intersect = user_rooms_intersect + + def get_joined_hosts_for_room(room_id): + return [] + self.mock_datastore.get_joined_hosts_for_room = get_joined_hosts_for_room + self.presence = hs.get_handlers().presence_handler self.u_apple = hs.parse_userid("@apple:test") @@ -292,10 +309,14 @@ class PresenceEventStreamTestCase(unittest.TestCase): @defer.inlineCallbacks def test_shortpoll(self): + self.room_members = [self.u_apple, self.u_banana] + self.mock_datastore.set_presence_state.return_value = defer.succeed( - {"state": ONLINE}) + {"state": ONLINE} + ) self.mock_datastore.get_presence_list.return_value = defer.succeed( - []) + [] + ) (code, response) = yield self.mock_resource.trigger("GET", "/events?timeout=0", None) @@ -311,9 +332,11 @@ class PresenceEventStreamTestCase(unittest.TestCase): ) self.mock_datastore.set_presence_state.return_value = defer.succeed( - {"state": ONLINE}) + {"state": ONLINE} + ) self.mock_datastore.get_presence_list.return_value = defer.succeed( - []) + [] + ) yield self.presence.set_state(self.u_banana, self.u_banana, state={"presence": ONLINE} @@ -328,7 +351,6 @@ class PresenceEventStreamTestCase(unittest.TestCase): "content": { "user_id": "@banana:test", "presence": ONLINE, - "state": ONLINE, "displayname": "Frank", "last_active_ago": 0, }}, diff --git a/tests/rest/test_profile.py b/tests/rest/test_profile.py index 24456769c7..f41810df1f 100644 --- a/tests/rest/test_profile.py +++ b/tests/rest/test_profile.py @@ -1,5 +1,5 @@ # -*- coding: utf-8 -*- -# Copyright 2014 matrix.org +# Copyright 2014 OpenMarket Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. diff --git a/tests/rest/test_rooms.py b/tests/rest/test_rooms.py index cdaf948a3b..4ea5828d4f 100644 --- a/tests/rest/test_rooms.py +++ b/tests/rest/test_rooms.py @@ -1,5 +1,5 @@ # -*- coding: utf-8 -*- -# Copyright 2014 matrix.org +# Copyright 2014 OpenMarket Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. @@ -30,7 +30,7 @@ import urllib from ..utils import MockHttpResource, MemoryDataStore from .utils import RestTestCase -from mock import Mock +from mock import Mock, NonCallableMock PATH_PREFIX = "/_matrix/client/api/v1" @@ -58,7 +58,14 @@ class RoomPermissionsTestCase(RestTestCase): replication_layer=Mock(), state_handler=state_handler, persistence_service=persistence_service, + ratelimiter=NonCallableMock(spec_set=[ + "send_message", + ]), + config=NonCallableMock(), ) + self.ratelimiter = hs.get_ratelimiter() + self.ratelimiter.send_message.return_value = (True, 0) + hs.get_handlers().federation_handler = Mock() def _get_user_by_token(token=None): @@ -405,7 +412,14 @@ class RoomsMemberListTestCase(RestTestCase): replication_layer=Mock(), state_handler=state_handler, persistence_service=persistence_service, + ratelimiter=NonCallableMock(spec_set=[ + "send_message", + ]), + config=NonCallableMock(), ) + self.ratelimiter = hs.get_ratelimiter() + self.ratelimiter.send_message.return_value = (True, 0) + hs.get_handlers().federation_handler = Mock() self.auth_user_id = self.user_id @@ -483,7 +497,14 @@ class RoomsCreateTestCase(RestTestCase): replication_layer=Mock(), state_handler=state_handler, persistence_service=persistence_service, + ratelimiter=NonCallableMock(spec_set=[ + "send_message", + ]), + config=NonCallableMock(), ) + self.ratelimiter = hs.get_ratelimiter() + self.ratelimiter.send_message.return_value = (True, 0) + hs.get_handlers().federation_handler = Mock() def _get_user_by_token(token=None): @@ -573,7 +594,14 @@ class RoomTopicTestCase(RestTestCase): replication_layer=Mock(), state_handler=state_handler, persistence_service=persistence_service, + ratelimiter=NonCallableMock(spec_set=[ + "send_message", + ]), + config=NonCallableMock(), ) + self.ratelimiter = hs.get_ratelimiter() + self.ratelimiter.send_message.return_value = (True, 0) + hs.get_handlers().federation_handler = Mock() def _get_user_by_token(token=None): @@ -676,7 +704,14 @@ class RoomMemberStateTestCase(RestTestCase): replication_layer=Mock(), state_handler=state_handler, persistence_service=persistence_service, + ratelimiter=NonCallableMock(spec_set=[ + "send_message", + ]), + config=NonCallableMock(), ) + self.ratelimiter = hs.get_ratelimiter() + self.ratelimiter.send_message.return_value = (True, 0) + hs.get_handlers().federation_handler = Mock() def _get_user_by_token(token=None): @@ -801,7 +836,14 @@ class RoomMessagesTestCase(RestTestCase): replication_layer=Mock(), state_handler=state_handler, persistence_service=persistence_service, + ratelimiter=NonCallableMock(spec_set=[ + "send_message", + ]), + config=NonCallableMock(), ) + self.ratelimiter = hs.get_ratelimiter() + self.ratelimiter.send_message.return_value = (True, 0) + hs.get_handlers().federation_handler = Mock() def _get_user_by_token(token=None): diff --git a/tests/rest/utils.py b/tests/rest/utils.py index ef9a6071e2..77f5ecf0df 100644 --- a/tests/rest/utils.py +++ b/tests/rest/utils.py @@ -1,5 +1,5 @@ # -*- coding: utf-8 -*- -# Copyright 2014 matrix.org +# Copyright 2014 OpenMarket Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. diff --git a/tests/storage/test_base.py b/tests/storage/test_base.py index 5567480921..330311448d 100644 --- a/tests/storage/test_base.py +++ b/tests/storage/test_base.py @@ -1,5 +1,5 @@ # -*- coding: utf-8 -*- -# Copyright 2014 matrix.org +# Copyright 2014 OpenMarket Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. diff --git a/tests/test_distributor.py b/tests/test_distributor.py index 21c91f335b..04933f0ecf 100644 --- a/tests/test_distributor.py +++ b/tests/test_distributor.py @@ -1,5 +1,5 @@ # -*- coding: utf-8 -*- -# Copyright 2014 matrix.org +# Copyright 2014 OpenMarket Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. @@ -13,9 +13,8 @@ # See the License for the specific language governing permissions and # limitations under the License. -import unittest - from twisted.internet import defer +from twisted.trial import unittest from mock import Mock, patch @@ -75,6 +74,24 @@ class DistributorTestCase(unittest.TestCase): self.assertIsInstance(mock_logger.warning.call_args[0][0], str) + @defer.inlineCallbacks + def test_signal_catch_no_suppress(self): + # Gut-wrenching + self.dist.suppress_failures = False + + self.dist.declare("whail") + + observer = Mock() + observer.return_value = defer.fail( + Exception("Oopsie") + ) + + self.dist.observe("whail", observer) + + d = self.dist.fire("whail") + + yield self.assertFailure(d, Exception) + def test_signal_prereg(self): observer = Mock() self.dist.observe("flare", observer) @@ -85,5 +102,6 @@ class DistributorTestCase(unittest.TestCase): observer.assert_called_with(4, 5) def test_signal_undeclared(self): - with self.assertRaises(KeyError): + def code(): self.dist.fire("notification") + self.assertRaises(KeyError, code) diff --git a/tests/test_state.py b/tests/test_state.py index 58fd0bf3be..a1f5ee869b 100644 --- a/tests/test_state.py +++ b/tests/test_state.py @@ -1,5 +1,5 @@ # -*- coding: utf-8 -*- -# Copyright 2014 matrix.org +# Copyright 2014 OpenMarket Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. diff --git a/tests/test_types.py b/tests/test_types.py index d2ccbcfa55..571938356c 100644 --- a/tests/test_types.py +++ b/tests/test_types.py @@ -1,5 +1,5 @@ # -*- coding: utf-8 -*- -# Copyright 2014 matrix.org +# Copyright 2014 OpenMarket Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. diff --git a/tests/util/__init__.py b/tests/util/__init__.py index 2216c0f1ca..9bff9ec169 100644 --- a/tests/util/__init__.py +++ b/tests/util/__init__.py @@ -1,5 +1,5 @@ # -*- coding: utf-8 -*- -# Copyright 2014 matrix.org +# Copyright 2014 OpenMarket Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. diff --git a/tests/util/test_lock.py b/tests/util/test_lock.py index dd83d204d9..5623d78423 100644 --- a/tests/util/test_lock.py +++ b/tests/util/test_lock.py @@ -1,5 +1,5 @@ # -*- coding: utf-8 -*- -# Copyright 2014 matrix.org +# Copyright 2014 OpenMarket Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. diff --git a/tests/utils.py b/tests/utils.py index aa7e499e15..d90214e418 100644 --- a/tests/utils.py +++ b/tests/utils.py @@ -1,5 +1,5 @@ # -*- coding: utf-8 -*- -# Copyright 2014 matrix.org +# Copyright 2014 OpenMarket Ltd # # Licensed under the Apache License, Version 2.0 (the "License"); # you may not use this file except in compliance with the License. diff --git a/webclient/app-controller.js b/webclient/app-controller.js index 42c45f7c31..ea48cbb011 100644 --- a/webclient/app-controller.js +++ b/webclient/app-controller.js @@ -1,5 +1,5 @@ /* -Copyright 2014 matrix.org +Copyright 2014 OpenMarket Ltd Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. @@ -85,7 +85,7 @@ angular.module('MatrixWebClientController', ['matrixService', 'mPresence', 'even $scope.logout(); }); - $scope.updateHeader = function() { + $rootScope.updateHeader = function() { $scope.user_id = matrixService.config().user_id; }; diff --git a/webclient/app-directive.js b/webclient/app-directive.js index eee0d3842f..75283598ab 100644 --- a/webclient/app-directive.js +++ b/webclient/app-directive.js @@ -1,5 +1,5 @@ /* - Copyright 2014 matrix.org + Copyright 2014 OpenMarket Ltd Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. diff --git a/webclient/app-filter.js b/webclient/app-filter.js index b8d3d2a0d8..27f435674f 100644 --- a/webclient/app-filter.js +++ b/webclient/app-filter.js @@ -1,5 +1,5 @@ /* - Copyright 2014 matrix.org + Copyright 2014 OpenMarket Ltd Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. @@ -97,13 +97,55 @@ angular.module('matrixWebClient') // Else, build the name from its users var room = $rootScope.events.rooms[room_id]; if (room) { - if (room.members) { + var room_name_event = room["m.room.name"]; + + if (room_name_event) { + roomName = room_name_event.content.name; + } + else if (room.members) { // Limit the room renaming to 1:1 room if (2 === Object.keys(room.members).length) { for (var i in room.members) { var member = room.members[i]; - if (member.user_id !== matrixService.config().user_id) { - roomName = member.content.displayname ? member.content.displayname : member.user_id; + if (member.state_key !== matrixService.config().user_id) { + + if (member.state_key in $rootScope.presence) { + // If the user is available in presence, use the displayname there + // as it is the most uptodate + roomName = $rootScope.presence[member.state_key].content.displayname; + } + else if (member.content.displayname) { + roomName = member.content.displayname; + } + else { + roomName = member.state_key; + } + } + } + } + else if (1 === Object.keys(room.members).length) { + // The other member may be in the invite list, get all invited users + var invitedUserIDs = []; + for (var i in room.messages) { + var message = room.messages[i]; + if ("m.room.member" === message.type && "invite" === message.membership) { + // Make sure there is no duplicate user + if (-1 === invitedUserIDs.indexOf(message.state_key)) { + invitedUserIDs.push(message.state_key); + } + } + } + + // For now, only 1:1 room needs to be renamed. It means only 1 invited user + if (1 === invitedUserIDs.length) { + var userID = invitedUserIDs[0]; + + // Try to resolve his displayname in presence global data + if (userID in $rootScope.presence) { + roomName = $rootScope.presence[userID].content.displayname; + } + else { + roomName = userID; } } } diff --git a/webclient/app.css b/webclient/app.css index c27ec797a4..425d5bb11a 100755 --- a/webclient/app.css +++ b/webclient/app.css @@ -270,9 +270,9 @@ a:active { color: #000; } .userAvatar .userPowerLevel { position: absolute; - bottom: 20px; - height: 1px; - background-color: red; + bottom: 0px; + height: 2px; + background-color: #f00; } .userPresence { @@ -525,3 +525,13 @@ a:active { color: #000; } font-size: 24px; } +#user-displayname-input { + width: 160px; + max-width: 155px; +} + +#user-save-button { + width: 160px; + font-size: 14px; +} + diff --git a/webclient/app.js b/webclient/app.js index dac4f048cd..d25e2a6234 100644 --- a/webclient/app.js +++ b/webclient/app.js @@ -1,5 +1,5 @@ /* -Copyright 2014 matrix.org +Copyright 2014 OpenMarket Ltd Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. diff --git a/webclient/components/fileInput/file-input-directive.js b/webclient/components/fileInput/file-input-directive.js index c5e4ae07a8..14e2f772f7 100644 --- a/webclient/components/fileInput/file-input-directive.js +++ b/webclient/components/fileInput/file-input-directive.js @@ -1,5 +1,5 @@ /* - Copyright 2014 matrix.org + Copyright 2014 OpenMarket Ltd Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. diff --git a/webclient/components/fileUpload/file-upload-service.js b/webclient/components/fileUpload/file-upload-service.js index 699a3cbffc..e0f67b2c6c 100644 --- a/webclient/components/fileUpload/file-upload-service.js +++ b/webclient/components/fileUpload/file-upload-service.js @@ -1,5 +1,5 @@ /* - Copyright 2014 matrix.org + Copyright 2014 OpenMarket Ltd Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. diff --git a/webclient/components/matrix/event-handler-service.js b/webclient/components/matrix/event-handler-service.js index d6a0600132..ee478d2eb0 100644 --- a/webclient/components/matrix/event-handler-service.js +++ b/webclient/components/matrix/event-handler-service.js @@ -1,5 +1,5 @@ /* -Copyright 2014 matrix.org +Copyright 2014 OpenMarket Ltd Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. @@ -32,7 +32,9 @@ angular.module('eventHandlerService', []) var MSG_EVENT = "MSG_EVENT"; var MEMBER_EVENT = "MEMBER_EVENT"; var PRESENCE_EVENT = "PRESENCE_EVENT"; + var POWERLEVEL_EVENT = "POWERLEVEL_EVENT"; var CALL_EVENT = "CALL_EVENT"; + var NAME_EVENT = "NAME_EVENT"; var InitialSyncDeferred = $q.defer(); @@ -95,7 +97,7 @@ angular.module('eventHandlerService', []) } } - $rootScope.events.rooms[event.room_id].members[event.user_id] = event; + $rootScope.events.rooms[event.room_id].members[event.state_key] = event; $rootScope.$broadcast(MEMBER_EVENT, event, isLiveEvent); }; @@ -107,10 +109,20 @@ angular.module('eventHandlerService', []) var handlePowerLevels = function(event, isLiveEvent) { initRoom(event.room_id); - $rootScope.events.rooms[event.room_id][event.type] = event; + // Keep the latest data. Do not care of events that come when paginating back + if (!$rootScope.events.rooms[event.room_id][event.type] || isLiveEvent) { + $rootScope.events.rooms[event.room_id][event.type] = event; + $rootScope.$broadcast(POWERLEVEL_EVENT, event, isLiveEvent); + } + }; - //TODO - //$rootScope.$broadcast(PRESENCE_EVENT, event, isLiveEvent); + var handleRoomName = function(event, isLiveEvent) { + console.log("handleRoomName " + isLiveEvent); + + initRoom(event.room_id); + + $rootScope.events.rooms[event.room_id][event.type] = event; + $rootScope.$broadcast(NAME_EVENT, event, isLiveEvent); }; var handleCallEvent = function(event, isLiveEvent) { @@ -122,7 +134,9 @@ angular.module('eventHandlerService', []) MSG_EVENT: MSG_EVENT, MEMBER_EVENT: MEMBER_EVENT, PRESENCE_EVENT: PRESENCE_EVENT, + POWERLEVEL_EVENT: POWERLEVEL_EVENT, CALL_EVENT: CALL_EVENT, + NAME_EVENT: NAME_EVENT, handleEvent: function(event, isLiveEvent) { @@ -146,7 +160,9 @@ angular.module('eventHandlerService', []) case 'm.room.power_levels': handlePowerLevels(event, isLiveEvent); break; - + case 'm.room.name': + handleRoomName(event, isLiveEvent); + break; default: console.log("Unable to handle event type " + event.type); console.log(JSON.stringify(event, undefined, 4)); diff --git a/webclient/components/matrix/event-stream-service.js b/webclient/components/matrix/event-stream-service.js index 441148670e..1c0f7712b4 100644 --- a/webclient/components/matrix/event-stream-service.js +++ b/webclient/components/matrix/event-stream-service.js @@ -1,5 +1,5 @@ /* -Copyright 2014 matrix.org +Copyright 2014 OpenMarket Ltd Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. diff --git a/webclient/components/matrix/matrix-call.js b/webclient/components/matrix/matrix-call.js index 47b63d7f2f..3e13e4e81f 100644 --- a/webclient/components/matrix/matrix-call.js +++ b/webclient/components/matrix/matrix-call.js @@ -1,5 +1,5 @@ /* -Copyright 2014 matrix.org +Copyright 2014 OpenMarket Ltd Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. diff --git a/webclient/components/matrix/matrix-phone-service.js b/webclient/components/matrix/matrix-phone-service.js index d9e2e8baa3..ca86b473e7 100644 --- a/webclient/components/matrix/matrix-phone-service.js +++ b/webclient/components/matrix/matrix-phone-service.js @@ -1,5 +1,5 @@ /* -Copyright 2014 matrix.org +Copyright 2014 OpenMarket Ltd Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. diff --git a/webclient/components/matrix/matrix-service.js b/webclient/components/matrix/matrix-service.js index 2ae55bea9f..7c6d4ae50f 100644 --- a/webclient/components/matrix/matrix-service.js +++ b/webclient/components/matrix/matrix-service.js @@ -1,5 +1,5 @@ /* -Copyright 2014 matrix.org +Copyright 2014 OpenMarket Ltd Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. @@ -84,13 +84,14 @@ angular.module('matrixService', []) prefix: prefixPath, // Register an user - register: function(user_name, password) { + register: function(user_name, password, threepidCreds) { // The REST path spec var path = "/register"; return doRequest("POST", path, undefined, { user_id: user_name, - password: password + password: password, + threepidCreds: threepidCreds }); }, @@ -166,6 +167,29 @@ angular.module('matrixService', []) return doRequest("POST", path, undefined, data); }, + // Change the membership of an another user + setMembership: function(room_id, user_id, membershipValue) { + // The REST path spec + var path = "/rooms/$room_id/state/m.room.member/$user_id"; + path = path.replace("$room_id", encodeURIComponent(room_id)); + path = path.replace("$user_id", user_id); + + return doRequest("PUT", path, undefined, { + membership: membershipValue + }); + }, + + // Bans a user from from a room + ban: function(room_id, user_id, reason) { + var path = "/rooms/$room_id/ban"; + path = path.replace("$room_id", encodeURIComponent(room_id)); + + return doRequest("POST", path, undefined, { + user_id: user_id, + reason: reason + }); + }, + // Retrieves the room ID corresponding to a room alias resolveRoomAlias:function(room_alias) { var path = "/_matrix/client/api/v1/directory/room/$room_alias"; @@ -252,7 +276,7 @@ angular.module('matrixService', []) // get a list of public rooms on your home server publicRooms: function() { - var path = "/publicRooms" + var path = "/publicRooms"; return doRequest("GET", path); }, @@ -308,16 +332,16 @@ angular.module('matrixService', []) // hit the Identity Server for a 3PID request. linkEmail: function(email, clientSecret, sendAttempt) { - var path = "/_matrix/identity/api/v1/validate/email/requestToken" + var path = "/_matrix/identity/api/v1/validate/email/requestToken"; var data = "clientSecret="+clientSecret+"&email=" + encodeURIComponent(email)+"&sendAttempt="+sendAttempt; var headers = {}; headers["Content-Type"] = "application/x-www-form-urlencoded"; return doBaseRequest(config.identityServer, "POST", path, {}, data, headers); }, - authEmail: function(clientSecret, tokenId, code) { + authEmail: function(clientSecret, sid, code) { var path = "/_matrix/identity/api/v1/validate/email/submitToken"; - var data = "token="+code+"&sid="+tokenId+"&clientSecret="+clientSecret; + var data = "token="+code+"&sid="+sid+"&clientSecret="+clientSecret; var headers = {}; headers["Content-Type"] = "application/x-www-form-urlencoded"; return doBaseRequest(config.identityServer, "POST", path, {}, data, headers); @@ -330,6 +354,11 @@ angular.module('matrixService', []) headers["Content-Type"] = "application/x-www-form-urlencoded"; return doBaseRequest(config.identityServer, "POST", path, {}, data, headers); }, + + lookup3pid: function(medium, address) { + var path = "/_matrix/identity/api/v1/lookup?medium="+encodeURIComponent(medium)+"&address="+encodeURIComponent(address); + return doBaseRequest(config.identityServer, "GET", path, {}, undefined, {}); + }, uploadContent: function(file) { var path = "/_matrix/content"; @@ -408,7 +437,8 @@ angular.module('matrixService', []) state: presence }); }, - + + /****** Permanent storage of user information ******/ // Returns the current config @@ -508,6 +538,35 @@ angular.module('matrixService', []) } } return powerLevel; + }, + + /** + * Change or reset the power level of a user + * @param {String} room_id the room id + * @param {String} user_id the user id + * @param {Number} powerLevel a value between 0 and 10 + * If undefined, the user power level will be reset, ie he will use the default room user power level + * @returns {promise} an $http promise + */ + setUserPowerLevel: function(room_id, user_id, powerLevel) { + + // Hack: currently, there is no home server API so do it by hand by updating + // the current m.room.power_levels of the room and send it to the server + var room = $rootScope.events.rooms[room_id]; + if (room && room["m.room.power_levels"]) { + var content = angular.copy(room["m.room.power_levels"].content); + content[user_id] = powerLevel; + + var path = "/rooms/$room_id/state/m.room.power_levels"; + path = path.replace("$room_id", encodeURIComponent(room_id)); + + return doRequest("PUT", path, undefined, content); + } + + // The room does not exist or does not contain power_levels data + var deferred = $q.defer(); + deferred.reject({data:{error: "Invalid room: " + room_id}}); + return deferred.promise; } }; diff --git a/webclient/components/matrix/presence-service.js b/webclient/components/matrix/presence-service.js index 555118133b..952c8ec8a9 100644 --- a/webclient/components/matrix/presence-service.js +++ b/webclient/components/matrix/presence-service.js @@ -1,5 +1,5 @@ /* - Copyright 2014 matrix.org + Copyright 2014 OpenMarket Ltd Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. diff --git a/webclient/components/utilities/utilities-service.js b/webclient/components/utilities/utilities-service.js index 3df2f04458..b417cc5b39 100644 --- a/webclient/components/utilities/utilities-service.js +++ b/webclient/components/utilities/utilities-service.js @@ -1,5 +1,5 @@ /* - Copyright 2014 matrix.org + Copyright 2014 OpenMarket Ltd Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. diff --git a/webclient/home/home-controller.js b/webclient/home/home-controller.js index f4ce3053ea..85e8990c29 100644 --- a/webclient/home/home-controller.js +++ b/webclient/home/home-controller.js @@ -1,5 +1,5 @@ /* -Copyright 2014 matrix.org +Copyright 2014 OpenMarket Ltd Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. @@ -74,7 +74,7 @@ angular.module('HomeController', ['matrixService', 'eventHandlerService', 'Recen response.data.room_id, response.data.room_alias); }, function(error) { - $scope.feedback = "Failure: " + error.data; + $scope.feedback = "Failure: " + JSON.stringify(error.data); }); }; @@ -94,7 +94,7 @@ angular.module('HomeController', ['matrixService', 'eventHandlerService', 'Recen $location.url("room/" + room_id); }, function(error) { - $scope.feedback = "Can't join room: " + error.data; + $scope.feedback = "Can't join room: " + JSON.stringify(error.data); } ); }; @@ -106,7 +106,7 @@ angular.module('HomeController', ['matrixService', 'eventHandlerService', 'Recen $location.url("room/" + room_alias); }, function(error) { - $scope.feedback = "Can't join room: " + error.data; + $scope.feedback = "Can't join room: " + JSON.stringify(error.data); } ); }; diff --git a/webclient/login/login-controller.js b/webclient/login/login-controller.js index 7369a28ef0..5ef39a7122 100644 --- a/webclient/login/login-controller.js +++ b/webclient/login/login-controller.js @@ -1,5 +1,5 @@ /* - Copyright 2014 matrix.org + Copyright 2014 OpenMarket Ltd Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. @@ -15,8 +15,8 @@ */ angular.module('LoginController', ['matrixService']) -.controller('LoginController', ['$scope', '$location', 'matrixService', 'eventStreamService', - function($scope, $location, matrixService, eventStreamService) { +.controller('LoginController', ['$scope', '$rootScope', '$location', 'matrixService', 'eventStreamService', + function($scope, $rootScope, $location, matrixService, eventStreamService) { 'use strict'; @@ -51,10 +51,36 @@ angular.module('LoginController', ['matrixService']) matrixService.setConfig({ homeserver: $scope.account.homeserver, identityServer: $scope.account.identityServer, + }); + switch ($scope.login_type) { + case 'mxid': + $scope.login_with_mxid($scope.account.user_id, $scope.account.password); + break; + case 'email': + matrixService.lookup3pid('email', $scope.account.user_id).then( + function(response) { + if (response.data['address'] == undefined) { + $scope.login_error_msg = "Invalid email address / password"; + } else { + console.log("Got address "+response.data['mxid']+" for email "+$scope.account.user_id); + $scope.login_with_mxid(response.data['mxid'], $scope.account.password); + } + }, + function() { + $scope.login_error_msg = "Couldn't look up email address. Is your identity server set correctly?"; + } + ); + } + }; + + $scope.login_with_mxid = function(mxid, password) { + matrixService.setConfig({ + homeserver: $scope.account.homeserver, + identityServer: $scope.account.identityServer, user_id: $scope.account.user_id }); // try to login - matrixService.login($scope.account.user_id, $scope.account.password).then( + matrixService.login(mxid, password).then( function(response) { if ("access_token" in response.data) { $scope.feedback = "Login successful."; @@ -65,6 +91,7 @@ angular.module('LoginController', ['matrixService']) access_token: response.data.access_token }); matrixService.saveConfig(); + $rootScope.updateHeader(); eventStreamService.resume(); $location.url("home"); } diff --git a/webclient/login/register-controller.js b/webclient/login/register-controller.js index 0ece57502b..b7584a7d33 100644 --- a/webclient/login/register-controller.js +++ b/webclient/login/register-controller.js @@ -1,5 +1,5 @@ /* - Copyright 2014 matrix.org + Copyright 2014 OpenMarket Ltd Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. @@ -15,8 +15,8 @@ */ angular.module('RegisterController', ['matrixService']) -.controller('RegisterController', ['$scope', '$location', 'matrixService', 'eventStreamService', - function($scope, $location, matrixService, eventStreamService) { +.controller('RegisterController', ['$scope', '$rootScope', '$location', 'matrixService', 'eventStreamService', + function($scope, $rootScope, $location, matrixService, eventStreamService) { 'use strict'; // FIXME: factor out duplication with login-controller.js @@ -30,6 +30,17 @@ angular.module('RegisterController', ['matrixService']) { hs_url += ":" + $location.port(); } + + var generateClientSecret = function() { + var ret = ""; + var chars = "ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789"; + + for (var i = 0; i < 32; i++) { + ret += chars.charAt(Math.floor(Math.random() * chars.length)); + } + + return ret; + }; $scope.account = { homeserver: hs_url, @@ -43,7 +54,6 @@ angular.module('RegisterController', ['matrixService']) }; $scope.register = function() { - // Set the urls matrixService.setConfig({ homeserver: $scope.account.homeserver, @@ -59,7 +69,25 @@ angular.module('RegisterController', ['matrixService']) return; } - matrixService.register($scope.account.desired_user_id, $scope.account.pwd1).then( + if ($scope.account.email) { + $scope.clientSecret = generateClientSecret(); + matrixService.linkEmail($scope.account.email, $scope.clientSecret, 1).then( + function(response) { + $scope.wait_3pid_code = true; + $scope.sid = response.data.sid; + $scope.feedback = ""; + }, + function(response) { + $scope.feedback = "Couldn't request verification email!"; + } + ); + } else { + registerWithMxidAndPassword($scope.account.desired_user_id, $scope.account.pwd1); + } + }; + + $scope.registerWithMxidAndPassword = function(mxid, password, threepidCreds) { + matrixService.register(mxid, password, threepidCreds).then( function(response) { $scope.feedback = "Success"; // Update the current config @@ -74,7 +102,7 @@ angular.module('RegisterController', ['matrixService']) matrixService.saveConfig(); // Update the global scoped used_id var (used in the app header) - $scope.updateHeader(); + $rootScope.updateHeader(); eventStreamService.resume(); @@ -87,15 +115,32 @@ angular.module('RegisterController', ['matrixService']) $location.url("home"); }, function(error) { + console.trace("Registration error: "+error); if (error.data) { if (error.data.errcode === "M_USER_IN_USE") { $scope.feedback = "Username already taken."; + $scope.reenter_username = true; } } else if (error.status === 0) { $scope.feedback = "Unable to talk to the server."; } }); + } + + $scope.verifyToken = function() { + matrixService.authEmail($scope.clientSecret, $scope.sid, $scope.account.threepidtoken).then( + function(response) { + if (!response.data.success) { + $scope.feedback = "Unable to verify code."; + } else { + $scope.registerWithMxidAndPassword($scope.account.desired_user_id, $scope.account.pwd1, [{'sid':$scope.sid, 'clientSecret':$scope.clientSecret, 'idServer': $scope.account.identityServer.split('//')[1]}]); + } + }, + function(error) { + $scope.feedback = "Unable to verify code."; + } + ); }; }]); diff --git a/webclient/login/register.html b/webclient/login/register.html index 81995f1ae0..06a6526b70 100644 --- a/webclient/login/register.html +++ b/webclient/login/register.html @@ -12,26 +12,34 @@ <div style="text-align: center"> <br/> - <input id="email" size="32" type="text" ng-focus="true" ng-model="account.email" placeholder="Email address (optional)" style="display: none"/> - <div class="smallPrint" style="display: none;">Specifying an email address lets other users find you on Matrix more easily,<br/> - and gives you a way to reset your password</div> - <input id="desired_user_id" size="32" type="text" ng-model="account.desired_user_id" placeholder="Matrix ID (e.g. bob)"/> - <br/> - <input id="pwd1" size="32" type="password" ng-model="account.pwd1" placeholder="Type a password"/> - <br/> - <input id="pwd2" size="32" type="password" ng-model="account.pwd2" placeholder="Confirm your password"/> - <br/> - <input id="displayName" size="32" type="text" ng-model="account.displayName" placeholder="Display name (e.g. Bob Obson)"/> - <br/> - <br/> + + <input ng-show="!wait_3pid_code" id="email" size="32" type="text" ng-focus="true" ng-model="account.email" placeholder="Email address (optional)"/> + <div ng-show="!wait_3pid_code" class="smallPrint">Specifying an email address lets other users find you on Matrix more easily,<br/> + and will give you a way to reset your password in the future</div> + <span ng-show="reenter_username">Choose another username:</span> + <input ng-show="!wait_3pid_code || reenter_username" id="desired_user_id" size="32" type="text" ng-model="account.desired_user_id" placeholder="Matrix ID (e.g. bob)"/> + <br ng-show="!wait_3pid_code" /> + <input ng-show="!wait_3pid_code" id="pwd1" size="32" type="password" ng-model="account.pwd1" placeholder="Type a password"/> + <br ng-show="!wait_3pid_code" /> + <input ng-show="!wait_3pid_code" id="pwd2" size="32" type="password" ng-model="account.pwd2" placeholder="Confirm your password"/> + <br ng-show="!wait_3pid_code" /> + <input ng-show="!wait_3pid_code" id="displayName" size="32" type="text" ng-model="account.displayName" placeholder="Display name (e.g. Bob Obson)"/> + <br ng-show="!wait_3pid_code" /> + <br ng-show="!wait_3pid_code" /> - <button ng-click="register()" ng-disabled="!account.desired_user_id || !account.homeserver || !account.pwd1 || !account.pwd2 || account.pwd1 !== account.pwd2">Sign up</button> + <button ng-show="!wait_3pid_code" ng-click="register()" ng-disabled="!account.desired_user_id || !account.homeserver || !account.pwd1 || !account.pwd2 || account.pwd1 !== account.pwd2">Sign up</button> + + <div ng-show="wait_3pid_code"> + <span>Please enter the verification code sent to {{ account.email }}</span><br /> + <input id="threepidtoken" size="32" type="text" ng-focus="true" ng-model="account.threepidtoken" placeholder="Verification Code"/><br /> + <button ng-click="verifyToken()" ng-disabled="!account.threepidtoken">Validate</button> + </div> <br/><br/> </div> <div class="feedback">{{ feedback }} {{ login_error_msg }}</div> - <div id="serverConfig"> + <div id="serverConfig" ng-show="!wait_3pid_code"> <label for="homeserver">Home Server:</label> <input id="homeserver" size="32" type="text" ng-model="account.homeserver" placeholder="URL (e.g. http://matrix.org:8080)"/> <div class="smallPrint">Your home server stores all your conversation and account data.</div> diff --git a/webclient/recents/recents-controller.js b/webclient/recents/recents-controller.js index d7d3bf4053..3209f2cbdf 100644 --- a/webclient/recents/recents-controller.js +++ b/webclient/recents/recents-controller.js @@ -1,5 +1,5 @@ /* - Copyright 2014 matrix.org + Copyright 2014 OpenMarket Ltd Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. diff --git a/webclient/recents/recents-filter.js b/webclient/recents/recents-filter.js index 45653fca96..d80de6fbeb 100644 --- a/webclient/recents/recents-filter.js +++ b/webclient/recents/recents-filter.js @@ -1,5 +1,5 @@ /* - Copyright 2014 matrix.org + Copyright 2014 OpenMarket Ltd Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. diff --git a/webclient/recents/recents.html b/webclient/recents/recents.html index db3b0fb32f..9978e08b13 100644 --- a/webclient/recents/recents.html +++ b/webclient/recents/recents.html @@ -23,8 +23,8 @@ <div ng-hide="room.membership === 'invite'" ng-switch="room.lastMsg.type" > <div ng-switch-when="m.room.member"> {{ room.lastMsg.user_id }} - {{ {"join": "joined", "leave": "left", "invite": "invited"}[room.lastMsg.content.membership] }} - {{ room.lastMsg.content.membership === "invite" ? (room.lastMsg.state_key || '') : '' }} + {{ {"join": "joined", "leave": "left", "invite": "invited", "ban": "banned"}[msg.content.membership] }} + {{ (msg.content.membership === "invite" || msg.content.membership === "ban") ? (msg.state_key || '') : '' }} </div> <div ng-switch-when="m.room.message"> diff --git a/webclient/room/room-controller.js b/webclient/room/room-controller.js index 1f90472c67..c3f72c9d25 100644 --- a/webclient/room/room-controller.js +++ b/webclient/room/room-controller.js @@ -1,5 +1,5 @@ /* -Copyright 2014 matrix.org +Copyright 2014 OpenMarket Ltd Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. @@ -85,6 +85,14 @@ angular.module('RoomController', ['ngSanitize', 'mFileInput']) updatePresence(event); } }); + + $scope.$on(eventHandlerService.POWERLEVEL_EVENT, function(ngEvent, event, isLive) { + if (isLive && event.room_id === $scope.room_id) { + for (var user_id in event.content) { + updateUserPowerLevel(user_id); + } + } + }); $scope.memberCount = function() { return Object.keys($scope.members).length; @@ -161,10 +169,13 @@ angular.module('RoomController', ['ngSanitize', 'mFileInput']) var updateMemberList = function(chunk) { if (chunk.room_id != $scope.room_id) return; + // Ignore banned and kicked (leave) people + if ("ban" === chunk.membership || "leave" === chunk.membership) { + return; + } + // set target_user_id to keep things clear var target_user_id = chunk.state_key; - - var now = new Date().getTime(); var isNewMember = !(target_user_id in $scope.members); if (isNewMember) { @@ -174,6 +185,8 @@ angular.module('RoomController', ['ngSanitize', 'mFileInput']) } if ("last_active_ago" in chunk.content) { chunk.last_active_ago = chunk.content.last_active_ago; + $scope.now = new Date().getTime(); + chunk.last_updated = $scope.now; } if ("displayname" in chunk.content) { chunk.displayname = chunk.content.displayname; @@ -181,7 +194,6 @@ angular.module('RoomController', ['ngSanitize', 'mFileInput']) if ("avatar_url" in chunk.content) { chunk.avatar_url = chunk.content.avatar_url; } - chunk.last_updated = now; $scope.members[target_user_id] = chunk; if (target_user_id in $rootScope.presence) { @@ -197,6 +209,8 @@ angular.module('RoomController', ['ngSanitize', 'mFileInput']) } if ("last_active_ago" in chunk.content) { member.last_active_ago = chunk.content.last_active_ago; + $scope.now = new Date().getTime(); + member.last_updated = $scope.now; } } }; @@ -221,6 +235,8 @@ angular.module('RoomController', ['ngSanitize', 'mFileInput']) if ("last_active_ago" in chunk.content) { member.last_active_ago = chunk.content.last_active_ago; + $scope.now = new Date().getTime(); + member.last_updated = $scope.now; } // this may also contain a new display name or avatar url, so check. @@ -237,6 +253,29 @@ angular.module('RoomController', ['ngSanitize', 'mFileInput']) var member = $scope.members[user_id]; if (member) { member.powerLevel = matrixService.getUserPowerLevel($scope.room_id, user_id); + + normaliseMembersPowerLevels(); + } + } + + // Normalise users power levels so that the user with the higher power level + // will have a bar covering 100% of the width of his avatar + var normaliseMembersPowerLevels = function() { + // Find the max power level + var maxPowerLevel = 0; + for (var i in $scope.members) { + var member = $scope.members[i]; + if (member.powerLevel) { + maxPowerLevel = Math.max(maxPowerLevel, member.powerLevel); + } + } + + // Normalized them on a 0..100% scale to be use in css width + if (maxPowerLevel) { + for (var i in $scope.members) { + var member = $scope.members[i]; + member.powerLevelNorm = (member.powerLevel * 100) / maxPowerLevel; + } } } @@ -247,28 +286,93 @@ angular.module('RoomController', ['ngSanitize', 'mFileInput']) $scope.state.sending = true; - // Send the text message var promise; - // FIXME: handle other commands too - if ($scope.textInput.indexOf("/me") === 0) { - promise = matrixService.sendEmoteMessage($scope.room_id, $scope.textInput.substr(4)); - } - else if ($scope.textInput.indexOf("/nick ") === 0) { - // Change user display name - promise = matrixService.setDisplayName($scope.textInput.substr(6)); + + // Check for IRC style commands first + if ($scope.textInput.indexOf("/") === 0) { + var args = $scope.textInput.split(' '); + var cmd = args[0]; + + switch (cmd) { + case "/me": + var emoteMsg = args.slice(1).join(' '); + promise = matrixService.sendEmoteMessage($scope.room_id, emoteMsg); + break; + + case "/nick": + // Change user display name + if (2 === args.length) { + promise = matrixService.setDisplayName(args[1]); + } + break; + + case "/kick": + // Kick a user from the room + if (2 === args.length) { + var user_id = args[1]; + + // Set his state in the room as leave + promise = matrixService.setMembership($scope.room_id, user_id, "leave"); + } + break; + + case "/ban": + // Ban a user from the room + if (2 <= args.length) { + // TODO: The user may have entered the display name + // Need display name -> user_id resolution. Pb: how to manage user with same display names? + var user_id = args[1]; + + // Does the user provide a reason? + if (3 <= args.length) { + var reason = args.slice(2).join(' '); + } + promise = matrixService.ban($scope.room_id, user_id, reason); + } + break; + + case "/unban": + // Unban a user from the room + if (2 === args.length) { + var user_id = args[1]; + + // Reset the user membership to leave to unban him + promise = matrixService.setMembership($scope.room_id, user_id, "leave"); + } + break; + + case "/op": + // Define the power level of a user + if (3 === args.length) { + var user_id = args[1]; + var powerLevel = parseInt(args[2]); + promise = matrixService.setUserPowerLevel($scope.room_id, user_id, powerLevel); + } + break; + + case "/deop": + // Reset the power level of a user + if (2 === args.length) { + var user_id = args[1]; + promise = matrixService.setUserPowerLevel($scope.room_id, user_id, undefined); + } + break; + } } - else { + + if (!promise) { + // Send the text message promise = matrixService.sendTextMessage($scope.room_id, $scope.textInput); } promise.then( function() { - console.log("Sent message"); + console.log("Request successfully sent"); $scope.textInput = ""; $scope.state.sending = false; }, function(error) { - $scope.feedback = "Failed to send: " + error.data.error; + $scope.feedback = "Request failed: " + error.data.error; $scope.state.sending = false; }); }; @@ -332,10 +436,6 @@ angular.module('RoomController', ['ngSanitize', 'mFileInput']) eventHandlerService.waitForInitialSyncCompletion().then( function() { - // Some data has been retrieved from the iniialSync request - // So, the relative time starts here - $scope.now = new Date().getTime(); - var needsToJoin = true; // The room members is available in the data fetched by initialSync @@ -364,7 +464,8 @@ angular.module('RoomController', ['ngSanitize', 'mFileInput']) onInit3(); }, function(reason) { - $scope.feedback = "Can't join room: " + reason; + console.log("Can't join room: " + JSON.stringify(reason)); + $scope.feedback = "You do not have permission to join this room"; }); } else { diff --git a/webclient/room/room-directive.js b/webclient/room/room-directive.js index 1a99a37abb..659bcbc60f 100644 --- a/webclient/room/room-directive.js +++ b/webclient/room/room-directive.js @@ -1,5 +1,5 @@ /* - Copyright 2014 matrix.org + Copyright 2014 OpenMarket Ltd Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. diff --git a/webclient/room/room.html b/webclient/room/room.html index e672b1d7e2..6732a7b3ae 100644 --- a/webclient/room/room.html +++ b/webclient/room/room.html @@ -24,7 +24,7 @@ title="{{ member.id }}" width="80" height="80"/> <img class="userAvatarGradient" src="img/gradient.png" title="{{ member.id }}" width="80" height="24"/> - <div class="userPowerLevel" ng-style="{'width': (10 * member.powerLevel) +'%'}"></div> + <div class="userPowerLevel" ng-style="{'width': member.powerLevelNorm +'%'}"></div> <div class="userName">{{ member.displayname || member.id.substr(0, member.id.indexOf(':')) }}<br/>{{ member.displayname ? "" : member.id.substr(member.id.indexOf(':')) }}</div> </td> <td class="userPresence" ng-class="(member.presence === 'online' ? 'online' : (member.presence === 'unavailable' ? 'unavailable' : '')) + ' ' + (member.membership == 'invite' ? 'invited' : '')"> @@ -48,10 +48,23 @@ </td> <td ng-class="!msg.content.membership ? (msg.content.msgtype === 'm.emote' ? 'emote text' : 'text') : 'membership text'"> <div class="bubble"> - <span ng-show='msg.type === "m.room.member"'> + <span ng-if="'join' === msg.content.membership"> + {{ members[msg.state_key].displayname || msg.state_key }} joined + </span> + <span ng-if="'leave' === msg.content.membership"> + <span ng-if="msg.user_id === msg.state_key"> + {{ members[msg.state_key].displayname || msg.state_key }} left + </span> + <span ng-if="msg.user_id !== msg.state_key"> + {{ members[msg.user_id].displayname || msg.user_id }} + {{ {"join": "kicked", "ban": "unbanned"}[msg.content.prev] }} + {{ members[msg.state_key].displayname || msg.state_key }} + </span> + </span> + <span ng-if="'invite' === msg.content.membership || 'ban' === msg.content.membership"> {{ members[msg.user_id].displayname || msg.user_id }} - {{ {"join": "joined", "leave": "left", "invite": "invited"}[msg.content.membership] }} - {{ msg.content.membership === "invite" ? (msg.state_key || '') : '' }} + {{ {"invite": "invited", "ban": "banned"}[msg.content.membership] }} + {{ members[msg.state_key].displayname || msg.state_key }} </span> <span ng-show='msg.content.msgtype === "m.emote"' ng-bind-html="'* ' + (members[msg.user_id].displayname || msg.user_id) + ' ' + msg.content.body | linky:'_blank'"/> <span ng-show='msg.content.msgtype === "m.text"' ng-bind-html="((msg.content.msgtype === 'm.text') ? msg.content.body : '') | linky:'_blank'"/> diff --git a/webclient/settings/settings-controller.js b/webclient/settings/settings-controller.js index dc680ef075..7a26367a1b 100644 --- a/webclient/settings/settings-controller.js +++ b/webclient/settings/settings-controller.js @@ -1,5 +1,5 @@ /* -Copyright 2014 matrix.org +Copyright 2014 OpenMarket Ltd Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. diff --git a/webclient/settings/settings.html b/webclient/settings/settings.html index a69a8de300..b7fd5dfb50 100644 --- a/webclient/settings/settings.html +++ b/webclient/settings/settings.html @@ -12,18 +12,19 @@ <div class="profile-avatar"> <img ng-src="{{ (null !== profile.avatarUrl) ? profile.avatarUrl : 'img/default-profile.png' }}" m-file-input="profile.avatarFile"/> </div> - <div id="user-ids"> - <input size="40" ng-model="profile.displayName" placeholder="Your display name"/> + <div> + <input id="user-displayname-input" size="40" ng-model="profile.displayName" placeholder="Your display name"/> <br/> - <button ng-disabled="(profile.displayName == profileOnServer.displayName) && (profile.avatarUrl == profileOnServer.avatarUrl)" - ng-click="saveProfile()">Save</button> + <button id="user-save-button" + ng-disabled="(profile.displayName === profileOnServer.displayName) && (profile.avatarUrl === profileOnServer.avatarUrl)" + ng-click="saveProfile()">Save changes</button> </div> </form> </div> <br/> - <h3>Linked emails</h3> - <div class="section"> + <h3 style="display: none; ">Linked emails</h3> + <div class="section" style="display: none; "> <form> <input size="40" ng-model="linkedEmails.linkNewEmail" ng-enter="linkEmail(linkedEmails.linkNewEmail)" /> <button ng-disabled="!linkedEmails.linkNewEmail" ng-click="linkEmail(linkedEmails.linkNewEmail)"> @@ -73,6 +74,21 @@ <div>Access token: {{ config.access_token }} </div> </div> <br/> + + <h3>Commands</h3> + <div class="section"> + The following commands are available in the room chat: + <ul> + <li>/nick <display_name>: change your display name</li> + <li>/me <action>: send the action you are doing. /me will be replaced by your display name</li> + <li>/kick <user_id>: kick the user</li> + <li>/ban <user_id> [<reason>]: ban the user</li> + <li>/unban <user_id>: unban the user</li> + <li>/op <user_id> <power_level>: set user power level</li> + <li>/deop <user_id>: reset user power level to the room default value</li> + </ul> + </div> + <br/> {{ feedback }} diff --git a/webclient/user/user-controller.js b/webclient/user/user-controller.js index b5b2d439a2..3940db6683 100644 --- a/webclient/user/user-controller.js +++ b/webclient/user/user-controller.js @@ -1,5 +1,5 @@ /* -Copyright 2014 matrix.org +Copyright 2014 OpenMarket Ltd Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. |