diff --git a/docs/SUMMARY.md b/docs/SUMMARY.md
index 4fcd2b7852..fd0045e1ef 100644
--- a/docs/SUMMARY.md
+++ b/docs/SUMMARY.md
@@ -34,14 +34,16 @@
- [Application Services](application_services.md)
- [Server Notices](server_notices.md)
- [Consent Tracking](consent_tracking.md)
- - [URL Previews](url_previews.md)
+ - [URL Previews](development/url_previews.md)
- [User Directory](user_directory.md)
- [Message Retention Policies](message_retention_policies.md)
- - [Pluggable Modules](modules.md)
- - [Third Party Rules]()
- - [Spam Checker](spam_checker.md)
- - [Presence Router](presence_router_module.md)
- - [Media Storage Providers]()
+ - [Pluggable Modules](modules/index.md)
+ - [Writing a module](modules/writing_a_module.md)
+ - [Spam checker callbacks](modules/spam_checker_callbacks.md)
+ - [Third-party rules callbacks](modules/third_party_rules_callbacks.md)
+ - [Presence router callbacks](modules/presence_router_callbacks.md)
+ - [Account validity callbacks](modules/account_validity_callbacks.md)
+ - [Porting a legacy module to the new interface](modules/porting_legacy_module.md)
- [Workers](workers.md)
- [Using `synctl` with Workers](synctl_workers.md)
- [Systemd](systemd-with-workers/README.md)
diff --git a/docs/admin_api/rooms.md b/docs/admin_api/rooms.md
index 48777dd231..8e524e6509 100644
--- a/docs/admin_api/rooms.md
+++ b/docs/admin_api/rooms.md
@@ -481,32 +481,44 @@ The following fields are returned in the JSON response body:
* `new_room_id` - A string representing the room ID of the new room.
-## Undoing room shutdowns
+## Undoing room deletions
-*Note*: This guide may be outdated by the time you read it. By nature of room shutdowns being performed at the database level,
+*Note*: This guide may be outdated by the time you read it. By nature of room deletions being performed at the database level,
the structure can and does change without notice.
-First, it's important to understand that a room shutdown is very destructive. Undoing a shutdown is not as simple as pretending it
+First, it's important to understand that a room deletion is very destructive. Undoing a deletion is not as simple as pretending it
never happened - work has to be done to move forward instead of resetting the past. In fact, in some cases it might not be possible
to recover at all:
* If the room was invite-only, your users will need to be re-invited.
* If the room no longer has any members at all, it'll be impossible to rejoin.
-* The first user to rejoin will have to do so via an alias on a different server.
+* The first user to rejoin will have to do so via an alias on a different
+ server (or receive an invite from a user on a different server).
With all that being said, if you still want to try and recover the room:
-1. For safety reasons, shut down Synapse.
-2. In the database, run `DELETE FROM blocked_rooms WHERE room_id = '!example:example.org';`
- * For caution: it's recommended to run this in a transaction: `BEGIN; DELETE ...;`, verify you got 1 result, then `COMMIT;`.
- * The room ID is the same one supplied to the shutdown room API, not the Content Violation room.
-3. Restart Synapse.
+1. If the room was `block`ed, you must unblock it on your server. This can be
+ accomplished as follows:
-You will have to manually handle, if you so choose, the following:
+ 1. For safety reasons, shut down Synapse.
+ 2. In the database, run `DELETE FROM blocked_rooms WHERE room_id = '!example:example.org';`
+ * For caution: it's recommended to run this in a transaction: `BEGIN; DELETE ...;`, verify you got 1 result, then `COMMIT;`.
+ * The room ID is the same one supplied to the delete room API, not the Content Violation room.
+ 3. Restart Synapse.
-* Aliases that would have been redirected to the Content Violation room.
-* Users that would have been booted from the room (and will have been force-joined to the Content Violation room).
-* Removal of the Content Violation room if desired.
+ This step is unnecessary if `block` was not set.
+
+2. Any room aliases on your server that pointed to the deleted room may have
+ been deleted, or redirected to the Content Violation room. These will need
+ to be restored manually.
+
+3. Users on your server that were in the deleted room will have been kicked
+ from the room. Consider whether you want to update their membership
+ (possibly via the [Edit Room Membership API](room_membership.md)) or let
+ them handle rejoining themselves.
+
+4. If `new_room_user_id` was given, a 'Content Violation' will have been
+ created. Consider whether you want to delete that roomm.
## Deprecated endpoint
@@ -536,7 +548,7 @@ POST /_synapse/admin/v1/rooms/<room_id_or_alias>/make_room_admin
# Forward Extremities Admin API
Enables querying and deleting forward extremities from rooms. When a lot of forward
-extremities accumulate in a room, performance can become degraded. For details, see
+extremities accumulate in a room, performance can become degraded. For details, see
[#1760](https://github.com/matrix-org/synapse/issues/1760).
## Check for forward extremities
@@ -565,7 +577,7 @@ A response as follows will be returned:
## Deleting forward extremities
-**WARNING**: Please ensure you know what you're doing and have read
+**WARNING**: Please ensure you know what you're doing and have read
the related issue [#1760](https://github.com/matrix-org/synapse/issues/1760).
Under no situations should this API be executed as an automated maintenance task!
diff --git a/docs/development/url_previews.md b/docs/development/url_previews.md
new file mode 100644
index 0000000000..bbe05e281c
--- /dev/null
+++ b/docs/development/url_previews.md
@@ -0,0 +1,51 @@
+URL Previews
+============
+
+The `GET /_matrix/media/r0/preview_url` endpoint provides a generic preview API
+for URLs which outputs [Open Graph](https://ogp.me/) responses (with some Matrix
+specific additions).
+
+This does have trade-offs compared to other designs:
+
+* Pros:
+ * Simple and flexible; can be used by any clients at any point
+* Cons:
+ * If each homeserver provides one of these independently, all the HSes in a
+ room may needlessly DoS the target URI
+ * The URL metadata must be stored somewhere, rather than just using Matrix
+ itself to store the media.
+ * Matrix cannot be used to distribute the metadata between homeservers.
+
+When Synapse is asked to preview a URL it does the following:
+
+1. Checks against a URL blacklist (defined as `url_preview_url_blacklist` in the
+ config).
+2. Checks the in-memory cache by URLs and returns the result if it exists. (This
+ is also used to de-duplicate processing of multiple in-flight requests at once.)
+3. Kicks off a background process to generate a preview:
+ 1. Checks the database cache by URL and timestamp and returns the result if it
+ has not expired and was successful (a 2xx return code).
+ 2. Checks if the URL matches an oEmbed pattern. If it does, fetch the oEmbed
+ response. If this is an image, replace the URL to fetch and continue. If
+ if it is HTML content, use the HTML as the document and continue.
+ 3. If it doesn't match an oEmbed pattern, downloads the URL and stores it
+ into a file via the media storage provider and saves the local media
+ metadata.
+ 5. If the media is an image:
+ 1. Generates thumbnails.
+ 2. Generates an Open Graph response based on image properties.
+ 6. If the media is HTML:
+ 1. Decodes the HTML via the stored file.
+ 2. Generates an Open Graph response from the HTML.
+ 3. If an image exists in the Open Graph response:
+ 1. Downloads the URL and stores it into a file via the media storage
+ provider and saves the local media metadata.
+ 2. Generates thumbnails.
+ 3. Updates the Open Graph response based on image properties.
+ 7. Stores the result in the database cache.
+4. Returns the result.
+
+The in-memory cache expires after 1 hour.
+
+Expired entries in the database cache (and their associated media files) are
+deleted every 10 seconds. The default expiration time is 1 hour from download.
diff --git a/docs/manhole.md b/docs/manhole.md
index db92df88dc..715ed840f2 100644
--- a/docs/manhole.md
+++ b/docs/manhole.md
@@ -11,7 +11,7 @@ Note that this will give administrative access to synapse to **all users** with
shell access to the server. It should therefore **not** be enabled in
environments where untrusted users have shell access.
-***
+## Configuring the manhole
To enable it, first uncomment the `manhole` listener configuration in
`homeserver.yaml`. The configuration is slightly different if you're using docker.
@@ -52,16 +52,37 @@ listeners:
type: manhole
```
-#### Accessing synapse manhole
+### Security settings
+
+The following config options are available:
+
+- `username` - The username for the manhole (defaults to `matrix`)
+- `password` - The password for the manhole (defaults to `rabbithole`)
+- `ssh_priv_key` - The path to a private SSH key (defaults to a hardcoded value)
+- `ssh_pub_key` - The path to a public SSH key (defaults to a hardcoded value)
+
+For example:
+
+```yaml
+manhole_settings:
+ username: manhole
+ password: mypassword
+ ssh_priv_key: "/home/synapse/manhole_keys/id_rsa"
+ ssh_pub_key: "/home/synapse/manhole_keys/id_rsa.pub"
+```
+
+
+## Accessing synapse manhole
Then restart synapse, and point an ssh client at port 9000 on localhost, using
-the username `matrix`:
+the username and password configured in `homeserver.yaml` - with the default
+configuration, this would be:
```bash
ssh -p9000 matrix@localhost
```
-The password is `rabbithole`.
+Then enter the password when prompted (the default is `rabbithole`).
This gives a Python REPL in which `hs` gives access to the
`synapse.server.HomeServer` object - which in turn gives access to many other
diff --git a/docs/media_repository.md b/docs/media_repository.md
index 1bf8f16f55..99ee8f1ef7 100644
--- a/docs/media_repository.md
+++ b/docs/media_repository.md
@@ -27,4 +27,4 @@ Remote content is cached under `"remote_content"` directory. Each item of
remote content is assigned a local `"filesystem_id"` to ensure that the
directory structure `"remote_content/server_name/aa/bb/ccccccccdddddddddddd"`
is appropriate. Thumbnails for remote content are stored under
-`"remote_thumbnails/server_name/..."`
+`"remote_thumbnail/server_name/..."`
diff --git a/docs/modules.md b/docs/modules.md
deleted file mode 100644
index ae8d6f5b73..0000000000
--- a/docs/modules.md
+++ /dev/null
@@ -1,399 +0,0 @@
-# Modules
-
-Synapse supports extending its functionality by configuring external modules.
-
-## Using modules
-
-To use a module on Synapse, add it to the `modules` section of the configuration file:
-
-```yaml
-modules:
- - module: my_super_module.MySuperClass
- config:
- do_thing: true
- - module: my_other_super_module.SomeClass
- config: {}
-```
-
-Each module is defined by a path to a Python class as well as a configuration. This
-information for a given module should be available in the module's own documentation.
-
-**Note**: When using third-party modules, you effectively allow someone else to run
-custom code on your Synapse homeserver. Server admins are encouraged to verify the
-provenance of the modules they use on their homeserver and make sure the modules aren't
-running malicious code on their instance.
-
-Also note that we are currently in the process of migrating module interfaces to this
-system. While some interfaces might be compatible with it, others still require
-configuring modules in another part of Synapse's configuration file. Currently, only the
-spam checker interface is compatible with this new system.
-
-## Writing a module
-
-A module is a Python class that uses Synapse's module API to interact with the
-homeserver. It can register callbacks that Synapse will call on specific operations, as
-well as web resources to attach to Synapse's web server.
-
-When instantiated, a module is given its parsed configuration as well as an instance of
-the `synapse.module_api.ModuleApi` class. The configuration is a dictionary, and is
-either the output of the module's `parse_config` static method (see below), or the
-configuration associated with the module in Synapse's configuration file.
-
-See the documentation for the `ModuleApi` class
-[here](https://github.com/matrix-org/synapse/blob/master/synapse/module_api/__init__.py).
-
-### Handling the module's configuration
-
-A module can implement the following static method:
-
-```python
-@staticmethod
-def parse_config(config: dict) -> dict
-```
-
-This method is given a dictionary resulting from parsing the YAML configuration for the
-module. It may modify it (for example by parsing durations expressed as strings (e.g.
-"5d") into milliseconds, etc.), and return the modified dictionary. It may also verify
-that the configuration is correct, and raise an instance of
-`synapse.module_api.errors.ConfigError` if not.
-
-### Registering a web resource
-
-Modules can register web resources onto Synapse's web server using the following module
-API method:
-
-```python
-def ModuleApi.register_web_resource(path: str, resource: IResource) -> None
-```
-
-The path is the full absolute path to register the resource at. For example, if you
-register a resource for the path `/_synapse/client/my_super_module/say_hello`, Synapse
-will serve it at `http(s)://[HS_URL]/_synapse/client/my_super_module/say_hello`. Note
-that Synapse does not allow registering resources for several sub-paths in the `/_matrix`
-namespace (such as anything under `/_matrix/client` for example). It is strongly
-recommended that modules register their web resources under the `/_synapse/client`
-namespace.
-
-The provided resource is a Python class that implements Twisted's [IResource](https://twistedmatrix.com/documents/current/api/twisted.web.resource.IResource.html)
-interface (such as [Resource](https://twistedmatrix.com/documents/current/api/twisted.web.resource.Resource.html)).
-
-Only one resource can be registered for a given path. If several modules attempt to
-register a resource for the same path, the module that appears first in Synapse's
-configuration file takes priority.
-
-Modules **must** register their web resources in their `__init__` method.
-
-### Registering a callback
-
-Modules can use Synapse's module API to register callbacks. Callbacks are functions that
-Synapse will call when performing specific actions. Callbacks must be asynchronous, and
-are split in categories. A single module may implement callbacks from multiple categories,
-and is under no obligation to implement all callbacks from the categories it registers
-callbacks for.
-
-Modules can register callbacks using one of the module API's `register_[...]_callbacks`
-methods. The callback functions are passed to these methods as keyword arguments, with
-the callback name as the argument name and the function as its value. This is demonstrated
-in the example below. A `register_[...]_callbacks` method exists for each module type
-documented in this section.
-
-#### Spam checker callbacks
-
-Spam checker callbacks allow module developers to implement spam mitigation actions for
-Synapse instances. Spam checker callbacks can be registered using the module API's
-`register_spam_checker_callbacks` method.
-
-The available spam checker callbacks are:
-
-```python
-async def check_event_for_spam(event: "synapse.events.EventBase") -> Union[bool, str]
-```
-
-Called when receiving an event from a client or via federation. The module can return
-either a `bool` to indicate whether the event must be rejected because of spam, or a `str`
-to indicate the event must be rejected because of spam and to give a rejection reason to
-forward to clients.
-
-```python
-async def user_may_invite(inviter: str, invitee: str, room_id: str) -> bool
-```
-
-Called when processing an invitation. The module must return a `bool` indicating whether
-the inviter can invite the invitee to the given room. Both inviter and invitee are
-represented by their Matrix user ID (e.g. `@alice:example.com`).
-
-```python
-async def user_may_create_room(user: str) -> bool
-```
-
-Called when processing a room creation request. The module must return a `bool` indicating
-whether the given user (represented by their Matrix user ID) is allowed to create a room.
-
-```python
-async def user_may_create_room_alias(user: str, room_alias: "synapse.types.RoomAlias") -> bool
-```
-
-Called when trying to associate an alias with an existing room. The module must return a
-`bool` indicating whether the given user (represented by their Matrix user ID) is allowed
-to set the given alias.
-
-```python
-async def user_may_publish_room(user: str, room_id: str) -> bool
-```
-
-Called when trying to publish a room to the homeserver's public rooms directory. The
-module must return a `bool` indicating whether the given user (represented by their
-Matrix user ID) is allowed to publish the given room.
-
-```python
-async def check_username_for_spam(user_profile: Dict[str, str]) -> bool
-```
-
-Called when computing search results in the user directory. The module must return a
-`bool` indicating whether the given user profile can appear in search results. The profile
-is represented as a dictionary with the following keys:
-
-* `user_id`: The Matrix ID for this user.
-* `display_name`: The user's display name.
-* `avatar_url`: The `mxc://` URL to the user's avatar.
-
-The module is given a copy of the original dictionary, so modifying it from within the
-module cannot modify a user's profile when included in user directory search results.
-
-```python
-async def check_registration_for_spam(
- email_threepid: Optional[dict],
- username: Optional[str],
- request_info: Collection[Tuple[str, str]],
- auth_provider_id: Optional[str] = None,
-) -> "synapse.spam_checker_api.RegistrationBehaviour"
-```
-
-Called when registering a new user. The module must return a `RegistrationBehaviour`
-indicating whether the registration can go through or must be denied, or whether the user
-may be allowed to register but will be shadow banned.
-
-The arguments passed to this callback are:
-
-* `email_threepid`: The email address used for registering, if any.
-* `username`: The username the user would like to register. Can be `None`, meaning that
- Synapse will generate one later.
-* `request_info`: A collection of tuples, which first item is a user agent, and which
- second item is an IP address. These user agents and IP addresses are the ones that were
- used during the registration process.
-* `auth_provider_id`: The identifier of the SSO authentication provider, if any.
-
-```python
-async def check_media_file_for_spam(
- file_wrapper: "synapse.rest.media.v1.media_storage.ReadableFileWrapper",
- file_info: "synapse.rest.media.v1._base.FileInfo",
-) -> bool
-```
-
-Called when storing a local or remote file. The module must return a boolean indicating
-whether the given file can be stored in the homeserver's media store.
-
-#### Account validity callbacks
-
-Account validity callbacks allow module developers to add extra steps to verify the
-validity on an account, i.e. see if a user can be granted access to their account on the
-Synapse instance. Account validity callbacks can be registered using the module API's
-`register_account_validity_callbacks` method.
-
-The available account validity callbacks are:
-
-```python
-async def is_user_expired(user: str) -> Optional[bool]
-```
-
-Called when processing any authenticated request (except for logout requests). The module
-can return a `bool` to indicate whether the user has expired and should be locked out of
-their account, or `None` if the module wasn't able to figure it out. The user is
-represented by their Matrix user ID (e.g. `@alice:example.com`).
-
-If the module returns `True`, the current request will be denied with the error code
-`ORG_MATRIX_EXPIRED_ACCOUNT` and the HTTP status code 403. Note that this doesn't
-invalidate the user's access token.
-
-```python
-async def on_user_registration(user: str) -> None
-```
-
-Called after successfully registering a user, in case the module needs to perform extra
-operations to keep track of them. (e.g. add them to a database table). The user is
-represented by their Matrix user ID.
-
-#### Third party rules callbacks
-
-Third party rules callbacks allow module developers to add extra checks to verify the
-validity of incoming events. Third party event rules callbacks can be registered using
-the module API's `register_third_party_rules_callbacks` method.
-
-The available third party rules callbacks are:
-
-```python
-async def check_event_allowed(
- event: "synapse.events.EventBase",
- state_events: "synapse.types.StateMap",
-) -> Tuple[bool, Optional[dict]]
-```
-
-**<span style="color:red">
-This callback is very experimental and can and will break without notice. Module developers
-are encouraged to implement `check_event_for_spam` from the spam checker category instead.
-</span>**
-
-Called when processing any incoming event, with the event and a `StateMap`
-representing the current state of the room the event is being sent into. A `StateMap` is
-a dictionary that maps tuples containing an event type and a state key to the
-corresponding state event. For example retrieving the room's `m.room.create` event from
-the `state_events` argument would look like this: `state_events.get(("m.room.create", ""))`.
-The module must return a boolean indicating whether the event can be allowed.
-
-Note that this callback function processes incoming events coming via federation
-traffic (on top of client traffic). This means denying an event might cause the local
-copy of the room's history to diverge from that of remote servers. This may cause
-federation issues in the room. It is strongly recommended to only deny events using this
-callback function if the sender is a local user, or in a private federation in which all
-servers are using the same module, with the same configuration.
-
-If the boolean returned by the module is `True`, it may also tell Synapse to replace the
-event with new data by returning the new event's data as a dictionary. In order to do
-that, it is recommended the module calls `event.get_dict()` to get the current event as a
-dictionary, and modify the returned dictionary accordingly.
-
-Note that replacing the event only works for events sent by local users, not for events
-received over federation.
-
-```python
-async def on_create_room(
- requester: "synapse.types.Requester",
- request_content: dict,
- is_requester_admin: bool,
-) -> None
-```
-
-Called when processing a room creation request, with the `Requester` object for the user
-performing the request, a dictionary representing the room creation request's JSON body
-(see [the spec](https://matrix.org/docs/spec/client_server/latest#post-matrix-client-r0-createroom)
-for a list of possible parameters), and a boolean indicating whether the user performing
-the request is a server admin.
-
-Modules can modify the `request_content` (by e.g. adding events to its `initial_state`),
-or deny the room's creation by raising a `module_api.errors.SynapseError`.
-
-#### Presence router callbacks
-
-Presence router callbacks allow module developers to specify additional users (local or remote)
-to receive certain presence updates from local users. Presence router callbacks can be
-registered using the module API's `register_presence_router_callbacks` method.
-
-The available presence router callbacks are:
-
-```python
-async def get_users_for_states(
- self,
- state_updates: Iterable["synapse.api.UserPresenceState"],
-) -> Dict[str, Set["synapse.api.UserPresenceState"]]:
-```
-**Requires** `get_interested_users` to also be registered
-
-Called when processing updates to the presence state of one or more users. This callback can
-be used to instruct the server to forward that presence state to specific users. The module
-must return a dictionary that maps from Matrix user IDs (which can be local or remote) to the
-`UserPresenceState` changes that they should be forwarded.
-
-Synapse will then attempt to send the specified presence updates to each user when possible.
-
-```python
-async def get_interested_users(
- self,
- user_id: str
-) -> Union[Set[str], "synapse.module_api.PRESENCE_ALL_USERS"]
-```
-**Requires** `get_users_for_states` to also be registered
-
-Called when determining which users someone should be able to see the presence state of. This
-callback should return complementary results to `get_users_for_state` or the presence information
-may not be properly forwarded.
-
-The callback is given the Matrix user ID for a local user that is requesting presence data and
-should return the Matrix user IDs of the users whose presence state they are allowed to
-query. The returned users can be local or remote.
-
-Alternatively the callback can return `synapse.module_api.PRESENCE_ALL_USERS`
-to indicate that the user should receive updates from all known users.
-
-For example, if the user `@alice:example.org` is passed to this method, and the Set
-`{"@bob:example.com", "@charlie:somewhere.org"}` is returned, this signifies that Alice
-should receive presence updates sent by Bob and Charlie, regardless of whether these users
-share a room.
-
-### Porting an existing module that uses the old interface
-
-In order to port a module that uses Synapse's old module interface, its author needs to:
-
-* ensure the module's callbacks are all asynchronous.
-* register their callbacks using one or more of the `register_[...]_callbacks` methods
- from the `ModuleApi` class in the module's `__init__` method (see [this section](#registering-a-callback)
- for more info).
-
-Additionally, if the module is packaged with an additional web resource, the module
-should register this resource in its `__init__` method using the `register_web_resource`
-method from the `ModuleApi` class (see [this section](#registering-a-web-resource) for
-more info).
-
-The module's author should also update any example in the module's configuration to only
-use the new `modules` section in Synapse's configuration file (see [this section](#using-modules)
-for more info).
-
-### Example
-
-The example below is a module that implements the spam checker callback
-`user_may_create_room` to deny room creation to user `@evilguy:example.com`, and registers
-a web resource to the path `/_synapse/client/demo/hello` that returns a JSON object.
-
-```python
-import json
-
-from twisted.web.resource import Resource
-from twisted.web.server import Request
-
-from synapse.module_api import ModuleApi
-
-
-class DemoResource(Resource):
- def __init__(self, config):
- super(DemoResource, self).__init__()
- self.config = config
-
- def render_GET(self, request: Request):
- name = request.args.get(b"name")[0]
- request.setHeader(b"Content-Type", b"application/json")
- return json.dumps({"hello": name})
-
-
-class DemoModule:
- def __init__(self, config: dict, api: ModuleApi):
- self.config = config
- self.api = api
-
- self.api.register_web_resource(
- path="/_synapse/client/demo/hello",
- resource=DemoResource(self.config),
- )
-
- self.api.register_spam_checker_callbacks(
- user_may_create_room=self.user_may_create_room,
- )
-
- @staticmethod
- def parse_config(config):
- return config
-
- async def user_may_create_room(self, user: str) -> bool:
- if user == "@evilguy:example.com":
- return False
-
- return True
-```
diff --git a/docs/modules/account_validity_callbacks.md b/docs/modules/account_validity_callbacks.md
new file mode 100644
index 0000000000..80684b7828
--- /dev/null
+++ b/docs/modules/account_validity_callbacks.md
@@ -0,0 +1,33 @@
+# Account validity callbacks
+
+Account validity callbacks allow module developers to add extra steps to verify the
+validity on an account, i.e. see if a user can be granted access to their account on the
+Synapse instance. Account validity callbacks can be registered using the module API's
+`register_account_validity_callbacks` method.
+
+The available account validity callbacks are:
+
+### `is_user_expired`
+
+```python
+async def is_user_expired(user: str) -> Optional[bool]
+```
+
+Called when processing any authenticated request (except for logout requests). The module
+can return a `bool` to indicate whether the user has expired and should be locked out of
+their account, or `None` if the module wasn't able to figure it out. The user is
+represented by their Matrix user ID (e.g. `@alice:example.com`).
+
+If the module returns `True`, the current request will be denied with the error code
+`ORG_MATRIX_EXPIRED_ACCOUNT` and the HTTP status code 403. Note that this doesn't
+invalidate the user's access token.
+
+### `on_user_registration`
+
+```python
+async def on_user_registration(user: str) -> None
+```
+
+Called after successfully registering a user, in case the module needs to perform extra
+operations to keep track of them. (e.g. add them to a database table). The user is
+represented by their Matrix user ID.
diff --git a/docs/modules/index.md b/docs/modules/index.md
new file mode 100644
index 0000000000..3fda8cb7f0
--- /dev/null
+++ b/docs/modules/index.md
@@ -0,0 +1,34 @@
+# Modules
+
+Synapse supports extending its functionality by configuring external modules.
+
+## Using modules
+
+To use a module on Synapse, add it to the `modules` section of the configuration file:
+
+```yaml
+modules:
+ - module: my_super_module.MySuperClass
+ config:
+ do_thing: true
+ - module: my_other_super_module.SomeClass
+ config: {}
+```
+
+Each module is defined by a path to a Python class as well as a configuration. This
+information for a given module should be available in the module's own documentation.
+
+**Note**: When using third-party modules, you effectively allow someone else to run
+custom code on your Synapse homeserver. Server admins are encouraged to verify the
+provenance of the modules they use on their homeserver and make sure the modules aren't
+running malicious code on their instance.
+
+Also note that we are currently in the process of migrating module interfaces to this
+system. While some interfaces might be compatible with it, others still require
+configuring modules in another part of Synapse's configuration file.
+
+Currently, only the following pre-existing interfaces are compatible with this new system:
+
+* spam checker
+* third-party rules
+* presence router
diff --git a/docs/modules/porting_legacy_module.md b/docs/modules/porting_legacy_module.md
new file mode 100644
index 0000000000..a7a251e535
--- /dev/null
+++ b/docs/modules/porting_legacy_module.md
@@ -0,0 +1,17 @@
+# Porting an existing module that uses the old interface
+
+In order to port a module that uses Synapse's old module interface, its author needs to:
+
+* ensure the module's callbacks are all asynchronous.
+* register their callbacks using one or more of the `register_[...]_callbacks` methods
+ from the `ModuleApi` class in the module's `__init__` method (see [this section](writing_a_module.html#registering-a-callback)
+ for more info).
+
+Additionally, if the module is packaged with an additional web resource, the module
+should register this resource in its `__init__` method using the `register_web_resource`
+method from the `ModuleApi` class (see [this section](writing_a_module.html#registering-a-web-resource) for
+more info).
+
+The module's author should also update any example in the module's configuration to only
+use the new `modules` section in Synapse's configuration file (see [this section](index.html#using-modules)
+for more info).
diff --git a/docs/modules/presence_router_callbacks.md b/docs/modules/presence_router_callbacks.md
new file mode 100644
index 0000000000..4abcc9af47
--- /dev/null
+++ b/docs/modules/presence_router_callbacks.md
@@ -0,0 +1,90 @@
+# Presence router callbacks
+
+Presence router callbacks allow module developers to specify additional users (local or remote)
+to receive certain presence updates from local users. Presence router callbacks can be
+registered using the module API's `register_presence_router_callbacks` method.
+
+## Callbacks
+
+The available presence router callbacks are:
+
+### `get_users_for_states`
+
+```python
+async def get_users_for_states(
+ state_updates: Iterable["synapse.api.UserPresenceState"],
+) -> Dict[str, Set["synapse.api.UserPresenceState"]]
+```
+**Requires** `get_interested_users` to also be registered
+
+Called when processing updates to the presence state of one or more users. This callback can
+be used to instruct the server to forward that presence state to specific users. The module
+must return a dictionary that maps from Matrix user IDs (which can be local or remote) to the
+`UserPresenceState` changes that they should be forwarded.
+
+Synapse will then attempt to send the specified presence updates to each user when possible.
+
+### `get_interested_users`
+
+```python
+async def get_interested_users(
+ user_id: str
+) -> Union[Set[str], "synapse.module_api.PRESENCE_ALL_USERS"]
+```
+**Requires** `get_users_for_states` to also be registered
+
+Called when determining which users someone should be able to see the presence state of. This
+callback should return complementary results to `get_users_for_state` or the presence information
+may not be properly forwarded.
+
+The callback is given the Matrix user ID for a local user that is requesting presence data and
+should return the Matrix user IDs of the users whose presence state they are allowed to
+query. The returned users can be local or remote.
+
+Alternatively the callback can return `synapse.module_api.PRESENCE_ALL_USERS`
+to indicate that the user should receive updates from all known users.
+
+## Example
+
+The example below is a module that implements both presence router callbacks, and ensures
+that `@alice:example.org` receives all presence updates from `@bob:example.com` and
+`@charlie:somewhere.org`, regardless of whether Alice shares a room with any of them.
+
+```python
+from typing import Dict, Iterable, Set, Union
+
+from synapse.module_api import ModuleApi
+
+
+class CustomPresenceRouter:
+ def __init__(self, config: dict, api: ModuleApi):
+ self.api = api
+
+ self.api.register_presence_router_callbacks(
+ get_users_for_states=self.get_users_for_states,
+ get_interested_users=self.get_interested_users,
+ )
+
+ async def get_users_for_states(
+ self,
+ state_updates: Iterable["synapse.api.UserPresenceState"],
+ ) -> Dict[str, Set["synapse.api.UserPresenceState"]]:
+ res = {}
+ for update in state_updates:
+ if (
+ update.user_id == "@bob:example.com"
+ or update.user_id == "@charlie:somewhere.org"
+ ):
+ res.setdefault("@alice:example.com", set()).add(update)
+
+ return res
+
+ async def get_interested_users(
+ self,
+ user_id: str,
+ ) -> Union[Set[str], "synapse.module_api.PRESENCE_ALL_USERS"]:
+ if user_id == "@alice:example.com":
+ return {"@bob:example.com", "@charlie:somewhere.org"}
+
+ return set()
+```
diff --git a/docs/modules/spam_checker_callbacks.md b/docs/modules/spam_checker_callbacks.md
new file mode 100644
index 0000000000..c45eafcc4b
--- /dev/null
+++ b/docs/modules/spam_checker_callbacks.md
@@ -0,0 +1,160 @@
+# Spam checker callbacks
+
+Spam checker callbacks allow module developers to implement spam mitigation actions for
+Synapse instances. Spam checker callbacks can be registered using the module API's
+`register_spam_checker_callbacks` method.
+
+## Callbacks
+
+The available spam checker callbacks are:
+
+### `check_event_for_spam`
+
+```python
+async def check_event_for_spam(event: "synapse.events.EventBase") -> Union[bool, str]
+```
+
+Called when receiving an event from a client or via federation. The module can return
+either a `bool` to indicate whether the event must be rejected because of spam, or a `str`
+to indicate the event must be rejected because of spam and to give a rejection reason to
+forward to clients.
+
+### `user_may_invite`
+
+```python
+async def user_may_invite(inviter: str, invitee: str, room_id: str) -> bool
+```
+
+Called when processing an invitation. The module must return a `bool` indicating whether
+the inviter can invite the invitee to the given room. Both inviter and invitee are
+represented by their Matrix user ID (e.g. `@alice:example.com`).
+
+### `user_may_create_room`
+
+```python
+async def user_may_create_room(user: str) -> bool
+```
+
+Called when processing a room creation request. The module must return a `bool` indicating
+whether the given user (represented by their Matrix user ID) is allowed to create a room.
+
+### `user_may_create_room_alias`
+
+```python
+async def user_may_create_room_alias(user: str, room_alias: "synapse.types.RoomAlias") -> bool
+```
+
+Called when trying to associate an alias with an existing room. The module must return a
+`bool` indicating whether the given user (represented by their Matrix user ID) is allowed
+to set the given alias.
+
+### `user_may_publish_room`
+
+```python
+async def user_may_publish_room(user: str, room_id: str) -> bool
+```
+
+Called when trying to publish a room to the homeserver's public rooms directory. The
+module must return a `bool` indicating whether the given user (represented by their
+Matrix user ID) is allowed to publish the given room.
+
+### `check_username_for_spam`
+
+```python
+async def check_username_for_spam(user_profile: Dict[str, str]) -> bool
+```
+
+Called when computing search results in the user directory. The module must return a
+`bool` indicating whether the given user profile can appear in search results. The profile
+is represented as a dictionary with the following keys:
+
+* `user_id`: The Matrix ID for this user.
+* `display_name`: The user's display name.
+* `avatar_url`: The `mxc://` URL to the user's avatar.
+
+The module is given a copy of the original dictionary, so modifying it from within the
+module cannot modify a user's profile when included in user directory search results.
+
+### `check_registration_for_spam`
+
+```python
+async def check_registration_for_spam(
+ email_threepid: Optional[dict],
+ username: Optional[str],
+ request_info: Collection[Tuple[str, str]],
+ auth_provider_id: Optional[str] = None,
+) -> "synapse.spam_checker_api.RegistrationBehaviour"
+```
+
+Called when registering a new user. The module must return a `RegistrationBehaviour`
+indicating whether the registration can go through or must be denied, or whether the user
+may be allowed to register but will be shadow banned.
+
+The arguments passed to this callback are:
+
+* `email_threepid`: The email address used for registering, if any.
+* `username`: The username the user would like to register. Can be `None`, meaning that
+ Synapse will generate one later.
+* `request_info`: A collection of tuples, which first item is a user agent, and which
+ second item is an IP address. These user agents and IP addresses are the ones that were
+ used during the registration process.
+* `auth_provider_id`: The identifier of the SSO authentication provider, if any.
+
+### `check_media_file_for_spam`
+
+```python
+async def check_media_file_for_spam(
+ file_wrapper: "synapse.rest.media.v1.media_storage.ReadableFileWrapper",
+ file_info: "synapse.rest.media.v1._base.FileInfo",
+) -> bool
+```
+
+Called when storing a local or remote file. The module must return a boolean indicating
+whether the given file can be stored in the homeserver's media store.
+
+## Example
+
+The example below is a module that implements the spam checker callback
+`check_event_for_spam` to deny any message sent by users whose Matrix user IDs are
+mentioned in a configured list, and registers a web resource to the path
+`/_synapse/client/list_spam_checker/is_evil` that returns a JSON object indicating
+whether the provided user appears in that list.
+
+```python
+import json
+from typing import Union
+
+from twisted.web.resource import Resource
+from twisted.web.server import Request
+
+from synapse.module_api import ModuleApi
+
+
+class IsUserEvilResource(Resource):
+ def __init__(self, config):
+ super(IsUserEvilResource, self).__init__()
+ self.evil_users = config.get("evil_users") or []
+
+ def render_GET(self, request: Request):
+ user = request.args.get(b"user")[0]
+ request.setHeader(b"Content-Type", b"application/json")
+ return json.dumps({"evil": user in self.evil_users})
+
+
+class ListSpamChecker:
+ def __init__(self, config: dict, api: ModuleApi):
+ self.api = api
+ self.evil_users = config.get("evil_users") or []
+
+ self.api.register_spam_checker_callbacks(
+ check_event_for_spam=self.check_event_for_spam,
+ )
+
+ self.api.register_web_resource(
+ path="/_synapse/client/list_spam_checker/is_evil",
+ resource=IsUserEvilResource(config),
+ )
+
+ async def check_event_for_spam(self, event: "synapse.events.EventBase") -> Union[bool, str]:
+ return event.sender not in self.evil_users
+```
diff --git a/docs/modules/third_party_rules_callbacks.md b/docs/modules/third_party_rules_callbacks.md
new file mode 100644
index 0000000000..2ba6f39453
--- /dev/null
+++ b/docs/modules/third_party_rules_callbacks.md
@@ -0,0 +1,125 @@
+# Third party rules callbacks
+
+Third party rules callbacks allow module developers to add extra checks to verify the
+validity of incoming events. Third party event rules callbacks can be registered using
+the module API's `register_third_party_rules_callbacks` method.
+
+## Callbacks
+
+The available third party rules callbacks are:
+
+### `check_event_allowed`
+
+```python
+async def check_event_allowed(
+ event: "synapse.events.EventBase",
+ state_events: "synapse.types.StateMap",
+) -> Tuple[bool, Optional[dict]]
+```
+
+**<span style="color:red">
+This callback is very experimental and can and will break without notice. Module developers
+are encouraged to implement `check_event_for_spam` from the spam checker category instead.
+</span>**
+
+Called when processing any incoming event, with the event and a `StateMap`
+representing the current state of the room the event is being sent into. A `StateMap` is
+a dictionary that maps tuples containing an event type and a state key to the
+corresponding state event. For example retrieving the room's `m.room.create` event from
+the `state_events` argument would look like this: `state_events.get(("m.room.create", ""))`.
+The module must return a boolean indicating whether the event can be allowed.
+
+Note that this callback function processes incoming events coming via federation
+traffic (on top of client traffic). This means denying an event might cause the local
+copy of the room's history to diverge from that of remote servers. This may cause
+federation issues in the room. It is strongly recommended to only deny events using this
+callback function if the sender is a local user, or in a private federation in which all
+servers are using the same module, with the same configuration.
+
+If the boolean returned by the module is `True`, it may also tell Synapse to replace the
+event with new data by returning the new event's data as a dictionary. In order to do
+that, it is recommended the module calls `event.get_dict()` to get the current event as a
+dictionary, and modify the returned dictionary accordingly.
+
+Note that replacing the event only works for events sent by local users, not for events
+received over federation.
+
+### `on_create_room`
+
+```python
+async def on_create_room(
+ requester: "synapse.types.Requester",
+ request_content: dict,
+ is_requester_admin: bool,
+) -> None
+```
+
+Called when processing a room creation request, with the `Requester` object for the user
+performing the request, a dictionary representing the room creation request's JSON body
+(see [the spec](https://matrix.org/docs/spec/client_server/latest#post-matrix-client-r0-createroom)
+for a list of possible parameters), and a boolean indicating whether the user performing
+the request is a server admin.
+
+Modules can modify the `request_content` (by e.g. adding events to its `initial_state`),
+or deny the room's creation by raising a `module_api.errors.SynapseError`.
+
+### `check_threepid_can_be_invited`
+
+```python
+async def check_threepid_can_be_invited(
+ medium: str,
+ address: str,
+ state_events: "synapse.types.StateMap",
+) -> bool:
+```
+
+Called when processing an invite via a third-party identifier (i.e. email or phone number).
+The module must return a boolean indicating whether the invite can go through.
+
+### `check_visibility_can_be_modified`
+
+```python
+async def check_visibility_can_be_modified(
+ room_id: str,
+ state_events: "synapse.types.StateMap",
+ new_visibility: str,
+) -> bool:
+```
+
+Called when changing the visibility of a room in the local public room directory. The
+visibility is a string that's either "public" or "private". The module must return a
+boolean indicating whether the change can go through.
+
+## Example
+
+The example below is a module that implements the third-party rules callback
+`check_event_allowed` to censor incoming messages as dictated by a third-party service.
+
+```python
+from typing import Optional, Tuple
+
+from synapse.module_api import ModuleApi
+
+_DEFAULT_CENSOR_ENDPOINT = "https://my-internal-service.local/censor-event"
+
+class EventCensorer:
+ def __init__(self, config: dict, api: ModuleApi):
+ self.api = api
+ self._endpoint = config.get("endpoint", _DEFAULT_CENSOR_ENDPOINT)
+
+ self.api.register_third_party_rules_callbacks(
+ check_event_allowed=self.check_event_allowed,
+ )
+
+ async def check_event_allowed(
+ self,
+ event: "synapse.events.EventBase",
+ state_events: "synapse.types.StateMap",
+ ) -> Tuple[bool, Optional[dict]]:
+ event_dict = event.get_dict()
+ new_event_content = await self.api.http_client.post_json_get_json(
+ uri=self._endpoint, post_json=event_dict,
+ )
+ event_dict["content"] = new_event_content
+ return event_dict
+```
diff --git a/docs/modules/writing_a_module.md b/docs/modules/writing_a_module.md
new file mode 100644
index 0000000000..4f2fec8dc9
--- /dev/null
+++ b/docs/modules/writing_a_module.md
@@ -0,0 +1,70 @@
+# Writing a module
+
+A module is a Python class that uses Synapse's module API to interact with the
+homeserver. It can register callbacks that Synapse will call on specific operations, as
+well as web resources to attach to Synapse's web server.
+
+When instantiated, a module is given its parsed configuration as well as an instance of
+the `synapse.module_api.ModuleApi` class. The configuration is a dictionary, and is
+either the output of the module's `parse_config` static method (see below), or the
+configuration associated with the module in Synapse's configuration file.
+
+See the documentation for the `ModuleApi` class
+[here](https://github.com/matrix-org/synapse/blob/master/synapse/module_api/__init__.py).
+
+## Handling the module's configuration
+
+A module can implement the following static method:
+
+```python
+@staticmethod
+def parse_config(config: dict) -> dict
+```
+
+This method is given a dictionary resulting from parsing the YAML configuration for the
+module. It may modify it (for example by parsing durations expressed as strings (e.g.
+"5d") into milliseconds, etc.), and return the modified dictionary. It may also verify
+that the configuration is correct, and raise an instance of
+`synapse.module_api.errors.ConfigError` if not.
+
+## Registering a web resource
+
+Modules can register web resources onto Synapse's web server using the following module
+API method:
+
+```python
+def ModuleApi.register_web_resource(path: str, resource: IResource) -> None
+```
+
+The path is the full absolute path to register the resource at. For example, if you
+register a resource for the path `/_synapse/client/my_super_module/say_hello`, Synapse
+will serve it at `http(s)://[HS_URL]/_synapse/client/my_super_module/say_hello`. Note
+that Synapse does not allow registering resources for several sub-paths in the `/_matrix`
+namespace (such as anything under `/_matrix/client` for example). It is strongly
+recommended that modules register their web resources under the `/_synapse/client`
+namespace.
+
+The provided resource is a Python class that implements Twisted's [IResource](https://twistedmatrix.com/documents/current/api/twisted.web.resource.IResource.html)
+interface (such as [Resource](https://twistedmatrix.com/documents/current/api/twisted.web.resource.Resource.html)).
+
+Only one resource can be registered for a given path. If several modules attempt to
+register a resource for the same path, the module that appears first in Synapse's
+configuration file takes priority.
+
+Modules **must** register their web resources in their `__init__` method.
+
+## Registering a callback
+
+Modules can use Synapse's module API to register callbacks. Callbacks are functions that
+Synapse will call when performing specific actions. Callbacks must be asynchronous, and
+are split in categories. A single module may implement callbacks from multiple categories,
+and is under no obligation to implement all callbacks from the categories it registers
+callbacks for.
+
+Modules can register callbacks using one of the module API's `register_[...]_callbacks`
+methods. The callback functions are passed to these methods as keyword arguments, with
+the callback name as the argument name and the function as its value. This is demonstrated
+in the example below. A `register_[...]_callbacks` method exists for each category.
+
+Callbacks for each category can be found on their respective page of the
+[Synapse documentation website](https://matrix-org.github.io/synapse).
\ No newline at end of file
diff --git a/docs/sample_config.yaml b/docs/sample_config.yaml
index 935841dbfa..e15a832220 100644
--- a/docs/sample_config.yaml
+++ b/docs/sample_config.yaml
@@ -335,6 +335,24 @@ listeners:
# bind_addresses: ['::1', '127.0.0.1']
# type: manhole
+# Connection settings for the manhole
+#
+manhole_settings:
+ # The username for the manhole. This defaults to 'matrix'.
+ #
+ #username: manhole
+
+ # The password for the manhole. This defaults to 'rabbithole'.
+ #
+ #password: mypassword
+
+ # The private and public SSH key pair used to encrypt the manhole traffic.
+ # If these are left unset, then hardcoded and non-secret keys are used,
+ # which could allow traffic to be intercepted if sent over a public network.
+ #
+ #ssh_priv_key_path: CONFDIR/id_rsa
+ #ssh_pub_key_path: CONFDIR/id_rsa.pub
+
# Forward extremities can build up in a room due to networking delays between
# homeservers. Once this happens in a large room, calculation of the state of
# that room can become quite expensive. To mitigate this, once the number of
@@ -1075,6 +1093,27 @@ url_preview_accept_language:
# - en
+# oEmbed allows for easier embedding content from a website. It can be
+# used for generating URLs previews of services which support it.
+#
+oembed:
+ # A default list of oEmbed providers is included with Synapse.
+ #
+ # Uncomment the following to disable using these default oEmbed URLs.
+ # Defaults to 'false'.
+ #
+ #disable_default_providers: true
+
+ # Additional files with oEmbed configuration (each should be in the
+ # form of providers.json).
+ #
+ # By default, this list is empty (so only the default providers.json
+ # is used).
+ #
+ #additional_providers:
+ # - oembed/my_providers.json
+
+
## Captcha ##
# See docs/CAPTCHA_SETUP.md for full details of configuring this.
diff --git a/docs/upgrade.md b/docs/upgrade.md
index 453dbbabe7..f9b832cb3f 100644
--- a/docs/upgrade.md
+++ b/docs/upgrade.md
@@ -85,6 +85,15 @@ process, for example:
dpkg -i matrix-synapse-py3_1.3.0+stretch1_amd64.deb
```
+# Upgrading to v1.43.0
+
+## The spaces summary APIs can now be handled by workers
+
+The [available worker applications documentation](https://matrix-org.github.io/synapse/latest/workers.html#available-worker-applications)
+has been updated to reflect that calls to the `/spaces`, `/hierarchy`, and
+`/summary` endpoints can now be routed to workers for both client API and
+federation requests.
+
# Upgrading to v1.42.0
## Removal of old Room Admin API
@@ -112,7 +121,6 @@ process failed. See the default templates linked above for an example.
Users will stop receiving message updates via email for addresses that were
once, but not still, linked to their account.
-
# Upgrading to v1.41.0
## Add support for routing outbound HTTP requests via a proxy for federation
diff --git a/docs/url_previews.md b/docs/url_previews.md
deleted file mode 100644
index 665554e165..0000000000
--- a/docs/url_previews.md
+++ /dev/null
@@ -1,76 +0,0 @@
-URL Previews
-============
-
-Design notes on a URL previewing service for Matrix:
-
-Options are:
-
- 1. Have an AS which listens for URLs, downloads them, and inserts an event that describes their metadata.
- * Pros:
- * Decouples the implementation entirely from Synapse.
- * Uses existing Matrix events & content repo to store the metadata.
- * Cons:
- * Which AS should provide this service for a room, and why should you trust it?
- * Doesn't work well with E2E; you'd have to cut the AS into every room
- * the AS would end up subscribing to every room anyway.
-
- 2. Have a generic preview API (nothing to do with Matrix) that provides a previewing service:
- * Pros:
- * Simple and flexible; can be used by any clients at any point
- * Cons:
- * If each HS provides one of these independently, all the HSes in a room may needlessly DoS the target URI
- * We need somewhere to store the URL metadata rather than just using Matrix itself
- * We can't piggyback on matrix to distribute the metadata between HSes.
-
- 3. Make the synapse of the sending user responsible for spidering the URL and inserting an event asynchronously which describes the metadata.
- * Pros:
- * Works transparently for all clients
- * Piggy-backs nicely on using Matrix for distributing the metadata.
- * No confusion as to which AS
- * Cons:
- * Doesn't work with E2E
- * We might want to decouple the implementation of the spider from the HS, given spider behaviour can be quite complicated and evolve much more rapidly than the HS. It's more like a bot than a core part of the server.
-
- 4. Make the sending client use the preview API and insert the event itself when successful.
- * Pros:
- * Works well with E2E
- * No custom server functionality
- * Lets the client customise the preview that they send (like on FB)
- * Cons:
- * Entirely specific to the sending client, whereas it'd be nice if /any/ URL was correctly previewed if clients support it.
-
- 5. Have the option of specifying a shared (centralised) previewing service used by a room, to avoid all the different HSes in the room DoSing the target.
-
-Best solution is probably a combination of both 2 and 4.
- * Sending clients do their best to create and send a preview at the point of sending the message, perhaps delaying the message until the preview is computed? (This also lets the user validate the preview before sending)
- * Receiving clients have the option of going and creating their own preview if one doesn't arrive soon enough (or if the original sender didn't create one)
-
-This is a bit magical though in that the preview could come from two entirely different sources - the sending HS or your local one. However, this can always be exposed to users: "Generate your own URL previews if none are available?"
-
-This is tantamount also to senders calculating their own thumbnails for sending in advance of the main content - we are trusting the sender not to lie about the content in the thumbnail. Whereas currently thumbnails are calculated by the receiving homeserver to avoid this attack.
-
-However, this kind of phishing attack does exist whether we let senders pick their thumbnails or not, in that a malicious sender can send normal text messages around the attachment claiming it to be legitimate. We could rely on (future) reputation/abuse management to punish users who phish (be it with bogus metadata or bogus descriptions). Bogus metadata is particularly bad though, especially if it's avoidable.
-
-As a first cut, let's do #2 and have the receiver hit the API to calculate its own previews (as it does currently for image thumbnails). We can then extend/optimise this to option 4 as a special extra if needed.
-
-API
----
-
-```
-GET /_matrix/media/r0/preview_url?url=http://wherever.com
-200 OK
-{
- "og:type" : "article"
- "og:url" : "https://twitter.com/matrixdotorg/status/684074366691356672"
- "og:title" : "Matrix on Twitter"
- "og:image" : "https://pbs.twimg.com/profile_images/500400952029888512/yI0qtFi7_400x400.png"
- "og:description" : "“Synapse 0.12 is out! Lots of polishing, performance &amp; bugfixes: /sync API, /r0 prefix, fulltext search, 3PID invites https://t.co/5alhXLLEGP”"
- "og:site_name" : "Twitter"
-}
-```
-
-* Downloads the URL
- * If HTML, just stores it in RAM and parses it for OG meta tags
- * Download any media OG meta tags to the media repo, and refer to them in the OG via mxc:// URIs.
- * If a media filetype we know we can thumbnail: store it on disk, and hand it to the thumbnailer. Generate OG meta tags from the thumbnailer contents.
- * Otherwise, don't bother downloading further.
diff --git a/docs/workers.md b/docs/workers.md
index 3121241894..f1673d67d0 100644
--- a/docs/workers.md
+++ b/docs/workers.md
@@ -209,6 +209,8 @@ expressions:
^/_matrix/federation/v1/user/devices/
^/_matrix/federation/v1/get_groups_publicised$
^/_matrix/key/v2/query
+ ^/_matrix/federation/unstable/org.matrix.msc2946/spaces/
+ ^/_matrix/federation/unstable/org.matrix.msc2946/hierarchy/
# Inbound federation transaction request
^/_matrix/federation/v1/send/
@@ -220,6 +222,9 @@ expressions:
^/_matrix/client/(api/v1|r0|unstable)/rooms/.*/context/.*$
^/_matrix/client/(api/v1|r0|unstable)/rooms/.*/members$
^/_matrix/client/(api/v1|r0|unstable)/rooms/.*/state$
+ ^/_matrix/client/unstable/org.matrix.msc2946/rooms/.*/spaces$
+ ^/_matrix/client/unstable/org.matrix.msc2946/rooms/.*/hierarchy$
+ ^/_matrix/client/unstable/im.nheko.summary/rooms/.*/summary$
^/_matrix/client/(api/v1|r0|unstable)/account/3pid$
^/_matrix/client/(api/v1|r0|unstable)/devices$
^/_matrix/client/(api/v1|r0|unstable)/keys/query$
|
