Merge branch 'develop' into dmr/pyproject-poetry

Signed-off-by: Dan Callahan <danc@element.io>

8 files changed, 142 insertions, 93 deletions

diff --git a/docs/SUMMARY.md b/docs/SUMMARY.md
index 21f80efc99..6aa48e1919 100644
--- a/docs/SUMMARY.md
+++ b/docs/SUMMARY.md
@@ -45,6 +45,7 @@
         - [Account validity callbacks](modules/account_validity_callbacks.md)
         - [Password auth provider callbacks](modules/password_auth_provider_callbacks.md)
         - [Background update controller callbacks](modules/background_update_controller_callbacks.md)
+        - [Account data callbacks](modules/account_data_callbacks.md)
         - [Porting a legacy module to the new interface](modules/porting_legacy_module.md)
     - [Workers](workers.md)
       - [Using `synctl` with Workers](synctl_workers.md)
diff --git a/docs/development/contributing_guide.md b/docs/development/contributing_guide.md
index 4a8e402012..4b531c01ff 100644
--- a/docs/development/contributing_guide.md
+++ b/docs/development/contributing_guide.md
@@ -201,9 +201,10 @@ To do so, [configure Postgres](../postgres.md) and run `trial` with the
 following environment variables matching your configuration:
 
 - `SYNAPSE_POSTGRES` to anything nonempty
-- `SYNAPSE_POSTGRES_HOST`
-- `SYNAPSE_POSTGRES_USER`
-- `SYNAPSE_POSTGRES_PASSWORD`
+- `SYNAPSE_POSTGRES_HOST` (optional if it's the default: UNIX socket)
+- `SYNAPSE_POSTGRES_PORT` (optional if it's the default: 5432)
+- `SYNAPSE_POSTGRES_USER` (optional if using a UNIX socket)
+- `SYNAPSE_POSTGRES_PASSWORD` (optional if using a UNIX socket)
 
 For example:
 
@@ -215,6 +216,13 @@ export SYNAPSE_POSTGRES_PASSWORD=mydevenvpassword
 trial
 ```
 
+You don't need to specify the host, user, port or password if your Postgres
+server is set to authenticate you over the UNIX socket (i.e. if the `psql` command
+works without further arguments).
+
+Your Postgres account needs to be able to create databases.
+
+
 ## Run the integration tests ([Sytest](https://github.com/matrix-org/sytest)).
 
 The integration tests are a more comprehensive suite of tests. They
diff --git a/docs/development/room-dag-concepts.md b/docs/development/room-dag-concepts.md
index 3eb4d5acc4..76709487f8 100644
--- a/docs/development/room-dag-concepts.md
+++ b/docs/development/room-dag-concepts.md
@@ -39,7 +39,8 @@ yet correlated to the DAG.
 Outliers typically arise when we fetch the auth chain or state for a given
 event. When that happens, we just grab the events in the state/auth chain,
 without calculating the state at those events, or backfilling their
-`prev_events`.
+`prev_events`. Since we don't have the state at any events fetched in that
+way, we mark them as outliers.
 
 So, typically, we won't have the `prev_events` of an `outlier` in the database,
 (though it's entirely possible that we *might* have them for some other
diff --git a/docs/modules/account_data_callbacks.md b/docs/modules/account_data_callbacks.md
new file mode 100644
index 0000000000..25de911627
--- /dev/null
+++ b/docs/modules/account_data_callbacks.md
@@ -0,0 +1,106 @@
+# Account data callbacks
+
+Account data callbacks allow module developers to react to changes of the account data
+of local users. Account data callbacks can be registered using the module API's
+`register_account_data_callbacks` method.
+
+## Callbacks
+
+The available account data callbacks are:
+
+### `on_account_data_updated`
+
+_First introduced in Synapse v1.57.0_
+
+```python
+async def on_account_data_updated(
+    user_id: str,
+    room_id: Optional[str],
+    account_data_type: str,
+    content: "synapse.module_api.JsonDict",
+) -> None:
+```
+
+Called after user's account data has been updated. The module is given the
+Matrix ID of the user whose account data is changing, the room ID the data is associated
+with, the type associated with the change, as well as the new content. If the account
+data is not associated with a specific room, then the room ID is `None`.
+
+This callback is triggered when new account data is added or when the data associated with
+a given type (and optionally room) changes. This includes deletion, since in Matrix,
+deleting account data consists of replacing the data associated with a given type
+(and optionally room) with an empty dictionary (`{}`).
+
+Note that this doesn't trigger when changing the tags associated with a room, as these are
+processed separately by Synapse.
+
+If multiple modules implement this callback, Synapse runs them all in order.
+
+## Example
+
+The example below is a module that implements the `on_account_data_updated` callback, and
+sends an event to an audit room when a user changes their account data.
+
+```python
+import json
+import attr
+from typing import Any, Dict, Optional
+
+from synapse.module_api import JsonDict, ModuleApi
+from synapse.module_api.errors import ConfigError
+
+
+@attr.s(auto_attribs=True)
+class CustomAccountDataConfig:
+    audit_room: str
+    sender: str
+
+
+class CustomAccountDataModule:
+    def __init__(self, config: CustomAccountDataConfig, api: ModuleApi):
+        self.api = api
+        self.config = config
+
+        self.api.register_account_data_callbacks(
+            on_account_data_updated=self.log_new_account_data,
+        )
+
+    @staticmethod
+    def parse_config(config: Dict[str, Any]) -> CustomAccountDataConfig:
+        def check_in_config(param: str):
+            if param not in config:
+                raise ConfigError(f"'{param}' is required")
+
+        check_in_config("audit_room")
+        check_in_config("sender")
+
+        return CustomAccountDataConfig(
+            audit_room=config["audit_room"],
+            sender=config["sender"],
+        )
+
+    async def log_new_account_data(
+        self,
+        user_id: str,
+        room_id: Optional[str],
+        account_data_type: str,
+        content: JsonDict,
+    ) -> None:
+        content_raw = json.dumps(content)
+        msg_content = f"{user_id} has changed their account data for type {account_data_type} to: {content_raw}"
+
+        if room_id is not None:
+            msg_content += f" (in room {room_id})"
+
+        await self.api.create_and_send_event_into_room(
+            {
+                "room_id": self.config.audit_room,
+                "sender": self.config.sender,
+                "type": "m.room.message",
+                "content": {
+                    "msgtype": "m.text",
+                    "body": msg_content
+                }
+            }
+        )
+```
diff --git a/docs/modules/writing_a_module.md b/docs/modules/writing_a_module.md
index e7c0ffad58..e6303b739e 100644
--- a/docs/modules/writing_a_module.md
+++ b/docs/modules/writing_a_module.md
@@ -33,7 +33,7 @@ A module can implement the following static method:
 
 ```python
 @staticmethod
-def parse_config(config: dict) -> dict
+def parse_config(config: dict) -> Any
 ```
 
 This method is given a dictionary resulting from parsing the YAML configuration for the
diff --git a/docs/structured_logging.md b/docs/structured_logging.md
index 805c867653..a6667e1a11 100644
--- a/docs/structured_logging.md
+++ b/docs/structured_logging.md
@@ -78,82 +78,3 @@ loggers:
 The above logging config will set Synapse as 'INFO' logging level by default,
 with the SQL layer at 'WARNING', and will log JSON formatted messages to a
 remote endpoint at 10.1.2.3:9999.
-
-## Upgrading from legacy structured logging configuration
-
-Versions of Synapse prior to v1.54.0 automatically converted the legacy
-structured logging configuration, which was deprecated in v1.23.0, to the standard
-library logging configuration.
-
-The following reference can be used to update your configuration. Based on the
-drain `type`, we can pick a new handler:
-
-1. For a type of `console`, `console_json`, or `console_json_terse`: a handler
-   with a class of `logging.StreamHandler` and a `stream` of `ext://sys.stdout`
-   or `ext://sys.stderr` should be used.
-2. For a type of `file` or `file_json`: a handler of `logging.FileHandler` with
-   a location of the file path should be used.
-3. For a type of `network_json_terse`: a handler of `synapse.logging.RemoteHandler`
-   with the host and port should be used.
-
-Then based on the drain `type` we can pick a new formatter:
-
-1. For a type of `console` or `file` no formatter is necessary.
-2. For a type of `console_json` or `file_json`: a formatter of
-   `synapse.logging.JsonFormatter` should be used.
-3. For a type of `console_json_terse` or `network_json_terse`: a formatter of
-   `synapse.logging.TerseJsonFormatter` should be used.
-
-For each new handler and formatter they should be added to the logging configuration
-and then assigned to either a logger or the root logger.
-
-An example legacy configuration:
-
-```yaml
-structured: true
-
-loggers:
-    synapse:
-        level: INFO
-    synapse.storage.SQL:
-        level: WARNING
-
-drains:
-    console:
-        type: console
-        location: stdout
-    file:
-        type: file_json
-        location: homeserver.log
-```
-
-Would be converted into a new configuration:
-
-```yaml
-version: 1
-
-formatters:
-    json:
-        class: synapse.logging.JsonFormatter
-
-handlers:
-    console:
-        class: logging.StreamHandler
-        stream: ext://sys.stdout
-    file:
-        class: logging.FileHandler
-        formatter: json
-        filename: homeserver.log
-
-loggers:
-    synapse:
-        level: INFO
-        handlers: [console, file]
-    synapse.storage.SQL:
-        level: WARNING
-```
-
-The new logging configuration is a bit more verbose, but significantly more
-flexible. It allows for configuration that were not previously possible, such as
-sending plain logs over the network, or using different handlers for different
-modules.
diff --git a/docs/upgrade.md b/docs/upgrade.md
index 18649a17d3..3a2d1c5c13 100644
--- a/docs/upgrade.md
+++ b/docs/upgrade.md
@@ -88,8 +88,26 @@ process, for example:
     dpkg -i matrix-synapse-py3_1.3.0+stretch1_amd64.deb
     ```
 
+# Upgrading to v1.57.0
+
+## Changes to database schema for application services
+
+Synapse v1.57.0 includes a [change](https://github.com/matrix-org/synapse/pull/12209) to the
+way transaction IDs are managed for application services. If your deployment uses a dedicated
+worker for application service traffic, **it must be stopped** when the database is upgraded
+(which normally happens when the main process is upgraded), to ensure the change is made safely
+without any risk of reusing transaction IDs.
+
+Deployments which do not use separate worker processes can be upgraded as normal. Similarly,
+deployments where no applciation services are in use can be upgraded as normal.
+
 # Upgrading to v1.56.0
 
+## Open registration without verification is now disabled by default
+
+Synapse will refuse to start if registration is enabled without email, captcha, or token-based verification unless the new config
+flag `enable_registration_without_verification` is set to "true".
+
 ## Groups/communities feature has been deprecated
 
 The non-standard groups/communities feature in Synapse has been deprecated and will
@@ -111,12 +129,6 @@ for more information and instructions on how to fix a database with incorrect va
 
 # Upgrading to v1.55.0
 
-## Open registration without verification is now disabled by default
-
-Synapse will refuse to start if registration is enabled without email, captcha, or token-based verification unless the new config 
-flag `enable_registration_without_verification` is set to "true".
-
-
 ## `synctl` script has been moved
 
 The `synctl` script
@@ -151,7 +163,7 @@ please upgrade Mjolnir to version 1.3.2 or later before upgrading Synapse.
 This release removes support for the `structured: true` logging configuration
 which was deprecated in Synapse v1.23.0. If your logging configuration contains
 `structured: true` then it should be modified based on the
-[structured logging documentation](structured_logging.md).
+[structured logging documentation](https://matrix-org.github.io/synapse/v1.56/structured_logging.html#upgrading-from-legacy-structured-logging-configuration).
 
 # Upgrading to v1.53.0
 
@@ -768,7 +780,7 @@ lock down external access to the Admin API endpoints.
 This release deprecates use of the `structured: true` logging
 configuration for structured logging. If your logging configuration
 contains `structured: true` then it should be modified based on the
-[structured logging documentation](structured_logging.md).
+[structured logging documentation](https://matrix-org.github.io/synapse/v1.56/structured_logging.html#upgrading-from-legacy-structured-logging-configuration).
 
 The `structured` and `drains` logging options are now deprecated and
 should be replaced by standard logging configuration of `handlers` and
diff --git a/docs/workers.md b/docs/workers.md
index 8ac95e39bb..caef44b614 100644
--- a/docs/workers.md
+++ b/docs/workers.md
@@ -27,7 +27,7 @@ feeds streams of newly written data between processes so they can be kept in
 sync with the database state.
 
 When configured to do so, Synapse uses a
-[Redis pub/sub channel](https://redis.io/topics/pubsub) to send the replication
+[Redis pub/sub channel](https://redis.io/docs/manual/pubsub/) to send the replication
 stream between all configured Synapse processes. Additionally, processes may
 make HTTP requests to each other, primarily for operations which need to wait
 for a reply ─ such as sending an event.