diff --git a/CHANGES.md b/CHANGES.md
index 0e5e052951..01766af39c 100644
--- a/CHANGES.md
+++ b/CHANGES.md
@@ -1,3 +1,82 @@
+Synapse 1.41.0rc1 (2021-08-18)
+==============================
+
+Features
+--------
+
+- Add `get_userinfo_by_id` method to ModuleApi. ([\#9581](https://github.com/matrix-org/synapse/issues/9581))
+- Initial local support for [MSC3266](https://github.com/matrix-org/synapse/pull/10394), Room Summary over the unstable `/rooms/{roomIdOrAlias}/summary` API. ([\#10394](https://github.com/matrix-org/synapse/issues/10394))
+- Experimental support for [MSC3288](https://github.com/matrix-org/matrix-doc/pull/3288), sending `room_type` to the identity server for 3pid invites over the `/store-invite` API. ([\#10435](https://github.com/matrix-org/synapse/issues/10435))
+- Add support for sending federation requests through a proxy. Contributed by @Bubu and @dklimpel. ([\#10475](https://github.com/matrix-org/synapse/issues/10475))
+- Add support for "marker" events which makes historical events discoverable for servers that already have all of the scrollback history (part of [MSC2716](https://github.com/matrix-org/matrix-doc/pull/2716)). ([\#10498](https://github.com/matrix-org/synapse/issues/10498))
+- Add a configuration setting for the time a `/sync` response is cached for. ([\#10513](https://github.com/matrix-org/synapse/issues/10513))
+- The default logging handler for new installations is now `PeriodicallyFlushingMemoryHandler`, a buffered logging handler which periodically flushes itself. ([\#10518](https://github.com/matrix-org/synapse/issues/10518))
+- Add support for new redaction rules for historical events specified in [MSC2716](https://github.com/matrix-org/matrix-doc/pull/2716). ([\#10538](https://github.com/matrix-org/synapse/issues/10538))
+- Add a setting to disable TLS when sending email. ([\#10546](https://github.com/matrix-org/synapse/issues/10546))
+- Add pagination to the spaces summary based on updates to [MSC2946](https://github.com/matrix-org/matrix-doc/pull/2946). ([\#10549](https://github.com/matrix-org/synapse/issues/10549), [\#10560](https://github.com/matrix-org/synapse/issues/10560), [\#10569](https://github.com/matrix-org/synapse/issues/10569), [\#10574](https://github.com/matrix-org/synapse/issues/10574), [\#10575](https://github.com/matrix-org/synapse/issues/10575), [\#10579](https://github.com/matrix-org/synapse/issues/10579), [\#10583](https://github.com/matrix-org/synapse/issues/10583))
+- Admin API to delete several media for a specific user. Contributed by @dklimpel. ([\#10558](https://github.com/matrix-org/synapse/issues/10558), [\#10628](https://github.com/matrix-org/synapse/issues/10628))
+- Add support for routing `/createRoom` to workers. ([\#10564](https://github.com/matrix-org/synapse/issues/10564))
+- Update the Synapse Grafana dashboard. ([\#10570](https://github.com/matrix-org/synapse/issues/10570))
+- Add an admin API (`GET /_synapse/admin/username_available`) to check if a username is available (regardless of registration settings). ([\#10578](https://github.com/matrix-org/synapse/issues/10578))
+- Allow editing a user's `external_ids` via the "Edit User" admin API. Contributed by @dklimpel. ([\#10598](https://github.com/matrix-org/synapse/issues/10598))
+- The Synapse manhole no longer needs coroutines to be wrapped in `defer.ensureDeferred`. ([\#10602](https://github.com/matrix-org/synapse/issues/10602))
+- Add option to allow modules to run periodic tasks on all instances, rather than just the one configured to run background tasks. ([\#10638](https://github.com/matrix-org/synapse/issues/10638))
+
+
+Bugfixes
+--------
+
+- Add some clarification to the sample config file. Contributed by @Kentokamoto. ([\#10129](https://github.com/matrix-org/synapse/issues/10129))
+- Fix a long-standing bug where protocols which are not implemented by any appservices were incorrectly returned via `GET /_matrix/client/r0/thirdparty/protocols`. ([\#10532](https://github.com/matrix-org/synapse/issues/10532))
+- Fix exceptions in logs when failing to get remote room list. ([\#10541](https://github.com/matrix-org/synapse/issues/10541))
+- Fix longstanding bug which caused the user "status" to be reset when the user went offline. Contributed by @dklimpel. ([\#10550](https://github.com/matrix-org/synapse/issues/10550))
+- Allow public rooms to be previewed in the spaces summary APIs from [MSC2946](https://github.com/matrix-org/matrix-doc/pull/2946). ([\#10580](https://github.com/matrix-org/synapse/issues/10580))
+- Fix a bug introduced in v1.37.1 where an error could occur in the asynchronous processing of PDUs when the queue was empty. ([\#10592](https://github.com/matrix-org/synapse/issues/10592))
+- Fix errors on /sync when read receipt data is a string. Only affects homeservers with the experimental flag for [MSC2285](https://github.com/matrix-org/matrix-doc/pull/2285) enabled. Contributed by @SimonBrandner. ([\#10606](https://github.com/matrix-org/synapse/issues/10606))
+- Additional validation for the spaces summary API to avoid errors like `ValueError: Stop argument for islice() must be None or an integer`. The missing validation has existed since v1.31.0. ([\#10611](https://github.com/matrix-org/synapse/issues/10611))
+- Revert behaviour introduced in v1.38.0 that strips `org.matrix.msc2732.device_unused_fallback_key_types` from `/sync` when its value is empty. This field should instead always be present according to [MSC2732](https://github.com/matrix-org/matrix-doc/blob/master/proposals/2732-olm-fallback-keys.md). ([\#10623](https://github.com/matrix-org/synapse/issues/10623))
+
+
+Improved Documentation
+----------------------
+
+- Add documentation for configuration a forward proxy. ([\#10443](https://github.com/matrix-org/synapse/issues/10443))
+- Updated the reverse proxy documentation to highlight the homserver configuration that is needed to make Synapse aware that is is intentionally reverse proxied. ([\#10551](https://github.com/matrix-org/synapse/issues/10551))
+- Update CONTRIBUTING.md to fix index links and the instructions for SyTest in docker. ([\#10599](https://github.com/matrix-org/synapse/issues/10599))
+
+
+Deprecations and Removals
+-------------------------
+
+- No longer build `.deb` packages for Ubuntu 20.10 LTS Groovy Gorilla, which has now EOLed. ([\#10588](https://github.com/matrix-org/synapse/issues/10588))
+- The `template_dir` configuration settings in the `sso`, `account_validity` and `email` sections of the configuration file are now deprecated in favour of the global `templates.custom_template_directory` setting. See the [upgrade notes](https://matrix-org.github.io/synapse/latest/upgrade.html) for more information. ([\#10596](https://github.com/matrix-org/synapse/issues/10596))
+
+
+Internal Changes
+----------------
+
+- Improve event caching mechanism to avoid having multiple copies of an event in memory at a time. ([\#10119](https://github.com/matrix-org/synapse/issues/10119))
+- Reduce errors in PostgreSQL logs due to concurrent serialization errors. ([\#10504](https://github.com/matrix-org/synapse/issues/10504))
+- Include room ID in ignored EDU log messages. Contributed by @ilmari. ([\#10507](https://github.com/matrix-org/synapse/issues/10507))
+- Add pagination to the spaces summary based on updates to [MSC2946](https://github.com/matrix-org/matrix-doc/pull/2946). ([\#10527](https://github.com/matrix-org/synapse/issues/10527), [\#10530](https://github.com/matrix-org/synapse/issues/10530))
+- Fix CI to not break when run against branches rather than pull requests. ([\#10529](https://github.com/matrix-org/synapse/issues/10529))
+- Mark all events stemming from the [MSC2716](https://github.com/matrix-org/matrix-doc/pull/2716) `/batch_send` endpoint as historical. ([\#10537](https://github.com/matrix-org/synapse/issues/10537))
+- Clean up some of the federation event authentication code for clarity. ([\#10539](https://github.com/matrix-org/synapse/issues/10539), [\#10591](https://github.com/matrix-org/synapse/issues/10591))
+- Convert `Transaction` and `Edu` objects to attrs. ([\#10542](https://github.com/matrix-org/synapse/issues/10542))
+- Update `/batch_send` endpoint to only return `state_events` created by the `state_events_from_before` passed in. ([\#10552](https://github.com/matrix-org/synapse/issues/10552))
+- Update contributing.md to warn against rebasing an open PR. ([\#10563](https://github.com/matrix-org/synapse/issues/10563))
+- Remove the unused public rooms replication stream. ([\#10565](https://github.com/matrix-org/synapse/issues/10565))
+- Clarify error message when failing to join a restricted room. ([\#10572](https://github.com/matrix-org/synapse/issues/10572))
+- Remove references to BuildKite in favour of GitHub Actions. ([\#10573](https://github.com/matrix-org/synapse/issues/10573))
+- Move `/batch_send` endpoint defined by [MSC2716](https://github.com/matrix-org/matrix-doc/pull/2716) to the `/v2_alpha` directory. ([\#10576](https://github.com/matrix-org/synapse/issues/10576))
+- Allow multiple custom directories in `read_templates`. ([\#10587](https://github.com/matrix-org/synapse/issues/10587))
+- Re-organize the `synapse.federation.transport.server` module to create smaller files. ([\#10590](https://github.com/matrix-org/synapse/issues/10590))
+- Flatten the `synapse.rest.client` package by moving the contents of `v1` and `v2_alpha` into the parent. ([\#10600](https://github.com/matrix-org/synapse/issues/10600))
+- Build Debian packages for Debian 12 (Bookworm). ([\#10612](https://github.com/matrix-org/synapse/issues/10612))
+- Fix up a couple of links to the database schema documentation. ([\#10620](https://github.com/matrix-org/synapse/issues/10620))
+- Fix a broken link to the upgrade notes. ([\#10631](https://github.com/matrix-org/synapse/issues/10631))
+
+
Synapse 1.40.0 (2021-08-10)
===========================
diff --git a/UPGRADE.rst b/UPGRADE.rst
index 17ecd935fd..6c7f9cb18e 100644
--- a/UPGRADE.rst
+++ b/UPGRADE.rst
@@ -1,7 +1,7 @@
Upgrading Synapse
=================
-This document has moved to the `Synapse documentation website <https://matrix-org.github.io/synapse/latest/upgrading>`_.
+This document has moved to the `Synapse documentation website <https://matrix-org.github.io/synapse/latest/upgrade>`_.
Please update your links.
The markdown source is available in `docs/upgrade.md <docs/upgrade.md>`_.
diff --git a/changelog.d/10119.misc b/changelog.d/10119.misc
deleted file mode 100644
index f70dc6496f..0000000000
--- a/changelog.d/10119.misc
+++ /dev/null
@@ -1 +0,0 @@
-Improve event caching mechanism to avoid having multiple copies of an event in memory at a time.
diff --git a/changelog.d/10129.bugfix b/changelog.d/10129.bugfix
deleted file mode 100644
index 292676ec8d..0000000000
--- a/changelog.d/10129.bugfix
+++ /dev/null
@@ -1 +0,0 @@
-Add some clarification to the sample config file. Contributed by @Kentokamoto.
diff --git a/changelog.d/10394.feature b/changelog.d/10394.feature
deleted file mode 100644
index c8bbc5a740..0000000000
--- a/changelog.d/10394.feature
+++ /dev/null
@@ -1 +0,0 @@
-Initial local support for [MSC3266](https://github.com/matrix-org/synapse/pull/10394), Room Summary over the unstable `/rooms/{roomIdOrAlias}/summary` API.
diff --git a/changelog.d/10435.feature b/changelog.d/10435.feature
deleted file mode 100644
index f93ef5b415..0000000000
--- a/changelog.d/10435.feature
+++ /dev/null
@@ -1 +0,0 @@
-Experimental support for [MSC3288](https://github.com/matrix-org/matrix-doc/pull/3288), sending `room_type` to the identity server for 3pid invites over the `/store-invite` API.
diff --git a/changelog.d/10443.doc b/changelog.d/10443.doc
deleted file mode 100644
index 3588e5487f..0000000000
--- a/changelog.d/10443.doc
+++ /dev/null
@@ -1 +0,0 @@
-Add documentation for configuration a forward proxy.
diff --git a/changelog.d/10475.feature b/changelog.d/10475.feature
deleted file mode 100644
index 52eab11b03..0000000000
--- a/changelog.d/10475.feature
+++ /dev/null
@@ -1 +0,0 @@
-Add support for sending federation requests through a proxy. Contributed by @Bubu and @dklimpel.
\ No newline at end of file
diff --git a/changelog.d/10498.feature b/changelog.d/10498.feature
deleted file mode 100644
index 5df896572d..0000000000
--- a/changelog.d/10498.feature
+++ /dev/null
@@ -1 +0,0 @@
-Add support for "marker" events which makes historical events discoverable for servers that already have all of the scrollback history (part of MSC2716).
diff --git a/changelog.d/10504.misc b/changelog.d/10504.misc
deleted file mode 100644
index 1479a5022d..0000000000
--- a/changelog.d/10504.misc
+++ /dev/null
@@ -1 +0,0 @@
-Reduce errors in PostgreSQL logs due to concurrent serialization errors.
diff --git a/changelog.d/10507.misc b/changelog.d/10507.misc
deleted file mode 100644
index 5dfd116e60..0000000000
--- a/changelog.d/10507.misc
+++ /dev/null
@@ -1 +0,0 @@
-Include room ID in ignored EDU log messages. Contributed by @ilmari.
diff --git a/changelog.d/10513.feature b/changelog.d/10513.feature
deleted file mode 100644
index 153b2df7b2..0000000000
--- a/changelog.d/10513.feature
+++ /dev/null
@@ -1 +0,0 @@
-Add a configuration setting for the time a `/sync` response is cached for.
diff --git a/changelog.d/10527.misc b/changelog.d/10527.misc
deleted file mode 100644
index ffc4e4289c..0000000000
--- a/changelog.d/10527.misc
+++ /dev/null
@@ -1 +0,0 @@
-Add pagination to the spaces summary based on updates to [MSC2946](https://github.com/matrix-org/matrix-doc/pull/2946).
diff --git a/changelog.d/10529.misc b/changelog.d/10529.misc
deleted file mode 100644
index 4caf22523c..0000000000
--- a/changelog.d/10529.misc
+++ /dev/null
@@ -1 +0,0 @@
-Fix CI to not break when run against branches rather than pull requests.
diff --git a/changelog.d/10530.misc b/changelog.d/10530.misc
deleted file mode 100644
index ffc4e4289c..0000000000
--- a/changelog.d/10530.misc
+++ /dev/null
@@ -1 +0,0 @@
-Add pagination to the spaces summary based on updates to [MSC2946](https://github.com/matrix-org/matrix-doc/pull/2946).
diff --git a/changelog.d/10532.bugfix b/changelog.d/10532.bugfix
deleted file mode 100644
index d95e3d9b59..0000000000
--- a/changelog.d/10532.bugfix
+++ /dev/null
@@ -1 +0,0 @@
-Fix a long-standing bug where protocols which are not implemented by any appservices were incorrectly returned via `GET /_matrix/client/r0/thirdparty/protocols`.
diff --git a/changelog.d/10537.misc b/changelog.d/10537.misc
deleted file mode 100644
index c9e045300c..0000000000
--- a/changelog.d/10537.misc
+++ /dev/null
@@ -1 +0,0 @@
-Mark all events stemming from the MSC2716 `/batch_send` endpoint as historical.
diff --git a/changelog.d/10538.feature b/changelog.d/10538.feature
deleted file mode 100644
index 120c8e8ca0..0000000000
--- a/changelog.d/10538.feature
+++ /dev/null
@@ -1 +0,0 @@
-Add support for new redaction rules for historical events specified in [MSC2716](https://github.com/matrix-org/matrix-doc/pull/2716).
diff --git a/changelog.d/10539.misc b/changelog.d/10539.misc
deleted file mode 100644
index 9a765435db..0000000000
--- a/changelog.d/10539.misc
+++ /dev/null
@@ -1 +0,0 @@
-Clean up some of the federation event authentication code for clarity.
diff --git a/changelog.d/10541.bugfix b/changelog.d/10541.bugfix
deleted file mode 100644
index bb946e0920..0000000000
--- a/changelog.d/10541.bugfix
+++ /dev/null
@@ -1 +0,0 @@
-Fix exceptions in logs when failing to get remote room list.
diff --git a/changelog.d/10542.misc b/changelog.d/10542.misc
deleted file mode 100644
index 44b70b4730..0000000000
--- a/changelog.d/10542.misc
+++ /dev/null
@@ -1 +0,0 @@
-Convert `Transaction` and `Edu` objects to attrs.
diff --git a/changelog.d/10546.feature b/changelog.d/10546.feature
deleted file mode 100644
index 7709d010b3..0000000000
--- a/changelog.d/10546.feature
+++ /dev/null
@@ -1 +0,0 @@
-Add a setting to disable TLS when sending email.
diff --git a/changelog.d/10549.feature b/changelog.d/10549.feature
deleted file mode 100644
index ffc4e4289c..0000000000
--- a/changelog.d/10549.feature
+++ /dev/null
@@ -1 +0,0 @@
-Add pagination to the spaces summary based on updates to [MSC2946](https://github.com/matrix-org/matrix-doc/pull/2946).
diff --git a/changelog.d/10550.bugfix b/changelog.d/10550.bugfix
deleted file mode 100644
index 2e1b7c8bbb..0000000000
--- a/changelog.d/10550.bugfix
+++ /dev/null
@@ -1 +0,0 @@
-Fix longstanding bug which caused the user "status" to be reset when the user went offline. Contributed by @dklimpel.
diff --git a/changelog.d/10551.doc b/changelog.d/10551.doc
deleted file mode 100644
index 4a2b0785bf..0000000000
--- a/changelog.d/10551.doc
+++ /dev/null
@@ -1 +0,0 @@
-Updated the reverse proxy documentation to highlight the homserver configuration that is needed to make Synapse aware that is is intentionally reverse proxied.
diff --git a/changelog.d/10552.misc b/changelog.d/10552.misc
deleted file mode 100644
index fc5f6aea5f..0000000000
--- a/changelog.d/10552.misc
+++ /dev/null
@@ -1 +0,0 @@
-Update `/batch_send` endpoint to only return `state_events` created by the `state_events_from_before` passed in.
diff --git a/changelog.d/10558.feature b/changelog.d/10558.feature
deleted file mode 100644
index 1f461bc70a..0000000000
--- a/changelog.d/10558.feature
+++ /dev/null
@@ -1 +0,0 @@
-Admin API to delete several media for a specific user. Contributed by @dklimpel.
diff --git a/changelog.d/10560.feature b/changelog.d/10560.feature
deleted file mode 100644
index ffc4e4289c..0000000000
--- a/changelog.d/10560.feature
+++ /dev/null
@@ -1 +0,0 @@
-Add pagination to the spaces summary based on updates to [MSC2946](https://github.com/matrix-org/matrix-doc/pull/2946).
diff --git a/changelog.d/10563.misc b/changelog.d/10563.misc
deleted file mode 100644
index 8e4e90c8f4..0000000000
--- a/changelog.d/10563.misc
+++ /dev/null
@@ -1 +0,0 @@
-Update contributing.md to warn against rebasing an open PR.
diff --git a/changelog.d/10569.feature b/changelog.d/10569.feature
deleted file mode 100644
index ffc4e4289c..0000000000
--- a/changelog.d/10569.feature
+++ /dev/null
@@ -1 +0,0 @@
-Add pagination to the spaces summary based on updates to [MSC2946](https://github.com/matrix-org/matrix-doc/pull/2946).
diff --git a/changelog.d/10570.feature b/changelog.d/10570.feature
deleted file mode 100644
index bd432685b3..0000000000
--- a/changelog.d/10570.feature
+++ /dev/null
@@ -1 +0,0 @@
-Update the Synapse Grafana dashboard.
diff --git a/changelog.d/10572.misc b/changelog.d/10572.misc
deleted file mode 100644
index 008d7be444..0000000000
--- a/changelog.d/10572.misc
+++ /dev/null
@@ -1 +0,0 @@
-Clarify error message when failing to join a restricted room.
diff --git a/changelog.d/10573.misc b/changelog.d/10573.misc
deleted file mode 100644
index fc9b1a2f70..0000000000
--- a/changelog.d/10573.misc
+++ /dev/null
@@ -1 +0,0 @@
-Remove references to BuildKite in favour of GitHub Actions.
\ No newline at end of file
diff --git a/changelog.d/10574.feature b/changelog.d/10574.feature
deleted file mode 100644
index ffc4e4289c..0000000000
--- a/changelog.d/10574.feature
+++ /dev/null
@@ -1 +0,0 @@
-Add pagination to the spaces summary based on updates to [MSC2946](https://github.com/matrix-org/matrix-doc/pull/2946).
diff --git a/changelog.d/10575.feature b/changelog.d/10575.feature
deleted file mode 100644
index ffc4e4289c..0000000000
--- a/changelog.d/10575.feature
+++ /dev/null
@@ -1 +0,0 @@
-Add pagination to the spaces summary based on updates to [MSC2946](https://github.com/matrix-org/matrix-doc/pull/2946).
diff --git a/changelog.d/10576.misc b/changelog.d/10576.misc
deleted file mode 100644
index f9f9c9a6fd..0000000000
--- a/changelog.d/10576.misc
+++ /dev/null
@@ -1 +0,0 @@
-Move `/batch_send` endpoint defined by [MSC2716](https://github.com/matrix-org/matrix-doc/pull/2716) to the `/v2_alpha` directory.
diff --git a/changelog.d/10578.feature b/changelog.d/10578.feature
deleted file mode 100644
index 02397f0009..0000000000
--- a/changelog.d/10578.feature
+++ /dev/null
@@ -1 +0,0 @@
-Add an admin API (`GET /_synapse/admin/username_available`) to check if a username is available (regardless of registration settings).
\ No newline at end of file
diff --git a/changelog.d/10579.feature b/changelog.d/10579.feature
deleted file mode 100644
index ffc4e4289c..0000000000
--- a/changelog.d/10579.feature
+++ /dev/null
@@ -1 +0,0 @@
-Add pagination to the spaces summary based on updates to [MSC2946](https://github.com/matrix-org/matrix-doc/pull/2946).
diff --git a/changelog.d/10580.bugfix b/changelog.d/10580.bugfix
deleted file mode 100644
index f8da7382b7..0000000000
--- a/changelog.d/10580.bugfix
+++ /dev/null
@@ -1 +0,0 @@
-Allow public rooms to be previewed in the spaces summary APIs from [MSC2946](https://github.com/matrix-org/matrix-doc/pull/2946).
diff --git a/changelog.d/10587.misc b/changelog.d/10587.misc
deleted file mode 100644
index 4c6167977c..0000000000
--- a/changelog.d/10587.misc
+++ /dev/null
@@ -1 +0,0 @@
-Allow multiple custom directories in `read_templates`.
diff --git a/changelog.d/10588.removal b/changelog.d/10588.removal
deleted file mode 100644
index 90c4b5cee2..0000000000
--- a/changelog.d/10588.removal
+++ /dev/null
@@ -1 +0,0 @@
-No longer build `.dev` packages for Ubuntu 20.10 LTS Groovy Gorilla, which has now EOLed.
\ No newline at end of file
diff --git a/changelog.d/10590.misc b/changelog.d/10590.misc
deleted file mode 100644
index 62fec717da..0000000000
--- a/changelog.d/10590.misc
+++ /dev/null
@@ -1 +0,0 @@
-Re-organize the `synapse.federation.transport.server` module to create smaller files.
diff --git a/changelog.d/10591.misc b/changelog.d/10591.misc
deleted file mode 100644
index 9a765435db..0000000000
--- a/changelog.d/10591.misc
+++ /dev/null
@@ -1 +0,0 @@
-Clean up some of the federation event authentication code for clarity.
diff --git a/changelog.d/10592.bugfix b/changelog.d/10592.bugfix
deleted file mode 100644
index efcdab1136..0000000000
--- a/changelog.d/10592.bugfix
+++ /dev/null
@@ -1 +0,0 @@
-Fix a bug introduced in v1.37.1 where an error could occur in the asyncronous processing of PDUs when the queue was empty.
diff --git a/changelog.d/10598.feature b/changelog.d/10598.feature
deleted file mode 100644
index 92c159118b..0000000000
--- a/changelog.d/10598.feature
+++ /dev/null
@@ -1 +0,0 @@
-Allow editing a user's `external_ids` via the "Edit User" admin API. Contributed by @dklimpel.
\ No newline at end of file
diff --git a/changelog.d/10599.doc b/changelog.d/10599.doc
deleted file mode 100644
index 66e72078f0..0000000000
--- a/changelog.d/10599.doc
+++ /dev/null
@@ -1 +0,0 @@
-Update CONTRIBUTING.md to fix index links and the instructions for SyTest in docker.
diff --git a/changelog.d/10600.misc b/changelog.d/10600.misc
deleted file mode 100644
index 489dc20b11..0000000000
--- a/changelog.d/10600.misc
+++ /dev/null
@@ -1 +0,0 @@
-Flatten the `synapse.rest.client` package by moving the contents of `v1` and `v2_alpha` into the parent.
diff --git a/changelog.d/10602.feature b/changelog.d/10602.feature
deleted file mode 100644
index ab18291a20..0000000000
--- a/changelog.d/10602.feature
+++ /dev/null
@@ -1 +0,0 @@
-The Synapse manhole no longer needs coroutines to be wrapped in `defer.ensureDeferred`.
diff --git a/changelog.d/10606.bugfix b/changelog.d/10606.bugfix
deleted file mode 100644
index bab9fd2a61..0000000000
--- a/changelog.d/10606.bugfix
+++ /dev/null
@@ -1 +0,0 @@
-Fix errors on /sync when read receipt data is a string. Only affects homeservers with the experimental flag for [MSC2285](https://github.com/matrix-org/matrix-doc/pull/2285) enabled. Contributed by @SimonBrandner.
diff --git a/changelog.d/10611.bugfix b/changelog.d/10611.bugfix
deleted file mode 100644
index ecbe408b47..0000000000
--- a/changelog.d/10611.bugfix
+++ /dev/null
@@ -1 +0,0 @@
-Additional validation for the spaces summary API to avoid errors like `ValueError: Stop argument for islice() must be None or an integer`. The missing validation has existed since v1.31.0.
diff --git a/changelog.d/10620.misc b/changelog.d/10620.misc
deleted file mode 100644
index 8b29668a1f..0000000000
--- a/changelog.d/10620.misc
+++ /dev/null
@@ -1 +0,0 @@
-Fix up a couple of links to the database schema documentation.
diff --git a/changelog.d/10623.bugfix b/changelog.d/10623.bugfix
deleted file mode 100644
index 759fba3513..0000000000
--- a/changelog.d/10623.bugfix
+++ /dev/null
@@ -1 +0,0 @@
-Revert behaviour introduced in v1.38.0 that strips `org.matrix.msc2732.device_unused_fallback_key_types` from `/sync` when its value is empty. This field should instead always be present according to [MSC2732](https://github.com/matrix-org/matrix-doc/blob/master/proposals/2732-olm-fallback-keys.md).
\ No newline at end of file
diff --git a/changelog.d/9581.feature b/changelog.d/9581.feature
deleted file mode 100644
index fa1949cd4b..0000000000
--- a/changelog.d/9581.feature
+++ /dev/null
@@ -1 +0,0 @@
-Add `get_userinfo_by_id` method to ModuleApi.
diff --git a/debian/changelog b/debian/changelog
index e101423fe4..68f309b0b2 100644
--- a/debian/changelog
+++ b/debian/changelog
@@ -1,3 +1,9 @@
+matrix-synapse-py3 (1.41.0~rc1) stable; urgency=medium
+
+ * New synapse release 1.41.0~rc1.
+
+ -- Synapse Packaging team <packages@matrix.org> Wed, 18 Aug 2021 15:52:00 +0100
+
matrix-synapse-py3 (1.40.0) stable; urgency=medium
* New synapse release 1.40.0.
diff --git a/docker/conf/log.config b/docker/conf/log.config
index a994626926..7a216a36a0 100644
--- a/docker/conf/log.config
+++ b/docker/conf/log.config
@@ -18,18 +18,31 @@ handlers:
backupCount: 6 # Does not include the current log file.
encoding: utf8
- # Default to buffering writes to log file for efficiency. This means that
- # there will be a delay for INFO/DEBUG logs to get written, but WARNING/ERROR
- # logs will still be flushed immediately.
+ # Default to buffering writes to log file for efficiency.
+ # WARNING/ERROR logs will still be flushed immediately, but there will be a
+ # delay (of up to `period` seconds, or until the buffer is full with
+ # `capacity` messages) before INFO/DEBUG logs get written.
buffer:
- class: logging.handlers.MemoryHandler
+ class: synapse.logging.handlers.PeriodicallyFlushingMemoryHandler
target: file
- # The capacity is the number of log lines that are buffered before
- # being written to disk. Increasing this will lead to better
+
+ # The capacity is the maximum number of log lines that are buffered
+ # before being written to disk. Increasing this will lead to better
# performance, at the expensive of it taking longer for log lines to
# be written to disk.
+ # This parameter is required.
capacity: 10
- flushLevel: 30 # Flush for WARNING logs as well
+
+ # Logs with a level at or above the flush level will cause the buffer to
+ # be flushed immediately.
+ # Default value: 40 (ERROR)
+ # Other values: 50 (CRITICAL), 30 (WARNING), 20 (INFO), 10 (DEBUG)
+ flushLevel: 30 # Flush immediately for WARNING logs and higher
+
+ # The period of time, in seconds, between forced flushes.
+ # Messages will not be delayed for longer than this time.
+ # Default value: 5 seconds
+ period: 5
{% endif %}
console:
diff --git a/docs/SUMMARY.md b/docs/SUMMARY.md
index 0714f151fd..634bc833ab 100644
--- a/docs/SUMMARY.md
+++ b/docs/SUMMARY.md
@@ -21,6 +21,7 @@
- [Homeserver Sample Config File](usage/configuration/homeserver_sample_config.md)
- [Logging Sample Config File](usage/configuration/logging_sample_config.md)
- [Structured Logging](structured_logging.md)
+ - [Templates](templates.md)
- [User Authentication](usage/configuration/user_authentication/README.md)
- [Single-Sign On]()
- [OpenID Connect](openid.md)
diff --git a/docs/sample_config.yaml b/docs/sample_config.yaml
index 6030e85a08..2b0c453242 100644
--- a/docs/sample_config.yaml
+++ b/docs/sample_config.yaml
@@ -551,6 +551,19 @@ retention:
#
#next_link_domain_whitelist: ["matrix.org"]
+# Templates to use when generating email or HTML page contents.
+#
+templates:
+ # Directory in which Synapse will try to find template files to use to generate
+ # email or HTML page contents.
+ # If not set, or a file is not found within the template directory, a default
+ # template from within the Synapse package will be used.
+ #
+ # See https://matrix-org.github.io/synapse/latest/templates.html for more
+ # information about using custom templates.
+ #
+ #custom_template_directory: /path/to/custom/templates/
+
## TLS ##
@@ -1881,6 +1894,9 @@ cas_config:
# Additional settings to use with single-sign on systems such as OpenID Connect,
# SAML2 and CAS.
#
+# Server admins can configure custom templates for pages related to SSO. See
+# https://matrix-org.github.io/synapse/latest/templates.html for more information.
+#
sso:
# A list of client URLs which are whitelisted so that the user does not
# have to confirm giving access to their account to the URL. Any client
@@ -1913,169 +1929,6 @@ sso:
#
#update_profile_information: true
- # Directory in which Synapse will try to find the template files below.
- # If not set, or the files named below are not found within the template
- # directory, default templates from within the Synapse package will be used.
- #
- # Synapse will look for the following templates in this directory:
- #
- # * HTML page to prompt the user to choose an Identity Provider during
- # login: 'sso_login_idp_picker.html'.
- #
- # This is only used if multiple SSO Identity Providers are configured.
- #
- # When rendering, this template is given the following variables:
- # * redirect_url: the URL that the user will be redirected to after
- # login.
- #
- # * server_name: the homeserver's name.
- #
- # * providers: a list of available Identity Providers. Each element is
- # an object with the following attributes:
- #
- # * idp_id: unique identifier for the IdP
- # * idp_name: user-facing name for the IdP
- # * idp_icon: if specified in the IdP config, an MXC URI for an icon
- # for the IdP
- # * idp_brand: if specified in the IdP config, a textual identifier
- # for the brand of the IdP
- #
- # The rendered HTML page should contain a form which submits its results
- # back as a GET request, with the following query parameters:
- #
- # * redirectUrl: the client redirect URI (ie, the `redirect_url` passed
- # to the template)
- #
- # * idp: the 'idp_id' of the chosen IDP.
- #
- # * HTML page to prompt new users to enter a userid and confirm other
- # details: 'sso_auth_account_details.html'. This is only shown if the
- # SSO implementation (with any user_mapping_provider) does not return
- # a localpart.
- #
- # When rendering, this template is given the following variables:
- #
- # * server_name: the homeserver's name.
- #
- # * idp: details of the SSO Identity Provider that the user logged in
- # with: an object with the following attributes:
- #
- # * idp_id: unique identifier for the IdP
- # * idp_name: user-facing name for the IdP
- # * idp_icon: if specified in the IdP config, an MXC URI for an icon
- # for the IdP
- # * idp_brand: if specified in the IdP config, a textual identifier
- # for the brand of the IdP
- #
- # * user_attributes: an object containing details about the user that
- # we received from the IdP. May have the following attributes:
- #
- # * display_name: the user's display_name
- # * emails: a list of email addresses
- #
- # The template should render a form which submits the following fields:
- #
- # * username: the localpart of the user's chosen user id
- #
- # * HTML page allowing the user to consent to the server's terms and
- # conditions. This is only shown for new users, and only if
- # `user_consent.require_at_registration` is set.
- #
- # When rendering, this template is given the following variables:
- #
- # * server_name: the homeserver's name.
- #
- # * user_id: the user's matrix proposed ID.
- #
- # * user_profile.display_name: the user's proposed display name, if any.
- #
- # * consent_version: the version of the terms that the user will be
- # shown
- #
- # * terms_url: a link to the page showing the terms.
- #
- # The template should render a form which submits the following fields:
- #
- # * accepted_version: the version of the terms accepted by the user
- # (ie, 'consent_version' from the input variables).
- #
- # * HTML page for a confirmation step before redirecting back to the client
- # with the login token: 'sso_redirect_confirm.html'.
- #
- # When rendering, this template is given the following variables:
- #
- # * redirect_url: the URL the user is about to be redirected to.
- #
- # * display_url: the same as `redirect_url`, but with the query
- # parameters stripped. The intention is to have a
- # human-readable URL to show to users, not to use it as
- # the final address to redirect to.
- #
- # * server_name: the homeserver's name.
- #
- # * new_user: a boolean indicating whether this is the user's first time
- # logging in.
- #
- # * user_id: the user's matrix ID.
- #
- # * user_profile.avatar_url: an MXC URI for the user's avatar, if any.
- # None if the user has not set an avatar.
- #
- # * user_profile.display_name: the user's display name. None if the user
- # has not set a display name.
- #
- # * HTML page which notifies the user that they are authenticating to confirm
- # an operation on their account during the user interactive authentication
- # process: 'sso_auth_confirm.html'.
- #
- # When rendering, this template is given the following variables:
- # * redirect_url: the URL the user is about to be redirected to.
- #
- # * description: the operation which the user is being asked to confirm
- #
- # * idp: details of the Identity Provider that we will use to confirm
- # the user's identity: an object with the following attributes:
- #
- # * idp_id: unique identifier for the IdP
- # * idp_name: user-facing name for the IdP
- # * idp_icon: if specified in the IdP config, an MXC URI for an icon
- # for the IdP
- # * idp_brand: if specified in the IdP config, a textual identifier
- # for the brand of the IdP
- #
- # * HTML page shown after a successful user interactive authentication session:
- # 'sso_auth_success.html'.
- #
- # Note that this page must include the JavaScript which notifies of a successful authentication
- # (see https://matrix.org/docs/spec/client_server/r0.6.0#fallback).
- #
- # This template has no additional variables.
- #
- # * HTML page shown after a user-interactive authentication session which
- # does not map correctly onto the expected user: 'sso_auth_bad_user.html'.
- #
- # When rendering, this template is given the following variables:
- # * server_name: the homeserver's name.
- # * user_id_to_verify: the MXID of the user that we are trying to
- # validate.
- #
- # * HTML page shown during single sign-on if a deactivated user (according to Synapse's database)
- # attempts to login: 'sso_account_deactivated.html'.
- #
- # This template has no additional variables.
- #
- # * HTML page to display to users if something goes wrong during the
- # OpenID Connect authentication process: 'sso_error.html'.
- #
- # When rendering, this template is given two variables:
- # * error: the technical name of the error
- # * error_description: a human-readable message for the error
- #
- # You can see the default templates at:
- # https://github.com/matrix-org/synapse/tree/master/synapse/res/templates
- #
- #template_dir: "res/templates"
-
# JSON web token integration. The following settings can be used to make
# Synapse JSON web tokens for authentication, instead of its internal
@@ -2206,6 +2059,9 @@ ui_auth:
# Configuration for sending emails from Synapse.
#
+# Server admins can configure custom templates for email content. See
+# https://matrix-org.github.io/synapse/latest/templates.html for more information.
+#
email:
# The hostname of the outgoing SMTP server to use. Defaults to 'localhost'.
#
@@ -2282,49 +2138,6 @@ email:
#
#invite_client_location: https://app.element.io
- # Directory in which Synapse will try to find the template files below.
- # If not set, or the files named below are not found within the template
- # directory, default templates from within the Synapse package will be used.
- #
- # Synapse will look for the following templates in this directory:
- #
- # * The contents of email notifications of missed events: 'notif_mail.html' and
- # 'notif_mail.txt'.
- #
- # * The contents of account expiry notice emails: 'notice_expiry.html' and
- # 'notice_expiry.txt'.
- #
- # * The contents of password reset emails sent by the homeserver:
- # 'password_reset.html' and 'password_reset.txt'
- #
- # * An HTML page that a user will see when they follow the link in the password
- # reset email. The user will be asked to confirm the action before their
- # password is reset: 'password_reset_confirmation.html'
- #
- # * HTML pages for success and failure that a user will see when they confirm
- # the password reset flow using the page above: 'password_reset_success.html'
- # and 'password_reset_failure.html'
- #
- # * The contents of address verification emails sent during registration:
- # 'registration.html' and 'registration.txt'
- #
- # * HTML pages for success and failure that a user will see when they follow
- # the link in an address verification email sent during registration:
- # 'registration_success.html' and 'registration_failure.html'
- #
- # * The contents of address verification emails sent when an address is added
- # to a Matrix account: 'add_threepid.html' and 'add_threepid.txt'
- #
- # * HTML pages for success and failure that a user will see when they follow
- # the link in an address verification email sent when an address is added
- # to a Matrix account: 'add_threepid_success.html' and
- # 'add_threepid_failure.html'
- #
- # You can see the default templates at:
- # https://github.com/matrix-org/synapse/tree/master/synapse/res/templates
- #
- #template_dir: "res/templates"
-
# Subjects to use when sending emails from Synapse.
#
# The placeholder '%(app)s' will be replaced with the value of the 'app_name'
diff --git a/docs/sample_log_config.yaml b/docs/sample_log_config.yaml
index 669e600081..2485ad25ed 100644
--- a/docs/sample_log_config.yaml
+++ b/docs/sample_log_config.yaml
@@ -24,18 +24,31 @@ handlers:
backupCount: 3 # Does not include the current log file.
encoding: utf8
- # Default to buffering writes to log file for efficiency. This means that
- # will be a delay for INFO/DEBUG logs to get written, but WARNING/ERROR
- # logs will still be flushed immediately.
+ # Default to buffering writes to log file for efficiency.
+ # WARNING/ERROR logs will still be flushed immediately, but there will be a
+ # delay (of up to `period` seconds, or until the buffer is full with
+ # `capacity` messages) before INFO/DEBUG logs get written.
buffer:
- class: logging.handlers.MemoryHandler
+ class: synapse.logging.handlers.PeriodicallyFlushingMemoryHandler
target: file
- # The capacity is the number of log lines that are buffered before
- # being written to disk. Increasing this will lead to better
+
+ # The capacity is the maximum number of log lines that are buffered
+ # before being written to disk. Increasing this will lead to better
# performance, at the expensive of it taking longer for log lines to
# be written to disk.
+ # This parameter is required.
capacity: 10
- flushLevel: 30 # Flush for WARNING logs as well
+
+ # Logs with a level at or above the flush level will cause the buffer to
+ # be flushed immediately.
+ # Default value: 40 (ERROR)
+ # Other values: 50 (CRITICAL), 30 (WARNING), 20 (INFO), 10 (DEBUG)
+ flushLevel: 30 # Flush immediately for WARNING logs and higher
+
+ # The period of time, in seconds, between forced flushes.
+ # Messages will not be delayed for longer than this time.
+ # Default value: 5 seconds
+ period: 5
# A handler that writes logs to stderr. Unused by default, but can be used
# instead of "buffer" and "file" in the logger handlers.
diff --git a/docs/templates.md b/docs/templates.md
new file mode 100644
index 0000000000..a240f58b54
--- /dev/null
+++ b/docs/templates.md
@@ -0,0 +1,239 @@
+# Templates
+
+Synapse uses parametrised templates to generate the content of emails it sends and
+webpages it shows to users.
+
+By default, Synapse will use the templates listed [here](https://github.com/matrix-org/synapse/tree/master/synapse/res/templates).
+Server admins can configure an additional directory for Synapse to look for templates
+in, allowing them to specify custom templates:
+
+```yaml
+templates:
+ custom_templates_directory: /path/to/custom/templates/
+```
+
+If this setting is not set, or the files named below are not found within the directory,
+default templates from within the Synapse package will be used.
+
+Templates that are given variables when being rendered are rendered using [Jinja 2](https://jinja.palletsprojects.com/en/2.11.x/).
+Templates rendered by Jinja 2 can also access two functions on top of the functions
+already available as part of Jinja 2:
+
+```python
+format_ts(value: int, format: str) -> str
+```
+
+Formats a timestamp in milliseconds.
+
+Example: `reason.last_sent_ts|format_ts("%c")`
+
+```python
+mxc_to_http(value: str, width: int, height: int, resize_method: str = "crop") -> str
+```
+
+Turns a `mxc://` URL for media content into an HTTP(S) one using the homeserver's
+`public_baseurl` configuration setting as the URL's base.
+
+Example: `message.sender_avatar_url|mxc_to_http(32,32)`
+
+
+## Email templates
+
+Below are the templates Synapse will look for when generating the content of an email:
+
+* `notif_mail.html` and `notif_mail.txt`: The contents of email notifications of missed
+ events.
+ When rendering, this template is given the following variables:
+ * `user_display_name`: the display name for the user receiving the notification
+ * `unsubscribe_link`: the link users can click to unsubscribe from email notifications
+ * `summary_text`: a summary of the notification(s). The text used can be customised
+ by configuring the various settings in the `email.subjects` section of the
+ configuration file.
+ * `rooms`: a list of rooms containing events to include in the email. Each element is
+ an object with the following attributes:
+ * `title`: a human-readable name for the room
+ * `hash`: a hash of the ID of the room
+ * `invite`: a boolean, which is `True` if the room is an invite the user hasn't
+ accepted yet, `False` otherwise
+ * `notifs`: a list of events, or an empty list if `invite` is `True`. Each element
+ is an object with the following attributes:
+ * `link`: a `matrix.to` link to the event
+ * `ts`: the time in milliseconds at which the event was received
+ * `messages`: a list of messages containing one message before the event, the
+ message in the event, and one message after the event. Each element is an
+ object with the following attributes:
+ * `event_type`: the type of the event
+ * `is_historical`: a boolean, which is `False` if the message is the one
+ that triggered the notification, `True` otherwise
+ * `id`: the ID of the event
+ * `ts`: the time in milliseconds at which the event was sent
+ * `sender_name`: the display name for the event's sender
+ * `sender_avatar_url`: the avatar URL (as a `mxc://` URL) for the event's
+ sender
+ * `sender_hash`: a hash of the user ID of the sender
+ * `link`: a `matrix.to` link to the room
+ * `reason`: information on the event that triggered the email to be sent. It's an
+ object with the following attributes:
+ * `room_id`: the ID of the room the event was sent in
+ * `room_name`: a human-readable name for the room the event was sent in
+ * `now`: the current time in milliseconds
+ * `received_at`: the time in milliseconds at which the event was received
+ * `delay_before_mail_ms`: the amount of time in milliseconds Synapse always waits
+ before ever emailing about a notification (to give the user a chance to respond
+ to other push or notice the window)
+ * `last_sent_ts`: the time in milliseconds at which a notification was last sent
+ for an event in this room
+ * `throttle_ms`: the minimum amount of time in milliseconds between two
+ notifications can be sent for this room
+* `password_reset.html` and `password_reset.txt`: The contents of password reset emails
+ sent by the homeserver.
+ When rendering, these templates are given a `link` variable which contains the link the
+ user must click in order to reset their password.
+* `registration.html` and `registration.txt`: The contents of address verification emails
+ sent during registration.
+ When rendering, these templates are given a `link` variable which contains the link the
+ user must click in order to validate their email address.
+* `add_threepid.html` and `add_threepid.txt`: The contents of address verification emails
+ sent when an address is added to a Matrix account.
+ When rendering, these templates are given a `link` variable which contains the link the
+ user must click in order to validate their email address.
+
+
+## HTML page templates for registration and password reset
+
+Below are the templates Synapse will look for when generating pages related to
+registration and password reset:
+
+* `password_reset_confirmation.html`: An HTML page that a user will see when they follow
+ the link in the password reset email. The user will be asked to confirm the action
+ before their password is reset.
+ When rendering, this template is given the following variables:
+ * `sid`: the session ID for the password reset
+ * `token`: the token for the password reset
+ * `client_secret`: the client secret for the password reset
+* `password_reset_success.html` and `password_reset_failure.html`: HTML pages for success
+ and failure that a user will see when they confirm the password reset flow using the
+ page above.
+ When rendering, `password_reset_success.html` is given no variable, and
+ `password_reset_failure.html` is given a `failure_reason`, which contains the reason
+ for the password reset failure.
+* `registration_success.html` and `registration_failure.html`: HTML pages for success and
+ failure that a user will see when they follow the link in an address verification email
+ sent during registration.
+ When rendering, `registration_success.html` is given no variable, and
+ `registration_failure.html` is given a `failure_reason`, which contains the reason
+ for the registration failure.
+* `add_threepid_success.html` and `add_threepid_failure.html`: HTML pages for success and
+ failure that a user will see when they follow the link in an address verification email
+ sent when an address is added to a Matrix account.
+ When rendering, `add_threepid_success.html` is given no variable, and
+ `add_threepid_failure.html` is given a `failure_reason`, which contains the reason
+ for the registration failure.
+
+
+## HTML page templates for Single Sign-On (SSO)
+
+Below are the templates Synapse will look for when generating pages related to SSO:
+
+* `sso_login_idp_picker.html`: HTML page to prompt the user to choose an
+ Identity Provider during login.
+ This is only used if multiple SSO Identity Providers are configured.
+ When rendering, this template is given the following variables:
+ * `redirect_url`: the URL that the user will be redirected to after
+ login.
+ * `server_name`: the homeserver's name.
+ * `providers`: a list of available Identity Providers. Each element is
+ an object with the following attributes:
+ * `idp_id`: unique identifier for the IdP
+ * `idp_name`: user-facing name for the IdP
+ * `idp_icon`: if specified in the IdP config, an MXC URI for an icon
+ for the IdP
+ * `idp_brand`: if specified in the IdP config, a textual identifier
+ for the brand of the IdP
+ The rendered HTML page should contain a form which submits its results
+ back as a GET request, with the following query parameters:
+ * `redirectUrl`: the client redirect URI (ie, the `redirect_url` passed
+ to the template)
+ * `idp`: the 'idp_id' of the chosen IDP.
+* `sso_auth_account_details.html`: HTML page to prompt new users to enter a
+ userid and confirm other details. This is only shown if the
+ SSO implementation (with any `user_mapping_provider`) does not return
+ a localpart.
+ When rendering, this template is given the following variables:
+ * `server_name`: the homeserver's name.
+ * `idp`: details of the SSO Identity Provider that the user logged in
+ with: an object with the following attributes:
+ * `idp_id`: unique identifier for the IdP
+ * `idp_name`: user-facing name for the IdP
+ * `idp_icon`: if specified in the IdP config, an MXC URI for an icon
+ for the IdP
+ * `idp_brand`: if specified in the IdP config, a textual identifier
+ for the brand of the IdP
+ * `user_attributes`: an object containing details about the user that
+ we received from the IdP. May have the following attributes:
+ * display_name: the user's display_name
+ * emails: a list of email addresses
+ The template should render a form which submits the following fields:
+ * `username`: the localpart of the user's chosen user id
+* `sso_new_user_consent.html`: HTML page allowing the user to consent to the
+ server's terms and conditions. This is only shown for new users, and only if
+ `user_consent.require_at_registration` is set.
+ When rendering, this template is given the following variables:
+ * `server_name`: the homeserver's name.
+ * `user_id`: the user's matrix proposed ID.
+ * `user_profile.display_name`: the user's proposed display name, if any.
+ * consent_version: the version of the terms that the user will be
+ shown
+ * `terms_url`: a link to the page showing the terms.
+ The template should render a form which submits the following fields:
+ * `accepted_version`: the version of the terms accepted by the user
+ (ie, 'consent_version' from the input variables).
+* `sso_redirect_confirm.html`: HTML page for a confirmation step before redirecting back
+ to the client with the login token.
+ When rendering, this template is given the following variables:
+ * `redirect_url`: the URL the user is about to be redirected to.
+ * `display_url`: the same as `redirect_url`, but with the query
+ parameters stripped. The intention is to have a
+ human-readable URL to show to users, not to use it as
+ the final address to redirect to.
+ * `server_name`: the homeserver's name.
+ * `new_user`: a boolean indicating whether this is the user's first time
+ logging in.
+ * `user_id`: the user's matrix ID.
+ * `user_profile.avatar_url`: an MXC URI for the user's avatar, if any.
+ `None` if the user has not set an avatar.
+ * `user_profile.display_name`: the user's display name. `None` if the user
+ has not set a display name.
+* `sso_auth_confirm.html`: HTML page which notifies the user that they are authenticating
+ to confirm an operation on their account during the user interactive authentication
+ process.
+ When rendering, this template is given the following variables:
+ * `redirect_url`: the URL the user is about to be redirected to.
+ * `description`: the operation which the user is being asked to confirm
+ * `idp`: details of the Identity Provider that we will use to confirm
+ the user's identity: an object with the following attributes:
+ * `idp_id`: unique identifier for the IdP
+ * `idp_name`: user-facing name for the IdP
+ * `idp_icon`: if specified in the IdP config, an MXC URI for an icon
+ for the IdP
+ * `idp_brand`: if specified in the IdP config, a textual identifier
+ for the brand of the IdP
+* `sso_auth_success.html`: HTML page shown after a successful user interactive
+ authentication session.
+ Note that this page must include the JavaScript which notifies of a successful
+ authentication (see https://matrix.org/docs/spec/client_server/r0.6.0#fallback).
+ This template has no additional variables.
+* `sso_auth_bad_user.html`: HTML page shown after a user-interactive authentication
+ session which does not map correctly onto the expected user.
+ When rendering, this template is given the following variables:
+ * `server_name`: the homeserver's name.
+ * `user_id_to_verify`: the MXID of the user that we are trying to
+ validate.
+* `sso_account_deactivated.html`: HTML page shown during single sign-on if a deactivated
+ user (according to Synapse's database) attempts to login.
+ This template has no additional variables.
+* `sso_error.html`: HTML page to display to users if something goes wrong during the
+ OpenID Connect authentication process.
+ When rendering, this template is given two variables:
+ * `error`: the technical name of the error
+ * `error_description`: a human-readable message for the error
diff --git a/docs/upgrade.md b/docs/upgrade.md
index adbd16fda5..6d4b8cb48e 100644
--- a/docs/upgrade.md
+++ b/docs/upgrade.md
@@ -95,11 +95,20 @@ The following admin APIs were deprecated in [Synapse 1.25](https://github.com/ma
- `POST /_synapse/admin/v1/purge_room`
- `POST /_synapse/admin/v1/shutdown_room/<room_id>`
-Any scripts still using the above APIs should be converted to use the
+Any scripts still using the above APIs should be converted to use the
[Delete Room API](https://matrix-org.github.io/synapse/latest/admin_api/rooms.html#delete-room-api).
+## User-interactive authentication fallback templates can now display errors
+
+This may affect you if you make use of custom HTML templates for the
+[reCAPTCHA](../synapse/res/templates/recaptcha.html) or
+[terms](../synapse/res/templates/terms.html) fallback pages.
+
+The template is now provided an `error` variable if the authentication
+process failed. See the default templates linked above for an example.
-# Upgrading to v1.xx.0
+
+# Upgrading to v1.41.0
## Add support for routing outbound HTTP requests via a proxy for federation
@@ -125,14 +134,23 @@ environment variable.
See [using a forward proxy with Synapse documentation](setup/forward_proxy.md) for
details.
-## User-interactive authentication fallback templates can now display errors
+## Deprecation of `template_dir`
-This may affect you if you make use of custom HTML templates for the
-[reCAPTCHA](../synapse/res/templates/recaptcha.html) or
-[terms](../synapse/res/templates/terms.html) fallback pages.
+The `template_dir` settings in the `sso`, `account_validity` and `email` sections of the
+configuration file are now deprecated. Server admins should use the new
+`templates.custom_template_directory` setting in the configuration file and use one single
+custom template directory for all aforementioned features. Template file names remain
+unchanged. See [the related documentation](https://matrix-org.github.io/synapse/latest/templates.html)
+for more information and examples.
-The template is now provided an `error` variable if the authentication
-process failed. See the default templates linked above for an example.
+We plan to remove support for these settings in October 2021.
+
+## `/_synapse/admin/v1/users/{userId}/media` must be handled by media workers
+
+The [media repository worker documentation](https://matrix-org.github.io/synapse/latest/workers.html#synapseappmedia_repository)
+has been updated to reflect that calls to `/_synapse/admin/v1/users/{userId}/media`
+must now be handled by media repository workers. This is due to the new `DELETE` method
+of this endpoint modifying the media store.
# Upgrading to v1.39.0
diff --git a/docs/workers.md b/docs/workers.md
index d8672324c3..2e63f03452 100644
--- a/docs/workers.md
+++ b/docs/workers.md
@@ -214,6 +214,7 @@ expressions:
^/_matrix/federation/v1/send/
# Client API requests
+ ^/_matrix/client/(api/v1|r0|unstable)/createRoom$
^/_matrix/client/(api/v1|r0|unstable)/publicRooms$
^/_matrix/client/(api/v1|r0|unstable)/rooms/.*/joined_members$
^/_matrix/client/(api/v1|r0|unstable)/rooms/.*/context/.*$
@@ -425,10 +426,12 @@ Handles the media repository. It can handle all endpoints starting with:
^/_synapse/admin/v1/user/.*/media.*$
^/_synapse/admin/v1/media/.*$
^/_synapse/admin/v1/quarantine_media/.*$
+ ^/_synapse/admin/v1/users/.*/media$
You should also set `enable_media_repo: False` in the shared configuration
file to stop the main synapse running background jobs related to managing the
-media repository.
+media repository. Note that doing so will prevent the main process from being
+able to handle the above endpoints.
In the `media_repository` worker configuration file, configure the http listener to
expose the `media` resource. For example:
diff --git a/scripts-dev/build_debian_packages b/scripts-dev/build_debian_packages
index 6153cb225f..e9f89e38ef 100755
--- a/scripts-dev/build_debian_packages
+++ b/scripts-dev/build_debian_packages
@@ -20,8 +20,9 @@ from concurrent.futures import ThreadPoolExecutor
from typing import Optional, Sequence
DISTS = (
- "debian:buster",
+ "debian:buster", # oldstable: EOL 2022-08
"debian:bullseye",
+ "debian:bookworm",
"debian:sid",
"ubuntu:bionic", # 18.04 LTS (our EOL forced by Py36 on 2021-12-23)
"ubuntu:focal", # 20.04 LTS (our EOL forced by Py38 on 2024-10-14)
diff --git a/synapse/__init__.py b/synapse/__init__.py
index 919293cd80..6ada20a77f 100644
--- a/synapse/__init__.py
+++ b/synapse/__init__.py
@@ -47,7 +47,7 @@ try:
except ImportError:
pass
-__version__ = "1.40.0"
+__version__ = "1.41.0rc1"
if bool(os.environ.get("SYNAPSE_TEST_PATCH_LOG_CONTEXTS", False)):
# We import here so that we don't have to install a bunch of deps when
diff --git a/synapse/app/admin_cmd.py b/synapse/app/admin_cmd.py
index 3234d9ebba..7396db93c6 100644
--- a/synapse/app/admin_cmd.py
+++ b/synapse/app/admin_cmd.py
@@ -38,7 +38,6 @@ from synapse.replication.slave.storage.groups import SlavedGroupServerStore
from synapse.replication.slave.storage.push_rule import SlavedPushRuleStore
from synapse.replication.slave.storage.receipts import SlavedReceiptsStore
from synapse.replication.slave.storage.registration import SlavedRegistrationStore
-from synapse.replication.slave.storage.room import RoomStore
from synapse.server import HomeServer
from synapse.util.logcontext import LoggingContext
from synapse.util.versionstring import get_version_string
@@ -58,7 +57,6 @@ class AdminCmdSlavedStore(
SlavedPushRuleStore,
SlavedEventStore,
SlavedClientIpStore,
- RoomStore,
BaseSlavedStore,
):
pass
diff --git a/synapse/app/generic_worker.py b/synapse/app/generic_worker.py
index d7b425a7ab..845e6a8220 100644
--- a/synapse/app/generic_worker.py
+++ b/synapse/app/generic_worker.py
@@ -64,7 +64,6 @@ from synapse.replication.slave.storage.push_rule import SlavedPushRuleStore
from synapse.replication.slave.storage.pushers import SlavedPusherStore
from synapse.replication.slave.storage.receipts import SlavedReceiptsStore
from synapse.replication.slave.storage.registration import SlavedRegistrationStore
-from synapse.replication.slave.storage.room import RoomStore
from synapse.rest.admin import register_servlets_for_media_repo
from synapse.rest.client import (
account_data,
@@ -114,6 +113,7 @@ from synapse.storage.databases.main.monthly_active_users import (
MonthlyActiveUsersWorkerStore,
)
from synapse.storage.databases.main.presence import PresenceStore
+from synapse.storage.databases.main.room import RoomWorkerStore
from synapse.storage.databases.main.search import SearchStore
from synapse.storage.databases.main.stats import StatsStore
from synapse.storage.databases.main.transactions import TransactionWorkerStore
@@ -237,7 +237,7 @@ class GenericWorkerSlavedStore(
ClientIpWorkerStore,
SlavedEventStore,
SlavedKeyStore,
- RoomStore,
+ RoomWorkerStore,
DirectoryStore,
SlavedApplicationServiceStore,
SlavedRegistrationStore,
diff --git a/synapse/config/account_validity.py b/synapse/config/account_validity.py
index 9acce5996e..52e63ab1f6 100644
--- a/synapse/config/account_validity.py
+++ b/synapse/config/account_validity.py
@@ -78,6 +78,11 @@ class AccountValidityConfig(Config):
)
# Read and store template content
+ custom_template_directories = (
+ self.root.server.custom_template_directory,
+ account_validity_template_dir,
+ )
+
(
self.account_validity_account_renewed_template,
self.account_validity_account_previously_renewed_template,
@@ -88,5 +93,5 @@ class AccountValidityConfig(Config):
"account_previously_renewed.html",
invalid_token_template_filename,
],
- (td for td in (account_validity_template_dir,) if td),
+ (td for td in custom_template_directories if td),
)
diff --git a/synapse/config/emailconfig.py b/synapse/config/emailconfig.py
index fc74b4a8b9..4477419196 100644
--- a/synapse/config/emailconfig.py
+++ b/synapse/config/emailconfig.py
@@ -258,7 +258,12 @@ class EmailConfig(Config):
add_threepid_template_success_html,
],
(
- td for td in (template_dir,) if td
+ td
+ for td in (
+ self.root.server.custom_template_directory,
+ template_dir,
+ )
+ if td
), # Filter out template_dir if not provided
)
@@ -299,7 +304,14 @@ class EmailConfig(Config):
self.email_notif_template_text,
) = self.read_templates(
[notif_template_html, notif_template_text],
- (td for td in (template_dir,) if td),
+ (
+ td
+ for td in (
+ self.root.server.custom_template_directory,
+ template_dir,
+ )
+ if td
+ ), # Filter out template_dir if not provided
)
self.email_notif_for_new_users = email_config.get(
@@ -322,7 +334,14 @@ class EmailConfig(Config):
self.account_validity_template_text,
) = self.read_templates(
[expiry_template_html, expiry_template_text],
- (td for td in (template_dir,) if td),
+ (
+ td
+ for td in (
+ self.root.server.custom_template_directory,
+ template_dir,
+ )
+ if td
+ ), # Filter out template_dir if not provided
)
subjects_config = email_config.get("subjects", {})
@@ -354,6 +373,9 @@ class EmailConfig(Config):
"""\
# Configuration for sending emails from Synapse.
#
+ # Server admins can configure custom templates for email content. See
+ # https://matrix-org.github.io/synapse/latest/templates.html for more information.
+ #
email:
# The hostname of the outgoing SMTP server to use. Defaults to 'localhost'.
#
@@ -430,49 +452,6 @@ class EmailConfig(Config):
#
#invite_client_location: https://app.element.io
- # Directory in which Synapse will try to find the template files below.
- # If not set, or the files named below are not found within the template
- # directory, default templates from within the Synapse package will be used.
- #
- # Synapse will look for the following templates in this directory:
- #
- # * The contents of email notifications of missed events: 'notif_mail.html' and
- # 'notif_mail.txt'.
- #
- # * The contents of account expiry notice emails: 'notice_expiry.html' and
- # 'notice_expiry.txt'.
- #
- # * The contents of password reset emails sent by the homeserver:
- # 'password_reset.html' and 'password_reset.txt'
- #
- # * An HTML page that a user will see when they follow the link in the password
- # reset email. The user will be asked to confirm the action before their
- # password is reset: 'password_reset_confirmation.html'
- #
- # * HTML pages for success and failure that a user will see when they confirm
- # the password reset flow using the page above: 'password_reset_success.html'
- # and 'password_reset_failure.html'
- #
- # * The contents of address verification emails sent during registration:
- # 'registration.html' and 'registration.txt'
- #
- # * HTML pages for success and failure that a user will see when they follow
- # the link in an address verification email sent during registration:
- # 'registration_success.html' and 'registration_failure.html'
- #
- # * The contents of address verification emails sent when an address is added
- # to a Matrix account: 'add_threepid.html' and 'add_threepid.txt'
- #
- # * HTML pages for success and failure that a user will see when they follow
- # the link in an address verification email sent when an address is added
- # to a Matrix account: 'add_threepid_success.html' and
- # 'add_threepid_failure.html'
- #
- # You can see the default templates at:
- # https://github.com/matrix-org/synapse/tree/master/synapse/res/templates
- #
- #template_dir: "res/templates"
-
# Subjects to use when sending emails from Synapse.
#
# The placeholder '%%(app)s' will be replaced with the value of the 'app_name'
diff --git a/synapse/config/logger.py b/synapse/config/logger.py
index ad4e6e61c3..4a398a7932 100644
--- a/synapse/config/logger.py
+++ b/synapse/config/logger.py
@@ -67,18 +67,31 @@ handlers:
backupCount: 3 # Does not include the current log file.
encoding: utf8
- # Default to buffering writes to log file for efficiency. This means that
- # will be a delay for INFO/DEBUG logs to get written, but WARNING/ERROR
- # logs will still be flushed immediately.
+ # Default to buffering writes to log file for efficiency.
+ # WARNING/ERROR logs will still be flushed immediately, but there will be a
+ # delay (of up to `period` seconds, or until the buffer is full with
+ # `capacity` messages) before INFO/DEBUG logs get written.
buffer:
- class: logging.handlers.MemoryHandler
+ class: synapse.logging.handlers.PeriodicallyFlushingMemoryHandler
target: file
- # The capacity is the number of log lines that are buffered before
- # being written to disk. Increasing this will lead to better
+
+ # The capacity is the maximum number of log lines that are buffered
+ # before being written to disk. Increasing this will lead to better
# performance, at the expensive of it taking longer for log lines to
# be written to disk.
+ # This parameter is required.
capacity: 10
- flushLevel: 30 # Flush for WARNING logs as well
+
+ # Logs with a level at or above the flush level will cause the buffer to
+ # be flushed immediately.
+ # Default value: 40 (ERROR)
+ # Other values: 50 (CRITICAL), 30 (WARNING), 20 (INFO), 10 (DEBUG)
+ flushLevel: 30 # Flush immediately for WARNING logs and higher
+
+ # The period of time, in seconds, between forced flushes.
+ # Messages will not be delayed for longer than this time.
+ # Default value: 5 seconds
+ period: 5
# A handler that writes logs to stderr. Unused by default, but can be used
# instead of "buffer" and "file" in the logger handlers.
diff --git a/synapse/config/server.py b/synapse/config/server.py
index 871422ea28..d2c900f50c 100644
--- a/synapse/config/server.py
+++ b/synapse/config/server.py
@@ -711,6 +711,18 @@ class ServerConfig(Config):
# Turn the list into a set to improve lookup speed.
self.next_link_domain_whitelist = set(next_link_domain_whitelist)
+ templates_config = config.get("templates") or {}
+ if not isinstance(templates_config, dict):
+ raise ConfigError("The 'templates' section must be a dictionary")
+
+ self.custom_template_directory = templates_config.get(
+ "custom_template_directory"
+ )
+ if self.custom_template_directory is not None and not isinstance(
+ self.custom_template_directory, str
+ ):
+ raise ConfigError("'custom_template_directory' must be a string")
+
def has_tls_listener(self) -> bool:
return any(listener.tls for listener in self.listeners)
@@ -1271,6 +1283,19 @@ class ServerConfig(Config):
# all domains.
#
#next_link_domain_whitelist: ["matrix.org"]
+
+ # Templates to use when generating email or HTML page contents.
+ #
+ templates:
+ # Directory in which Synapse will try to find template files to use to generate
+ # email or HTML page contents.
+ # If not set, or a file is not found within the template directory, a default
+ # template from within the Synapse package will be used.
+ #
+ # See https://matrix-org.github.io/synapse/latest/templates.html for more
+ # information about using custom templates.
+ #
+ #custom_template_directory: /path/to/custom/templates/
"""
% locals()
)
diff --git a/synapse/config/sso.py b/synapse/config/sso.py
index 4b590e0535..fe1177ab81 100644
--- a/synapse/config/sso.py
+++ b/synapse/config/sso.py
@@ -45,6 +45,11 @@ class SSOConfig(Config):
self.sso_template_dir = sso_config.get("template_dir")
# Read templates from disk
+ custom_template_directories = (
+ self.root.server.custom_template_directory,
+ self.sso_template_dir,
+ )
+
(
self.sso_login_idp_picker_template,
self.sso_redirect_confirm_template,
@@ -63,7 +68,7 @@ class SSOConfig(Config):
"sso_auth_success.html",
"sso_auth_bad_user.html",
],
- (td for td in (self.sso_template_dir,) if td),
+ (td for td in custom_template_directories if td),
)
# These templates have no placeholders, so render them here
@@ -94,6 +99,9 @@ class SSOConfig(Config):
# Additional settings to use with single-sign on systems such as OpenID Connect,
# SAML2 and CAS.
#
+ # Server admins can configure custom templates for pages related to SSO. See
+ # https://matrix-org.github.io/synapse/latest/templates.html for more information.
+ #
sso:
# A list of client URLs which are whitelisted so that the user does not
# have to confirm giving access to their account to the URL. Any client
@@ -125,167 +133,4 @@ class SSOConfig(Config):
# information when first signing in. Defaults to false.
#
#update_profile_information: true
-
- # Directory in which Synapse will try to find the template files below.
- # If not set, or the files named below are not found within the template
- # directory, default templates from within the Synapse package will be used.
- #
- # Synapse will look for the following templates in this directory:
- #
- # * HTML page to prompt the user to choose an Identity Provider during
- # login: 'sso_login_idp_picker.html'.
- #
- # This is only used if multiple SSO Identity Providers are configured.
- #
- # When rendering, this template is given the following variables:
- # * redirect_url: the URL that the user will be redirected to after
- # login.
- #
- # * server_name: the homeserver's name.
- #
- # * providers: a list of available Identity Providers. Each element is
- # an object with the following attributes:
- #
- # * idp_id: unique identifier for the IdP
- # * idp_name: user-facing name for the IdP
- # * idp_icon: if specified in the IdP config, an MXC URI for an icon
- # for the IdP
- # * idp_brand: if specified in the IdP config, a textual identifier
- # for the brand of the IdP
- #
- # The rendered HTML page should contain a form which submits its results
- # back as a GET request, with the following query parameters:
- #
- # * redirectUrl: the client redirect URI (ie, the `redirect_url` passed
- # to the template)
- #
- # * idp: the 'idp_id' of the chosen IDP.
- #
- # * HTML page to prompt new users to enter a userid and confirm other
- # details: 'sso_auth_account_details.html'. This is only shown if the
- # SSO implementation (with any user_mapping_provider) does not return
- # a localpart.
- #
- # When rendering, this template is given the following variables:
- #
- # * server_name: the homeserver's name.
- #
- # * idp: details of the SSO Identity Provider that the user logged in
- # with: an object with the following attributes:
- #
- # * idp_id: unique identifier for the IdP
- # * idp_name: user-facing name for the IdP
- # * idp_icon: if specified in the IdP config, an MXC URI for an icon
- # for the IdP
- # * idp_brand: if specified in the IdP config, a textual identifier
- # for the brand of the IdP
- #
- # * user_attributes: an object containing details about the user that
- # we received from the IdP. May have the following attributes:
- #
- # * display_name: the user's display_name
- # * emails: a list of email addresses
- #
- # The template should render a form which submits the following fields:
- #
- # * username: the localpart of the user's chosen user id
- #
- # * HTML page allowing the user to consent to the server's terms and
- # conditions. This is only shown for new users, and only if
- # `user_consent.require_at_registration` is set.
- #
- # When rendering, this template is given the following variables:
- #
- # * server_name: the homeserver's name.
- #
- # * user_id: the user's matrix proposed ID.
- #
- # * user_profile.display_name: the user's proposed display name, if any.
- #
- # * consent_version: the version of the terms that the user will be
- # shown
- #
- # * terms_url: a link to the page showing the terms.
- #
- # The template should render a form which submits the following fields:
- #
- # * accepted_version: the version of the terms accepted by the user
- # (ie, 'consent_version' from the input variables).
- #
- # * HTML page for a confirmation step before redirecting back to the client
- # with the login token: 'sso_redirect_confirm.html'.
- #
- # When rendering, this template is given the following variables:
- #
- # * redirect_url: the URL the user is about to be redirected to.
- #
- # * display_url: the same as `redirect_url`, but with the query
- # parameters stripped. The intention is to have a
- # human-readable URL to show to users, not to use it as
- # the final address to redirect to.
- #
- # * server_name: the homeserver's name.
- #
- # * new_user: a boolean indicating whether this is the user's first time
- # logging in.
- #
- # * user_id: the user's matrix ID.
- #
- # * user_profile.avatar_url: an MXC URI for the user's avatar, if any.
- # None if the user has not set an avatar.
- #
- # * user_profile.display_name: the user's display name. None if the user
- # has not set a display name.
- #
- # * HTML page which notifies the user that they are authenticating to confirm
- # an operation on their account during the user interactive authentication
- # process: 'sso_auth_confirm.html'.
- #
- # When rendering, this template is given the following variables:
- # * redirect_url: the URL the user is about to be redirected to.
- #
- # * description: the operation which the user is being asked to confirm
- #
- # * idp: details of the Identity Provider that we will use to confirm
- # the user's identity: an object with the following attributes:
- #
- # * idp_id: unique identifier for the IdP
- # * idp_name: user-facing name for the IdP
- # * idp_icon: if specified in the IdP config, an MXC URI for an icon
- # for the IdP
- # * idp_brand: if specified in the IdP config, a textual identifier
- # for the brand of the IdP
- #
- # * HTML page shown after a successful user interactive authentication session:
- # 'sso_auth_success.html'.
- #
- # Note that this page must include the JavaScript which notifies of a successful authentication
- # (see https://matrix.org/docs/spec/client_server/r0.6.0#fallback).
- #
- # This template has no additional variables.
- #
- # * HTML page shown after a user-interactive authentication session which
- # does not map correctly onto the expected user: 'sso_auth_bad_user.html'.
- #
- # When rendering, this template is given the following variables:
- # * server_name: the homeserver's name.
- # * user_id_to_verify: the MXID of the user that we are trying to
- # validate.
- #
- # * HTML page shown during single sign-on if a deactivated user (according to Synapse's database)
- # attempts to login: 'sso_account_deactivated.html'.
- #
- # This template has no additional variables.
- #
- # * HTML page to display to users if something goes wrong during the
- # OpenID Connect authentication process: 'sso_error.html'.
- #
- # When rendering, this template is given two variables:
- # * error: the technical name of the error
- # * error_description: a human-readable message for the error
- #
- # You can see the default templates at:
- # https://github.com/matrix-org/synapse/tree/master/synapse/res/templates
- #
- #template_dir: "res/templates"
"""
diff --git a/synapse/federation/federation_client.py b/synapse/federation/federation_client.py
index 0af953a5d6..29979414e3 100644
--- a/synapse/federation/federation_client.py
+++ b/synapse/federation/federation_client.py
@@ -1364,13 +1364,59 @@ class FederationClient(FederationBase):
return room, children, inaccessible_children
- # TODO Fallback to the old federation API and translate the results.
- return await self._try_destination_list(
- "fetch room hierarchy",
- destinations,
- send_request,
- failover_on_unknown_endpoint=True,
- )
+ try:
+ return await self._try_destination_list(
+ "fetch room hierarchy",
+ destinations,
+ send_request,
+ failover_on_unknown_endpoint=True,
+ )
+ except SynapseError as e:
+ # Fallback to the old federation API and translate the results if
+ # no servers implement the new API.
+ #
+ # The algorithm below is a bit inefficient as it only attempts to
+ # get information for the requested room, but the legacy API may
+ # return additional layers.
+ if e.code == 502:
+ legacy_result = await self.get_space_summary(
+ destinations,
+ room_id,
+ suggested_only,
+ max_rooms_per_space=None,
+ exclude_rooms=[],
+ )
+
+ # Find the requested room in the response (and remove it).
+ for _i, room in enumerate(legacy_result.rooms):
+ if room.get("room_id") == room_id:
+ break
+ else:
+ # The requested room was not returned, nothing we can do.
+ raise
+ requested_room = legacy_result.rooms.pop(_i)
+
+ # Find any children events of the requested room.
+ children_events = []
+ children_room_ids = set()
+ for event in legacy_result.events:
+ if event.room_id == room_id:
+ children_events.append(event.data)
+ children_room_ids.add(event.state_key)
+ # And add them under the requested room.
+ requested_room["children_state"] = children_events
+
+ # Find the children rooms.
+ children = []
+ for room in legacy_result.rooms:
+ if room.get("room_id") in children_room_ids:
+ children.append(room)
+
+ # It isn't clear from the response whether some of the rooms are
+ # not accessible.
+ return requested_room, children, ()
+
+ raise
@attr.s(frozen=True, slots=True, auto_attribs=True)
@@ -1430,7 +1476,7 @@ class FederationSpaceSummaryEventResult:
class FederationSpaceSummaryResult:
"""Represents the data returned by a successful get_space_summary call."""
- rooms: Sequence[JsonDict]
+ rooms: List[JsonDict]
events: Sequence[FederationSpaceSummaryEventResult]
@classmethod
@@ -1444,7 +1490,7 @@ class FederationSpaceSummaryResult:
ValueError if d is not a valid /spaces/ response
"""
rooms = d.get("rooms")
- if not isinstance(rooms, Sequence):
+ if not isinstance(rooms, List):
raise ValueError("'rooms' must be a list")
if any(not isinstance(r, dict) for r in rooms):
raise ValueError("Invalid room in 'rooms' list")
diff --git a/synapse/module_api/__init__.py b/synapse/module_api/__init__.py
index 84bb7264a4..b11fa6393b 100644
--- a/synapse/module_api/__init__.py
+++ b/synapse/module_api/__init__.py
@@ -95,6 +95,7 @@ class ModuleApi:
self._state = hs.get_state_handler()
self._clock: Clock = hs.get_clock()
self._send_email_handler = hs.get_send_email_handler()
+ self.custom_template_dir = hs.config.server.custom_template_directory
try:
app_name = self._hs.config.email_app_name
@@ -613,10 +614,15 @@ class ModuleApi:
msec: float,
*args,
desc: Optional[str] = None,
+ run_on_all_instances: bool = False,
**kwargs,
):
"""Wraps a function as a background process and calls it repeatedly.
+ NOTE: Will only run on the instance that is configured to run
+ background processes (which is the main process by default), unless
+ `run_on_all_workers` is set.
+
Waits `msec` initially before calling `f` for the first time.
Args:
@@ -627,12 +633,14 @@ class ModuleApi:
msec: How long to wait between calls in milliseconds.
*args: Positional arguments to pass to function.
desc: The background task's description. Default to the function's name.
+ run_on_all_instances: Whether to run this on all instances, rather
+ than just the instance configured to run background tasks.
**kwargs: Key arguments to pass to function.
"""
if desc is None:
desc = f.__name__
- if self._hs.config.run_background_tasks:
+ if self._hs.config.run_background_tasks or run_on_all_instances:
self._clock.looping_call(
run_as_background_process,
msec,
@@ -689,7 +697,7 @@ class ModuleApi:
"""
return self._hs.config.read_templates(
filenames,
- (td for td in (custom_template_directory,) if td),
+ (td for td in (self.custom_template_dir, custom_template_directory) if td),
)
diff --git a/synapse/replication/slave/storage/room.py b/synapse/replication/slave/storage/room.py
deleted file mode 100644
index 8cc6de3f46..0000000000
--- a/synapse/replication/slave/storage/room.py
+++ /dev/null
@@ -1,37 +0,0 @@
-# Copyright 2015, 2016 OpenMarket Ltd
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-from synapse.replication.tcp.streams import PublicRoomsStream
-from synapse.storage.database import DatabasePool
-from synapse.storage.databases.main.room import RoomWorkerStore
-
-from ._base import BaseSlavedStore
-from ._slaved_id_tracker import SlavedIdTracker
-
-
-class RoomStore(RoomWorkerStore, BaseSlavedStore):
- def __init__(self, database: DatabasePool, db_conn, hs):
- super().__init__(database, db_conn, hs)
- self._public_room_id_gen = SlavedIdTracker(
- db_conn, "public_room_list_stream", "stream_id"
- )
-
- def get_current_public_room_stream_id(self):
- return self._public_room_id_gen.get_current_token()
-
- def process_replication_rows(self, stream_name, instance_name, token, rows):
- if stream_name == PublicRoomsStream.NAME:
- self._public_room_id_gen.advance(instance_name, token)
-
- return super().process_replication_rows(stream_name, instance_name, token, rows)
diff --git a/synapse/replication/tcp/streams/__init__.py b/synapse/replication/tcp/streams/__init__.py
index 4c0023c68a..f41eabd85e 100644
--- a/synapse/replication/tcp/streams/__init__.py
+++ b/synapse/replication/tcp/streams/__init__.py
@@ -32,7 +32,6 @@ from synapse.replication.tcp.streams._base import (
GroupServerStream,
PresenceFederationStream,
PresenceStream,
- PublicRoomsStream,
PushersStream,
PushRulesStream,
ReceiptsStream,
@@ -57,7 +56,6 @@ STREAMS_MAP = {
PushRulesStream,
PushersStream,
CachesStream,
- PublicRoomsStream,
DeviceListsStream,
ToDeviceStream,
FederationStream,
@@ -79,7 +77,6 @@ __all__ = [
"PushRulesStream",
"PushersStream",
"CachesStream",
- "PublicRoomsStream",
"DeviceListsStream",
"ToDeviceStream",
"TagAccountDataStream",
diff --git a/synapse/replication/tcp/streams/_base.py b/synapse/replication/tcp/streams/_base.py
index 3716c41bea..9b905aba9d 100644
--- a/synapse/replication/tcp/streams/_base.py
+++ b/synapse/replication/tcp/streams/_base.py
@@ -447,31 +447,6 @@ class CachesStream(Stream):
)
-class PublicRoomsStream(Stream):
- """The public rooms list changed"""
-
- PublicRoomsStreamRow = namedtuple(
- "PublicRoomsStreamRow",
- (
- "room_id", # str
- "visibility", # str
- "appservice_id", # str, optional
- "network_id", # str, optional
- ),
- )
-
- NAME = "public_rooms"
- ROW_TYPE = PublicRoomsStreamRow
-
- def __init__(self, hs):
- store = hs.get_datastore()
- super().__init__(
- hs.get_instance_name(),
- current_token_without_instance(store.get_current_public_room_stream_id),
- store.get_all_new_public_rooms,
- )
-
-
class DeviceListsStream(Stream):
"""Either a user has updated their devices or a remote server needs to be
told about a device update.
diff --git a/synapse/rest/admin/__init__.py b/synapse/rest/admin/__init__.py
index 2acaea3003..7f3051aef1 100644
--- a/synapse/rest/admin/__init__.py
+++ b/synapse/rest/admin/__init__.py
@@ -59,7 +59,6 @@ from synapse.rest.admin.users import (
SearchUsersRestServlet,
ShadowBanRestServlet,
UserAdminServlet,
- UserMediaRestServlet,
UserMembershipRestServlet,
UserRegisterServlet,
UserRestServletV2,
@@ -222,7 +221,6 @@ def register_servlets(hs: "HomeServer", http_server: HttpServer) -> None:
SendServerNoticeServlet(hs).register(http_server)
VersionServlet(hs).register(http_server)
UserAdminServlet(hs).register(http_server)
- UserMediaRestServlet(hs).register(http_server)
UserMembershipRestServlet(hs).register(http_server)
UserTokenRestServlet(hs).register(http_server)
UserRestServletV2(hs).register(http_server)
diff --git a/synapse/rest/admin/media.py b/synapse/rest/admin/media.py
index 5f0555039d..8ce443049e 100644
--- a/synapse/rest/admin/media.py
+++ b/synapse/rest/admin/media.py
@@ -18,14 +18,15 @@ from typing import TYPE_CHECKING, Tuple
from synapse.api.errors import AuthError, Codes, NotFoundError, SynapseError
from synapse.http.server import HttpServer
-from synapse.http.servlet import RestServlet, parse_boolean, parse_integer
+from synapse.http.servlet import RestServlet, parse_boolean, parse_integer, parse_string
from synapse.http.site import SynapseRequest
from synapse.rest.admin._base import (
admin_patterns,
assert_requester_is_admin,
assert_user_is_admin,
)
-from synapse.types import JsonDict
+from synapse.storage.databases.main.media_repository import MediaSortOrder
+from synapse.types import JsonDict, UserID
if TYPE_CHECKING:
from synapse.server import HomeServer
@@ -314,6 +315,165 @@ class DeleteMediaByDateSize(RestServlet):
return 200, {"deleted_media": deleted_media, "total": total}
+class UserMediaRestServlet(RestServlet):
+ """
+ Gets information about all uploaded local media for a specific `user_id`.
+ With DELETE request you can delete all this media.
+
+ Example:
+ http://localhost:8008/_synapse/admin/v1/users/@user:server/media
+
+ Args:
+ The parameters `from` and `limit` are required for pagination.
+ By default, a `limit` of 100 is used.
+ Returns:
+ A list of media and an integer representing the total number of
+ media that exist given for this user
+ """
+
+ PATTERNS = admin_patterns("/users/(?P<user_id>[^/]+)/media$")
+
+ def __init__(self, hs: "HomeServer"):
+ self.is_mine = hs.is_mine
+ self.auth = hs.get_auth()
+ self.store = hs.get_datastore()
+ self.media_repository = hs.get_media_repository()
+
+ async def on_GET(
+ self, request: SynapseRequest, user_id: str
+ ) -> Tuple[int, JsonDict]:
+ # This will always be set by the time Twisted calls us.
+ assert request.args is not None
+
+ await assert_requester_is_admin(self.auth, request)
+
+ if not self.is_mine(UserID.from_string(user_id)):
+ raise SynapseError(400, "Can only look up local users")
+
+ user = await self.store.get_user_by_id(user_id)
+ if user is None:
+ raise NotFoundError("Unknown user")
+
+ start = parse_integer(request, "from", default=0)
+ limit = parse_integer(request, "limit", default=100)
+
+ if start < 0:
+ raise SynapseError(
+ 400,
+ "Query parameter from must be a string representing a positive integer.",
+ errcode=Codes.INVALID_PARAM,
+ )
+
+ if limit < 0:
+ raise SynapseError(
+ 400,
+ "Query parameter limit must be a string representing a positive integer.",
+ errcode=Codes.INVALID_PARAM,
+ )
+
+ # If neither `order_by` nor `dir` is set, set the default order
+ # to newest media is on top for backward compatibility.
+ if b"order_by" not in request.args and b"dir" not in request.args:
+ order_by = MediaSortOrder.CREATED_TS.value
+ direction = "b"
+ else:
+ order_by = parse_string(
+ request,
+ "order_by",
+ default=MediaSortOrder.CREATED_TS.value,
+ allowed_values=(
+ MediaSortOrder.MEDIA_ID.value,
+ MediaSortOrder.UPLOAD_NAME.value,
+ MediaSortOrder.CREATED_TS.value,
+ MediaSortOrder.LAST_ACCESS_TS.value,
+ MediaSortOrder.MEDIA_LENGTH.value,
+ MediaSortOrder.MEDIA_TYPE.value,
+ MediaSortOrder.QUARANTINED_BY.value,
+ MediaSortOrder.SAFE_FROM_QUARANTINE.value,
+ ),
+ )
+ direction = parse_string(
+ request, "dir", default="f", allowed_values=("f", "b")
+ )
+
+ media, total = await self.store.get_local_media_by_user_paginate(
+ start, limit, user_id, order_by, direction
+ )
+
+ ret = {"media": media, "total": total}
+ if (start + limit) < total:
+ ret["next_token"] = start + len(media)
+
+ return 200, ret
+
+ async def on_DELETE(
+ self, request: SynapseRequest, user_id: str
+ ) -> Tuple[int, JsonDict]:
+ # This will always be set by the time Twisted calls us.
+ assert request.args is not None
+
+ await assert_requester_is_admin(self.auth, request)
+
+ if not self.is_mine(UserID.from_string(user_id)):
+ raise SynapseError(400, "Can only look up local users")
+
+ user = await self.store.get_user_by_id(user_id)
+ if user is None:
+ raise NotFoundError("Unknown user")
+
+ start = parse_integer(request, "from", default=0)
+ limit = parse_integer(request, "limit", default=100)
+
+ if start < 0:
+ raise SynapseError(
+ 400,
+ "Query parameter from must be a string representing a positive integer.",
+ errcode=Codes.INVALID_PARAM,
+ )
+
+ if limit < 0:
+ raise SynapseError(
+ 400,
+ "Query parameter limit must be a string representing a positive integer.",
+ errcode=Codes.INVALID_PARAM,
+ )
+
+ # If neither `order_by` nor `dir` is set, set the default order
+ # to newest media is on top for backward compatibility.
+ if b"order_by" not in request.args and b"dir" not in request.args:
+ order_by = MediaSortOrder.CREATED_TS.value
+ direction = "b"
+ else:
+ order_by = parse_string(
+ request,
+ "order_by",
+ default=MediaSortOrder.CREATED_TS.value,
+ allowed_values=(
+ MediaSortOrder.MEDIA_ID.value,
+ MediaSortOrder.UPLOAD_NAME.value,
+ MediaSortOrder.CREATED_TS.value,
+ MediaSortOrder.LAST_ACCESS_TS.value,
+ MediaSortOrder.MEDIA_LENGTH.value,
+ MediaSortOrder.MEDIA_TYPE.value,
+ MediaSortOrder.QUARANTINED_BY.value,
+ MediaSortOrder.SAFE_FROM_QUARANTINE.value,
+ ),
+ )
+ direction = parse_string(
+ request, "dir", default="f", allowed_values=("f", "b")
+ )
+
+ media, _ = await self.store.get_local_media_by_user_paginate(
+ start, limit, user_id, order_by, direction
+ )
+
+ deleted_media, total = await self.media_repository.delete_local_media_ids(
+ ([row["media_id"] for row in media])
+ )
+
+ return 200, {"deleted_media": deleted_media, "total": total}
+
+
def register_servlets_for_media_repo(hs: "HomeServer", http_server: HttpServer) -> None:
"""
Media repo specific APIs.
@@ -328,3 +488,4 @@ def register_servlets_for_media_repo(hs: "HomeServer", http_server: HttpServer)
ListMediaInRoom(hs).register(http_server)
DeleteMediaByID(hs).register(http_server)
DeleteMediaByDateSize(hs).register(http_server)
+ UserMediaRestServlet(hs).register(http_server)
diff --git a/synapse/rest/admin/users.py b/synapse/rest/admin/users.py
index 93193b0864..3c8a0c6883 100644
--- a/synapse/rest/admin/users.py
+++ b/synapse/rest/admin/users.py
@@ -35,7 +35,6 @@ from synapse.rest.admin._base import (
assert_user_is_admin,
)
from synapse.rest.client._base import client_patterns
-from synapse.storage.databases.main.media_repository import MediaSortOrder
from synapse.storage.databases.main.stats import UserSortOrder
from synapse.types import JsonDict, UserID
@@ -851,165 +850,6 @@ class PushersRestServlet(RestServlet):
return 200, {"pushers": filtered_pushers, "total": len(filtered_pushers)}
-class UserMediaRestServlet(RestServlet):
- """
- Gets information about all uploaded local media for a specific `user_id`.
- With DELETE request you can delete all this media.
-
- Example:
- http://localhost:8008/_synapse/admin/v1/users/@user:server/media
-
- Args:
- The parameters `from` and `limit` are required for pagination.
- By default, a `limit` of 100 is used.
- Returns:
- A list of media and an integer representing the total number of
- media that exist given for this user
- """
-
- PATTERNS = admin_patterns("/users/(?P<user_id>[^/]+)/media$")
-
- def __init__(self, hs: "HomeServer"):
- self.is_mine = hs.is_mine
- self.auth = hs.get_auth()
- self.store = hs.get_datastore()
- self.media_repository = hs.get_media_repository()
-
- async def on_GET(
- self, request: SynapseRequest, user_id: str
- ) -> Tuple[int, JsonDict]:
- # This will always be set by the time Twisted calls us.
- assert request.args is not None
-
- await assert_requester_is_admin(self.auth, request)
-
- if not self.is_mine(UserID.from_string(user_id)):
- raise SynapseError(400, "Can only look up local users")
-
- user = await self.store.get_user_by_id(user_id)
- if user is None:
- raise NotFoundError("Unknown user")
-
- start = parse_integer(request, "from", default=0)
- limit = parse_integer(request, "limit", default=100)
-
- if start < 0:
- raise SynapseError(
- 400,
- "Query parameter from must be a string representing a positive integer.",
- errcode=Codes.INVALID_PARAM,
- )
-
- if limit < 0:
- raise SynapseError(
- 400,
- "Query parameter limit must be a string representing a positive integer.",
- errcode=Codes.INVALID_PARAM,
- )
-
- # If neither `order_by` nor `dir` is set, set the default order
- # to newest media is on top for backward compatibility.
- if b"order_by" not in request.args and b"dir" not in request.args:
- order_by = MediaSortOrder.CREATED_TS.value
- direction = "b"
- else:
- order_by = parse_string(
- request,
- "order_by",
- default=MediaSortOrder.CREATED_TS.value,
- allowed_values=(
- MediaSortOrder.MEDIA_ID.value,
- MediaSortOrder.UPLOAD_NAME.value,
- MediaSortOrder.CREATED_TS.value,
- MediaSortOrder.LAST_ACCESS_TS.value,
- MediaSortOrder.MEDIA_LENGTH.value,
- MediaSortOrder.MEDIA_TYPE.value,
- MediaSortOrder.QUARANTINED_BY.value,
- MediaSortOrder.SAFE_FROM_QUARANTINE.value,
- ),
- )
- direction = parse_string(
- request, "dir", default="f", allowed_values=("f", "b")
- )
-
- media, total = await self.store.get_local_media_by_user_paginate(
- start, limit, user_id, order_by, direction
- )
-
- ret = {"media": media, "total": total}
- if (start + limit) < total:
- ret["next_token"] = start + len(media)
-
- return 200, ret
-
- async def on_DELETE(
- self, request: SynapseRequest, user_id: str
- ) -> Tuple[int, JsonDict]:
- # This will always be set by the time Twisted calls us.
- assert request.args is not None
-
- await assert_requester_is_admin(self.auth, request)
-
- if not self.is_mine(UserID.from_string(user_id)):
- raise SynapseError(400, "Can only look up local users")
-
- user = await self.store.get_user_by_id(user_id)
- if user is None:
- raise NotFoundError("Unknown user")
-
- start = parse_integer(request, "from", default=0)
- limit = parse_integer(request, "limit", default=100)
-
- if start < 0:
- raise SynapseError(
- 400,
- "Query parameter from must be a string representing a positive integer.",
- errcode=Codes.INVALID_PARAM,
- )
-
- if limit < 0:
- raise SynapseError(
- 400,
- "Query parameter limit must be a string representing a positive integer.",
- errcode=Codes.INVALID_PARAM,
- )
-
- # If neither `order_by` nor `dir` is set, set the default order
- # to newest media is on top for backward compatibility.
- if b"order_by" not in request.args and b"dir" not in request.args:
- order_by = MediaSortOrder.CREATED_TS.value
- direction = "b"
- else:
- order_by = parse_string(
- request,
- "order_by",
- default=MediaSortOrder.CREATED_TS.value,
- allowed_values=(
- MediaSortOrder.MEDIA_ID.value,
- MediaSortOrder.UPLOAD_NAME.value,
- MediaSortOrder.CREATED_TS.value,
- MediaSortOrder.LAST_ACCESS_TS.value,
- MediaSortOrder.MEDIA_LENGTH.value,
- MediaSortOrder.MEDIA_TYPE.value,
- MediaSortOrder.QUARANTINED_BY.value,
- MediaSortOrder.SAFE_FROM_QUARANTINE.value,
- ),
- )
- direction = parse_string(
- request, "dir", default="f", allowed_values=("f", "b")
- )
-
- media, _ = await self.store.get_local_media_by_user_paginate(
- start, limit, user_id, order_by, direction
- )
-
- deleted_media, total = await self.media_repository.delete_local_media_ids(
- ([row["media_id"] for row in media])
- )
-
- return 200, {"deleted_media": deleted_media, "total": total}
-
-
class UserTokenRestServlet(RestServlet):
"""An admin API for logging in as a user.
diff --git a/synapse/rest/client/room.py b/synapse/rest/client/room.py
index ed238b2141..c5c54564be 100644
--- a/synapse/rest/client/room.py
+++ b/synapse/rest/client/room.py
@@ -1141,10 +1141,10 @@ def register_servlets(hs: "HomeServer", http_server, is_worker=False):
JoinedRoomsRestServlet(hs).register(http_server)
RoomAliasListServlet(hs).register(http_server)
SearchRestServlet(hs).register(http_server)
+ RoomCreateRestServlet(hs).register(http_server)
# Some servlets only get registered for the main process.
if not is_worker:
- RoomCreateRestServlet(hs).register(http_server)
RoomForgetRestServlet(hs).register(http_server)
diff --git a/synapse/rest/synapse/client/new_user_consent.py b/synapse/rest/synapse/client/new_user_consent.py
index 488b97b32e..fc62a09b7f 100644
--- a/synapse/rest/synapse/client/new_user_consent.py
+++ b/synapse/rest/synapse/client/new_user_consent.py
@@ -46,6 +46,8 @@ class NewUserConsentResource(DirectServeHtmlResource):
self._consent_version = hs.config.consent.user_consent_version
def template_search_dirs():
+ if hs.config.server.custom_template_directory:
+ yield hs.config.server.custom_template_directory
if hs.config.sso.sso_template_dir:
yield hs.config.sso.sso_template_dir
yield hs.config.sso.default_template_dir
diff --git a/synapse/rest/synapse/client/pick_username.py b/synapse/rest/synapse/client/pick_username.py
index ab24ec0a8e..c15b83c387 100644
--- a/synapse/rest/synapse/client/pick_username.py
+++ b/synapse/rest/synapse/client/pick_username.py
@@ -74,6 +74,8 @@ class AccountDetailsResource(DirectServeHtmlResource):
self._sso_handler = hs.get_sso_handler()
def template_search_dirs():
+ if hs.config.server.custom_template_directory:
+ yield hs.config.server.custom_template_directory
if hs.config.sso.sso_template_dir:
yield hs.config.sso.sso_template_dir
yield hs.config.sso.default_template_dir
diff --git a/synapse/storage/databases/main/__init__.py b/synapse/storage/databases/main/__init__.py
index 8d9f07111d..01b918e12e 100644
--- a/synapse/storage/databases/main/__init__.py
+++ b/synapse/storage/databases/main/__init__.py
@@ -127,9 +127,6 @@ class DataStore(
self._clock = hs.get_clock()
self.database_engine = database.engine
- self._public_room_id_gen = StreamIdGenerator(
- db_conn, "public_room_list_stream", "stream_id"
- )
self._device_list_id_gen = StreamIdGenerator(
db_conn,
"device_lists_stream",
@@ -170,6 +167,7 @@ class DataStore(
sequence_name="cache_invalidation_stream_seq",
writers=[],
)
+
else:
self._cache_id_gen = None
diff --git a/synapse/storage/databases/main/room.py b/synapse/storage/databases/main/room.py
index 443e5f3315..f98b892598 100644
--- a/synapse/storage/databases/main/room.py
+++ b/synapse/storage/databases/main/room.py
@@ -73,6 +73,40 @@ class RoomWorkerStore(SQLBaseStore):
self.config = hs.config
+ async def store_room(
+ self,
+ room_id: str,
+ room_creator_user_id: str,
+ is_public: bool,
+ room_version: RoomVersion,
+ ):
+ """Stores a room.
+
+ Args:
+ room_id: The desired room ID, can be None.
+ room_creator_user_id: The user ID of the room creator.
+ is_public: True to indicate that this room should appear in
+ public room lists.
+ room_version: The version of the room
+ Raises:
+ StoreError if the room could not be stored.
+ """
+ try:
+ await self.db_pool.simple_insert(
+ "rooms",
+ {
+ "room_id": room_id,
+ "creator": room_creator_user_id,
+ "is_public": is_public,
+ "room_version": room_version.identifier,
+ "has_auth_chain_index": True,
+ },
+ desc="store_room",
+ )
+ except Exception as e:
+ logger.error("store_room with room_id=%s failed: %s", room_id, e)
+ raise StoreError(500, "Problem creating room.")
+
async def get_room(self, room_id: str) -> dict:
"""Retrieve a room.
@@ -890,55 +924,6 @@ class RoomWorkerStore(SQLBaseStore):
return total_media_quarantined
- async def get_all_new_public_rooms(
- self, instance_name: str, last_id: int, current_id: int, limit: int
- ) -> Tuple[List[Tuple[int, tuple]], int, bool]:
- """Get updates for public rooms replication stream.
-
- Args:
- instance_name: The writer we want to fetch updates from. Unused
- here since there is only ever one writer.
- last_id: The token to fetch updates from. Exclusive.
- current_id: The token to fetch updates up to. Inclusive.
- limit: The requested limit for the number of rows to return. The
- function may return more or fewer rows.
-
- Returns:
- A tuple consisting of: the updates, a token to use to fetch
- subsequent updates, and whether we returned fewer rows than exists
- between the requested tokens due to the limit.
-
- The token returned can be used in a subsequent call to this
- function to get further updatees.
-
- The updates are a list of 2-tuples of stream ID and the row data
- """
- if last_id == current_id:
- return [], current_id, False
-
- def get_all_new_public_rooms(txn):
- sql = """
- SELECT stream_id, room_id, visibility, appservice_id, network_id
- FROM public_room_list_stream
- WHERE stream_id > ? AND stream_id <= ?
- ORDER BY stream_id ASC
- LIMIT ?
- """
-
- txn.execute(sql, (last_id, current_id, limit))
- updates = [(row[0], row[1:]) for row in txn]
- limited = False
- upto_token = current_id
- if len(updates) >= limit:
- upto_token = updates[-1][0]
- limited = True
-
- return updates, upto_token, limited
-
- return await self.db_pool.runInteraction(
- "get_all_new_public_rooms", get_all_new_public_rooms
- )
-
async def get_rooms_for_retention_period_in_range(
self, min_ms: Optional[int], max_ms: Optional[int], include_null: bool = False
) -> Dict[str, dict]:
@@ -1391,57 +1376,6 @@ class RoomStore(RoomBackgroundUpdateStore, RoomWorkerStore, SearchStore):
lock=False,
)
- async def store_room(
- self,
- room_id: str,
- room_creator_user_id: str,
- is_public: bool,
- room_version: RoomVersion,
- ):
- """Stores a room.
-
- Args:
- room_id: The desired room ID, can be None.
- room_creator_user_id: The user ID of the room creator.
- is_public: True to indicate that this room should appear in
- public room lists.
- room_version: The version of the room
- Raises:
- StoreError if the room could not be stored.
- """
- try:
-
- def store_room_txn(txn, next_id):
- self.db_pool.simple_insert_txn(
- txn,
- "rooms",
- {
- "room_id": room_id,
- "creator": room_creator_user_id,
- "is_public": is_public,
- "room_version": room_version.identifier,
- "has_auth_chain_index": True,
- },
- )
- if is_public:
- self.db_pool.simple_insert_txn(
- txn,
- table="public_room_list_stream",
- values={
- "stream_id": next_id,
- "room_id": room_id,
- "visibility": is_public,
- },
- )
-
- async with self._public_room_id_gen.get_next() as next_id:
- await self.db_pool.runInteraction(
- "store_room_txn", store_room_txn, next_id
- )
- except Exception as e:
- logger.error("store_room with room_id=%s failed: %s", room_id, e)
- raise StoreError(500, "Problem creating room.")
-
async def maybe_store_room_on_outlier_membership(
self, room_id: str, room_version: RoomVersion
):
@@ -1470,49 +1404,14 @@ class RoomStore(RoomBackgroundUpdateStore, RoomWorkerStore, SearchStore):
lock=False,
)
- async def set_room_is_public(self, room_id, is_public):
- def set_room_is_public_txn(txn, next_id):
- self.db_pool.simple_update_one_txn(
- txn,
- table="rooms",
- keyvalues={"room_id": room_id},
- updatevalues={"is_public": is_public},
- )
-
- entries = self.db_pool.simple_select_list_txn(
- txn,
- table="public_room_list_stream",
- keyvalues={
- "room_id": room_id,
- "appservice_id": None,
- "network_id": None,
- },
- retcols=("stream_id", "visibility"),
- )
-
- entries.sort(key=lambda r: r["stream_id"])
-
- add_to_stream = True
- if entries:
- add_to_stream = bool(entries[-1]["visibility"]) != is_public
-
- if add_to_stream:
- self.db_pool.simple_insert_txn(
- txn,
- table="public_room_list_stream",
- values={
- "stream_id": next_id,
- "room_id": room_id,
- "visibility": is_public,
- "appservice_id": None,
- "network_id": None,
- },
- )
+ async def set_room_is_public(self, room_id: str, is_public: bool) -> None:
+ await self.db_pool.simple_update_one(
+ table="rooms",
+ keyvalues={"room_id": room_id},
+ updatevalues={"is_public": is_public},
+ desc="set_room_is_public",
+ )
- async with self._public_room_id_gen.get_next() as next_id:
- await self.db_pool.runInteraction(
- "set_room_is_public", set_room_is_public_txn, next_id
- )
self.hs.get_notifier().on_new_replication_data()
async def set_room_is_public_appservice(
@@ -1533,68 +1432,33 @@ class RoomStore(RoomBackgroundUpdateStore, RoomWorkerStore, SearchStore):
list.
"""
- def set_room_is_public_appservice_txn(txn, next_id):
- if is_public:
- try:
- self.db_pool.simple_insert_txn(
- txn,
- table="appservice_room_list",
- values={
- "appservice_id": appservice_id,
- "network_id": network_id,
- "room_id": room_id,
- },
- )
- except self.database_engine.module.IntegrityError:
- # We've already inserted, nothing to do.
- return
- else:
- self.db_pool.simple_delete_txn(
- txn,
- table="appservice_room_list",
- keyvalues={
- "appservice_id": appservice_id,
- "network_id": network_id,
- "room_id": room_id,
- },
- )
-
- entries = self.db_pool.simple_select_list_txn(
- txn,
- table="public_room_list_stream",
+ if is_public:
+ await self.db_pool.simple_upsert(
+ table="appservice_room_list",
keyvalues={
+ "appservice_id": appservice_id,
+ "network_id": network_id,
"room_id": room_id,
+ },
+ values={},
+ insertion_values={
"appservice_id": appservice_id,
"network_id": network_id,
+ "room_id": room_id,
},
- retcols=("stream_id", "visibility"),
+ desc="set_room_is_public_appservice_true",
)
-
- entries.sort(key=lambda r: r["stream_id"])
-
- add_to_stream = True
- if entries:
- add_to_stream = bool(entries[-1]["visibility"]) != is_public
-
- if add_to_stream:
- self.db_pool.simple_insert_txn(
- txn,
- table="public_room_list_stream",
- values={
- "stream_id": next_id,
- "room_id": room_id,
- "visibility": is_public,
- "appservice_id": appservice_id,
- "network_id": network_id,
- },
- )
-
- async with self._public_room_id_gen.get_next() as next_id:
- await self.db_pool.runInteraction(
- "set_room_is_public_appservice",
- set_room_is_public_appservice_txn,
- next_id,
+ else:
+ await self.db_pool.simple_delete(
+ table="appservice_room_list",
+ keyvalues={
+ "appservice_id": appservice_id,
+ "network_id": network_id,
+ "room_id": room_id,
+ },
+ desc="set_room_is_public_appservice_false",
)
+
self.hs.get_notifier().on_new_replication_data()
async def add_event_report(
@@ -1787,9 +1651,6 @@ class RoomStore(RoomBackgroundUpdateStore, RoomWorkerStore, SearchStore):
"get_event_reports_paginate", _get_event_reports_paginate_txn
)
- def get_current_public_room_stream_id(self):
- return self._public_room_id_gen.get_current_token()
-
async def block_room(self, room_id: str, user_id: str) -> None:
"""Marks the room as blocked. Can be called multiple times.
diff --git a/synapse/storage/schema/__init__.py b/synapse/storage/schema/__init__.py
index 7e0687e197..a5bc0ee8a5 100644
--- a/synapse/storage/schema/__init__.py
+++ b/synapse/storage/schema/__init__.py
@@ -12,7 +12,7 @@
# See the License for the specific language governing permissions and
# limitations under the License.
-SCHEMA_VERSION = 62
+SCHEMA_VERSION = 63
"""Represents the expectations made by the codebase about the database schema
This should be incremented whenever the codebase changes its requirements on the
@@ -25,6 +25,11 @@ for more information on how this works.
Changes in SCHEMA_VERSION = 61:
- The `user_stats_historical` and `room_stats_historical` tables are not written and
are not read (previously, they were written but not read).
+
+Changes in SCHEMA_VERSION = 63:
+ - The `public_room_list_stream` table is not written nor read to
+ (previously, it was written and read to, but not for any significant purpose).
+ https://github.com/matrix-org/synapse/pull/10565
"""
|
