diff --git a/.buildkite/test_db.db b/.buildkite/test_db.db
index 361369a581..a0d9f16a75 100644
--- a/.buildkite/test_db.db
+++ b/.buildkite/test_db.db
Binary files differdiff --git a/CHANGES.md b/CHANGES.md
index 8bd7825089..dc7d245812 100644
--- a/CHANGES.md
+++ b/CHANGES.md
@@ -1,3 +1,89 @@
+Synapse 1.23.0rc1 (2020-11-13)
+==============================
+
+Features
+--------
+
+- Add a push rule that highlights when a jitsi conference is created in a room. ([\#8286](https://github.com/matrix-org/synapse/issues/8286))
+- Add an admin api to delete a single file or files were not used for a defined time from server. Contributed by @dklimpel. ([\#8519](https://github.com/matrix-org/synapse/issues/8519))
+- Split admin API for reported events (`GET /_synapse/admin/v1/event_reports`) into detail and list endpoints. This is a breaking change to #8217 which was introduced in Synapse v1.21.0. Those who already use this API should check their scripts. Contributed by @dklimpel. ([\#8539](https://github.com/matrix-org/synapse/issues/8539))
+- Support generating structured logs via the standard logging configuration. ([\#8607](https://github.com/matrix-org/synapse/issues/8607), [\#8685](https://github.com/matrix-org/synapse/issues/8685))
+- Add an admin APIs to allow server admins to list users' pushers. Contributed by @dklimpel. ([\#8610](https://github.com/matrix-org/synapse/issues/8610), [\#8689](https://github.com/matrix-org/synapse/issues/8689))
+- Add an admin API `GET /_synapse/admin/v1/users/<user_id>/media` to get information about uploaded media. Contributed by @dklimpel. ([\#8647](https://github.com/matrix-org/synapse/issues/8647))
+- Add an admin API for local user media statistics. Contributed by @dklimpel. ([\#8700](https://github.com/matrix-org/synapse/issues/8700))
+- Add `displayname` to Shared-Secret Registration for admins. ([\#8722](https://github.com/matrix-org/synapse/issues/8722))
+
+
+Bugfixes
+--------
+
+- Fix fetching of E2E cross signing keys over federation when only one of the master key and device signing key is cached already. ([\#8455](https://github.com/matrix-org/synapse/issues/8455))
+- Fix a bug where Synapse would blindly forward bad responses from federation to clients when retrieving profile information. ([\#8580](https://github.com/matrix-org/synapse/issues/8580))
+- Fix a bug where the account validity endpoint would silently fail if the user ID did not have an expiration time. It now returns a 400 error. ([\#8620](https://github.com/matrix-org/synapse/issues/8620))
+- Fix email notifications for invites without local state. ([\#8627](https://github.com/matrix-org/synapse/issues/8627))
+- Fix handling of invalid group IDs to return a 400 rather than log an exception and return a 500. ([\#8628](https://github.com/matrix-org/synapse/issues/8628))
+- Fix handling of User-Agent headers that are invalid UTF-8, which caused user agents of users to not get correctly recorded. ([\#8632](https://github.com/matrix-org/synapse/issues/8632))
+- Fix a bug in the `joined_rooms` admin API if the user has never joined any rooms. The bug was introduced, along with the API, in v1.21.0. ([\#8643](https://github.com/matrix-org/synapse/issues/8643))
+- Fix exception during handling multiple concurrent requests for remote media when using multiple media repositories. ([\#8682](https://github.com/matrix-org/synapse/issues/8682))
+- Fix bug where Synapse would not recover after losing connection to the database. ([\#8726](https://github.com/matrix-org/synapse/issues/8726))
+- Fix bug where the `/_synapse/admin/v1/send_server_notice` API could send notices to non-notice rooms. ([\#8728](https://github.com/matrix-org/synapse/issues/8728))
+- Fix port script fails when DB has no backfilled events. Broke in v1.21.0. ([\#8729](https://github.com/matrix-org/synapse/issues/8729))
+- Fix port script to correctly handle foreign key constraints. Broke in v1.21.0. ([\#8730](https://github.com/matrix-org/synapse/issues/8730))
+- Fix port script so that it can be run again after a failure. Broke in v1.21.0. ([\#8755](https://github.com/matrix-org/synapse/issues/8755))
+
+
+Improved Documentation
+----------------------
+
+- Instructions for Azure AD in the OpenID Connect documentation. Contributed by peterk. ([\#8582](https://github.com/matrix-org/synapse/issues/8582))
+- Improve the sample configuration for single sign-on providers. ([\#8635](https://github.com/matrix-org/synapse/issues/8635))
+- Fix the filepath of Dex's example config and the link to Dex's Getting Started guide in the OpenID Connect docs. ([\#8657](https://github.com/matrix-org/synapse/issues/8657))
+- Note support for Python 3.9. ([\#8665](https://github.com/matrix-org/synapse/issues/8665))
+- Minor updates to docs on running tests. ([\#8666](https://github.com/matrix-org/synapse/issues/8666))
+- Interlink prometheus/grafana documentation. ([\#8667](https://github.com/matrix-org/synapse/issues/8667))
+- Notes on SSO logins and media_repository worker. ([\#8701](https://github.com/matrix-org/synapse/issues/8701))
+- Document experimental support for running multiple event persisters. ([\#8706](https://github.com/matrix-org/synapse/issues/8706))
+- Add information regarding the various sources of, and expected contributions to, Synapse's documentation to `CONTRIBUTING.md`. ([\#8714](https://github.com/matrix-org/synapse/issues/8714))
+- Migrate documentation `docs/admin_api/event_reports` to markdown. ([\#8742](https://github.com/matrix-org/synapse/issues/8742))
+- Add some helpful hints to the README for new Synapse developers. Contributed by @chagai95. ([\#8746](https://github.com/matrix-org/synapse/issues/8746))
+
+
+Internal Changes
+----------------
+
+- Optimise `/createRoom` with multiple invited users. ([\#8559](https://github.com/matrix-org/synapse/issues/8559))
+- Implement and use an @lru_cache decorator. ([\#8595](https://github.com/matrix-org/synapse/issues/8595))
+- Don't instansiate Requester directly. ([\#8614](https://github.com/matrix-org/synapse/issues/8614))
+- Type hints for `RegistrationStore`. ([\#8615](https://github.com/matrix-org/synapse/issues/8615))
+- Change schema to support access tokens belonging to one user but granting access to another. ([\#8616](https://github.com/matrix-org/synapse/issues/8616))
+- Remove unused OPTIONS handlers. ([\#8621](https://github.com/matrix-org/synapse/issues/8621))
+- Run `mypy` as part of the lint.sh script. ([\#8633](https://github.com/matrix-org/synapse/issues/8633))
+- Correct Synapse's PyPI package name in the OpenID Connect installation instructions. ([\#8634](https://github.com/matrix-org/synapse/issues/8634))
+- Catch exceptions during initialization of `password_providers`. Contributed by Nicolai Søborg. ([\#8636](https://github.com/matrix-org/synapse/issues/8636))
+- Fix typos and spelling errors in the code. ([\#8639](https://github.com/matrix-org/synapse/issues/8639))
+- Reduce number of OpenTracing spans started. ([\#8640](https://github.com/matrix-org/synapse/issues/8640), [\#8668](https://github.com/matrix-org/synapse/issues/8668), [\#8670](https://github.com/matrix-org/synapse/issues/8670))
+- Add field `total` to device list in admin API. ([\#8644](https://github.com/matrix-org/synapse/issues/8644))
+- Add more type hints to the application services code. ([\#8655](https://github.com/matrix-org/synapse/issues/8655), [\#8693](https://github.com/matrix-org/synapse/issues/8693))
+- Tell Black to format code for Python 3.5. ([\#8664](https://github.com/matrix-org/synapse/issues/8664))
+- Don't pull event from DB when handling replication traffic. ([\#8669](https://github.com/matrix-org/synapse/issues/8669))
+- Abstract some invite-related code in preparation for landing knocking. ([\#8671](https://github.com/matrix-org/synapse/issues/8671), [\#8688](https://github.com/matrix-org/synapse/issues/8688))
+- Clarify representation of events in logfiles. ([\#8679](https://github.com/matrix-org/synapse/issues/8679))
+- Don't require `hiredis` package to be installed to run unit tests. ([\#8680](https://github.com/matrix-org/synapse/issues/8680))
+- Fix typing info on cache call signature to accept `on_invalidate`. ([\#8684](https://github.com/matrix-org/synapse/issues/8684))
+- Fail tests if they do not await coroutines. ([\#8690](https://github.com/matrix-org/synapse/issues/8690))
+- Improve start time by adding an index to `e2e_cross_signing_keys.stream_id`. ([\#8694](https://github.com/matrix-org/synapse/issues/8694))
+- Re-organize the structured logging code to separate the TCP transport handling from the JSON formatting. ([\#8697](https://github.com/matrix-org/synapse/issues/8697))
+- Use Python 3.8 in Docker images by default. ([\#8698](https://github.com/matrix-org/synapse/issues/8698))
+- Remove the "draft" status of the Room Details Admin API. ([\#8702](https://github.com/matrix-org/synapse/issues/8702))
+- Improve the error returned when a non-string displayname or avatar_url is used when updating a user's profile. ([\#8705](https://github.com/matrix-org/synapse/issues/8705))
+- Block attempts by clients to send server ACLs, or redactions of server ACLs, that would result in the local server being blocked from the room. ([\#8708](https://github.com/matrix-org/synapse/issues/8708))
+- Add metrics the allow the local sysadmin to track 3PID `/requestToken` requests. ([\#8712](https://github.com/matrix-org/synapse/issues/8712))
+- Consolidate duplicated lists of purged tables that are checked in tests. ([\#8713](https://github.com/matrix-org/synapse/issues/8713))
+- Add some `mdui:UIInfo` element examples for `saml2_config` in the homeserver config. ([\#8718](https://github.com/matrix-org/synapse/issues/8718))
+- Improve the error message returned when a remote server incorrectly sets the `Content-Type` header in response to a JSON request. ([\#8719](https://github.com/matrix-org/synapse/issues/8719))
+- Speed up repeated state resolutions on the same room by caching event ID to auth event ID lookups. ([\#8752](https://github.com/matrix-org/synapse/issues/8752))
+
+
Synapse 1.22.1 (2020-10-30)
===========================
diff --git a/README.rst b/README.rst
index 59d5a4389b..d724cf97da 100644
--- a/README.rst
+++ b/README.rst
@@ -261,18 +261,22 @@ to install using pip and a virtualenv::
pip install -e ".[all,test]"
This will run a process of downloading and installing all the needed
-dependencies into a virtual env.
+dependencies into a virtual env. If any dependencies fail to install,
+try installing the failing modules individually::
-Once this is done, you may wish to run Synapse's unit tests, to
-check that everything is installed as it should be::
+ pip install -e "module-name"
+
+Once this is done, you may wish to run Synapse's unit tests to
+check that everything is installed correctly::
python -m twisted.trial tests
-This should end with a 'PASSED' result::
+This should end with a 'PASSED' result (note that exact numbers will
+differ)::
- Ran 1266 tests in 643.930s
+ Ran 1337 tests in 716.064s
- PASSED (skips=15, successes=1251)
+ PASSED (skips=15, successes=1322)
Running the Integration Tests
=============================
diff --git a/changelog.d/8286.feature b/changelog.d/8286.feature
deleted file mode 100644
index 2c371419af..0000000000
--- a/changelog.d/8286.feature
+++ /dev/null
@@ -1 +0,0 @@
-Add a push rule that highlights when a jitsi conference is created in a room.
diff --git a/changelog.d/8455.bugfix b/changelog.d/8455.bugfix
deleted file mode 100644
index 561e73f5e0..0000000000
--- a/changelog.d/8455.bugfix
+++ /dev/null
@@ -1 +0,0 @@
-Fix fetching of E2E cross signing keys over federation when only one of the master key and device signing key is cached already.
diff --git a/changelog.d/8519.feature b/changelog.d/8519.feature
deleted file mode 100644
index e2ab548681..0000000000
--- a/changelog.d/8519.feature
+++ /dev/null
@@ -1 +0,0 @@
-Add an admin api to delete a single file or files were not used for a defined time from server. Contributed by @dklimpel.
\ No newline at end of file
diff --git a/changelog.d/8539.feature b/changelog.d/8539.feature
deleted file mode 100644
index 15ce02fb86..0000000000
--- a/changelog.d/8539.feature
+++ /dev/null
@@ -1 +0,0 @@
-Split admin API for reported events (`GET /_synapse/admin/v1/event_reports`) into detail and list endpoints. This is a breaking change to #8217 which was introduced in Synapse v1.21.0. Those who already use this API should check their scripts. Contributed by @dklimpel.
\ No newline at end of file
diff --git a/changelog.d/8559.misc b/changelog.d/8559.misc
deleted file mode 100644
index d7bd00964e..0000000000
--- a/changelog.d/8559.misc
+++ /dev/null
@@ -1 +0,0 @@
-Optimise `/createRoom` with multiple invited users.
diff --git a/changelog.d/8580.bugfix b/changelog.d/8580.bugfix
deleted file mode 100644
index 31734fd97d..0000000000
--- a/changelog.d/8580.bugfix
+++ /dev/null
@@ -1 +0,0 @@
-Fix a bug where Synapse would blindly forward bad responses from federation to clients when retrieving profile information.
diff --git a/changelog.d/8582.doc b/changelog.d/8582.doc
deleted file mode 100644
index 041f168717..0000000000
--- a/changelog.d/8582.doc
+++ /dev/null
@@ -1 +0,0 @@
-Instructions for Azure AD in the OpenID Connect documentation. Contributed by peterk.
diff --git a/changelog.d/8595.misc b/changelog.d/8595.misc
deleted file mode 100644
index 24fab65cda..0000000000
--- a/changelog.d/8595.misc
+++ /dev/null
@@ -1 +0,0 @@
-Implement and use an @lru_cache decorator.
diff --git a/changelog.d/8607.feature b/changelog.d/8607.feature
deleted file mode 100644
index fef1eccb92..0000000000
--- a/changelog.d/8607.feature
+++ /dev/null
@@ -1 +0,0 @@
-Support generating structured logs via the standard logging configuration.
diff --git a/changelog.d/8610.feature b/changelog.d/8610.feature
deleted file mode 100644
index ed8d926964..0000000000
--- a/changelog.d/8610.feature
+++ /dev/null
@@ -1 +0,0 @@
-Add an admin APIs to allow server admins to list users' pushers. Contributed by @dklimpel.
\ No newline at end of file
diff --git a/changelog.d/8614.misc b/changelog.d/8614.misc
deleted file mode 100644
index 1bf9ea08f0..0000000000
--- a/changelog.d/8614.misc
+++ /dev/null
@@ -1 +0,0 @@
-Don't instansiate Requester directly.
diff --git a/changelog.d/8615.misc b/changelog.d/8615.misc
deleted file mode 100644
index 79fa7b7ff8..0000000000
--- a/changelog.d/8615.misc
+++ /dev/null
@@ -1 +0,0 @@
-Type hints for `RegistrationStore`.
diff --git a/changelog.d/8616.misc b/changelog.d/8616.misc
deleted file mode 100644
index 385b14063e..0000000000
--- a/changelog.d/8616.misc
+++ /dev/null
@@ -1 +0,0 @@
-Change schema to support access tokens belonging to one user but granting access to another.
diff --git a/changelog.d/8620.bugfix b/changelog.d/8620.bugfix
deleted file mode 100644
index c1078a3fb5..0000000000
--- a/changelog.d/8620.bugfix
+++ /dev/null
@@ -1 +0,0 @@
-Fix a bug where the account validity endpoint would silently fail if the user ID did not have an expiration time. It now returns a 400 error.
diff --git a/changelog.d/8621.misc b/changelog.d/8621.misc
deleted file mode 100644
index 5720b665fe..0000000000
--- a/changelog.d/8621.misc
+++ /dev/null
@@ -1 +0,0 @@
-Remove unused OPTIONS handlers.
diff --git a/changelog.d/8627.bugfix b/changelog.d/8627.bugfix
deleted file mode 100644
index 143cf95f92..0000000000
--- a/changelog.d/8627.bugfix
+++ /dev/null
@@ -1 +0,0 @@
-Fix email notifications for invites without local state.
diff --git a/changelog.d/8628.bugfix b/changelog.d/8628.bugfix
deleted file mode 100644
index 1316136ca2..0000000000
--- a/changelog.d/8628.bugfix
+++ /dev/null
@@ -1 +0,0 @@
-Fix handling of invalid group IDs to return a 400 rather than log an exception and return a 500.
diff --git a/changelog.d/8632.bugfix b/changelog.d/8632.bugfix
deleted file mode 100644
index 7d834aa2e2..0000000000
--- a/changelog.d/8632.bugfix
+++ /dev/null
@@ -1 +0,0 @@
-Fix handling of User-Agent headers that are invalid UTF-8, which caused user agents of users to not get correctly recorded.
diff --git a/changelog.d/8633.misc b/changelog.d/8633.misc
deleted file mode 100644
index 8e1d006b36..0000000000
--- a/changelog.d/8633.misc
+++ /dev/null
@@ -1 +0,0 @@
-Run `mypy` as part of the lint.sh script.
diff --git a/changelog.d/8634.misc b/changelog.d/8634.misc
deleted file mode 100644
index c4f74ba7c9..0000000000
--- a/changelog.d/8634.misc
+++ /dev/null
@@ -1 +0,0 @@
-Correct Synapse's PyPI package name in the OpenID Connect installation instructions.
\ No newline at end of file
diff --git a/changelog.d/8635.doc b/changelog.d/8635.doc
deleted file mode 100644
index 00fb1e61a7..0000000000
--- a/changelog.d/8635.doc
+++ /dev/null
@@ -1 +0,0 @@
-Improve the sample configuration for single sign-on providers.
diff --git a/changelog.d/8636.misc b/changelog.d/8636.misc
deleted file mode 100644
index df4dca42f8..0000000000
--- a/changelog.d/8636.misc
+++ /dev/null
@@ -1 +0,0 @@
-Catch exceptions during initialization of `password_providers`. Contributed by Nicolai Søborg.
diff --git a/changelog.d/8639.misc b/changelog.d/8639.misc
deleted file mode 100644
index 20a213df39..0000000000
--- a/changelog.d/8639.misc
+++ /dev/null
@@ -1 +0,0 @@
-Fix typos and spelling errors in the code.
diff --git a/changelog.d/8640.misc b/changelog.d/8640.misc
deleted file mode 100644
index cf6023f783..0000000000
--- a/changelog.d/8640.misc
+++ /dev/null
@@ -1 +0,0 @@
-Reduce number of OpenTracing spans started.
diff --git a/changelog.d/8643.bugfix b/changelog.d/8643.bugfix
deleted file mode 100644
index fcda1ca871..0000000000
--- a/changelog.d/8643.bugfix
+++ /dev/null
@@ -1 +0,0 @@
-Fix a bug in the `joined_rooms` admin API if the user has never joined any rooms. The bug was introduced, along with the API, in v1.21.0.
diff --git a/changelog.d/8644.misc b/changelog.d/8644.misc
deleted file mode 100644
index 87f2b72924..0000000000
--- a/changelog.d/8644.misc
+++ /dev/null
@@ -1 +0,0 @@
-Add field `total` to device list in admin API.
\ No newline at end of file
diff --git a/changelog.d/8647.feature b/changelog.d/8647.feature
deleted file mode 100644
index 79e98f6e90..0000000000
--- a/changelog.d/8647.feature
+++ /dev/null
@@ -1 +0,0 @@
-Add an admin API `GET /_synapse/admin/v1/users/<user_id>/media` to get information about uploaded media. Contributed by @dklimpel.
\ No newline at end of file
diff --git a/changelog.d/8655.misc b/changelog.d/8655.misc
deleted file mode 100644
index b588bdd3e2..0000000000
--- a/changelog.d/8655.misc
+++ /dev/null
@@ -1 +0,0 @@
-Add more type hints to the application services code.
diff --git a/changelog.d/8657.doc b/changelog.d/8657.doc
deleted file mode 100644
index 3dcbb221af..0000000000
--- a/changelog.d/8657.doc
+++ /dev/null
@@ -1 +0,0 @@
-Fix the filepath of Dex's example config and the link to Dex's Getting Started guide in the OpenID Connect docs.
diff --git a/changelog.d/8664.misc b/changelog.d/8664.misc
deleted file mode 100644
index 278cf53adc..0000000000
--- a/changelog.d/8664.misc
+++ /dev/null
@@ -1 +0,0 @@
-Tell Black to format code for Python 3.5.
diff --git a/changelog.d/8665.doc b/changelog.d/8665.doc
deleted file mode 100644
index 3b75307dc5..0000000000
--- a/changelog.d/8665.doc
+++ /dev/null
@@ -1 +0,0 @@
-Note support for Python 3.9.
diff --git a/changelog.d/8666.doc b/changelog.d/8666.doc
deleted file mode 100644
index dee86b4a26..0000000000
--- a/changelog.d/8666.doc
+++ /dev/null
@@ -1 +0,0 @@
-Minor updates to docs on running tests.
diff --git a/changelog.d/8667.doc b/changelog.d/8667.doc
deleted file mode 100644
index 422d697da6..0000000000
--- a/changelog.d/8667.doc
+++ /dev/null
@@ -1 +0,0 @@
-Interlink prometheus/grafana documentation.
diff --git a/changelog.d/8668.misc b/changelog.d/8668.misc
deleted file mode 100644
index cf6023f783..0000000000
--- a/changelog.d/8668.misc
+++ /dev/null
@@ -1 +0,0 @@
-Reduce number of OpenTracing spans started.
diff --git a/changelog.d/8669.misc b/changelog.d/8669.misc
deleted file mode 100644
index 5228105cd3..0000000000
--- a/changelog.d/8669.misc
+++ /dev/null
@@ -1 +0,0 @@
-Don't pull event from DB when handling replication traffic.
diff --git a/changelog.d/8670.misc b/changelog.d/8670.misc
deleted file mode 100644
index cf6023f783..0000000000
--- a/changelog.d/8670.misc
+++ /dev/null
@@ -1 +0,0 @@
-Reduce number of OpenTracing spans started.
diff --git a/changelog.d/8671.misc b/changelog.d/8671.misc
deleted file mode 100644
index bef8dc425a..0000000000
--- a/changelog.d/8671.misc
+++ /dev/null
@@ -1 +0,0 @@
-Abstract some invite-related code in preparation for landing knocking.
\ No newline at end of file
diff --git a/changelog.d/8679.misc b/changelog.d/8679.misc
deleted file mode 100644
index 662eced4cf..0000000000
--- a/changelog.d/8679.misc
+++ /dev/null
@@ -1 +0,0 @@
-Clarify representation of events in logfiles.
diff --git a/changelog.d/8680.misc b/changelog.d/8680.misc
deleted file mode 100644
index 2ca2975464..0000000000
--- a/changelog.d/8680.misc
+++ /dev/null
@@ -1 +0,0 @@
-Don't require `hiredis` package to be installed to run unit tests.
diff --git a/changelog.d/8682.bugfix b/changelog.d/8682.bugfix
deleted file mode 100644
index e61276aa05..0000000000
--- a/changelog.d/8682.bugfix
+++ /dev/null
@@ -1 +0,0 @@
-Fix exception during handling multiple concurrent requests for remote media when using multiple media repositories.
diff --git a/changelog.d/8684.misc b/changelog.d/8684.misc
deleted file mode 100644
index 1d23d42926..0000000000
--- a/changelog.d/8684.misc
+++ /dev/null
@@ -1 +0,0 @@
-Fix typing info on cache call signature to accept `on_invalidate`.
diff --git a/changelog.d/8685.feature b/changelog.d/8685.feature
deleted file mode 100644
index fef1eccb92..0000000000
--- a/changelog.d/8685.feature
+++ /dev/null
@@ -1 +0,0 @@
-Support generating structured logs via the standard logging configuration.
diff --git a/changelog.d/8688.misc b/changelog.d/8688.misc
deleted file mode 100644
index bef8dc425a..0000000000
--- a/changelog.d/8688.misc
+++ /dev/null
@@ -1 +0,0 @@
-Abstract some invite-related code in preparation for landing knocking.
\ No newline at end of file
diff --git a/changelog.d/8689.feature b/changelog.d/8689.feature
deleted file mode 100644
index ed8d926964..0000000000
--- a/changelog.d/8689.feature
+++ /dev/null
@@ -1 +0,0 @@
-Add an admin APIs to allow server admins to list users' pushers. Contributed by @dklimpel.
\ No newline at end of file
diff --git a/changelog.d/8690.misc b/changelog.d/8690.misc
deleted file mode 100644
index 0f38ba1f5d..0000000000
--- a/changelog.d/8690.misc
+++ /dev/null
@@ -1 +0,0 @@
-Fail tests if they do not await coroutines.
diff --git a/changelog.d/8693.misc b/changelog.d/8693.misc
deleted file mode 100644
index b588bdd3e2..0000000000
--- a/changelog.d/8693.misc
+++ /dev/null
@@ -1 +0,0 @@
-Add more type hints to the application services code.
diff --git a/changelog.d/8694.misc b/changelog.d/8694.misc
deleted file mode 100644
index c90a6375ad..0000000000
--- a/changelog.d/8694.misc
+++ /dev/null
@@ -1 +0,0 @@
-Improve start time by adding an index to `e2e_cross_signing_keys.stream_id`.
diff --git a/changelog.d/8697.misc b/changelog.d/8697.misc
deleted file mode 100644
index 7982a4e46d..0000000000
--- a/changelog.d/8697.misc
+++ /dev/null
@@ -1 +0,0 @@
- Re-organize the structured logging code to separate the TCP transport handling from the JSON formatting.
diff --git a/changelog.d/8698.misc b/changelog.d/8698.misc
deleted file mode 100644
index 6b777fb295..0000000000
--- a/changelog.d/8698.misc
+++ /dev/null
@@ -1 +0,0 @@
-Use Python 3.8 in Docker images by default.
diff --git a/changelog.d/8700.feature b/changelog.d/8700.feature
deleted file mode 100644
index 47d63dce02..0000000000
--- a/changelog.d/8700.feature
+++ /dev/null
@@ -1 +0,0 @@
-Add an admin API for local user media statistics. Contributed by @dklimpel.
diff --git a/changelog.d/8701.doc b/changelog.d/8701.doc
deleted file mode 100644
index e2e8b2f79a..0000000000
--- a/changelog.d/8701.doc
+++ /dev/null
@@ -1 +0,0 @@
-Notes on SSO logins and media_repository worker.
\ No newline at end of file
diff --git a/changelog.d/8702.misc b/changelog.d/8702.misc
deleted file mode 100644
index f20085cbe4..0000000000
--- a/changelog.d/8702.misc
+++ /dev/null
@@ -1 +0,0 @@
-Remove the "draft" status of the Room Details Admin API.
\ No newline at end of file
diff --git a/changelog.d/8705.misc b/changelog.d/8705.misc
deleted file mode 100644
index 1189464a02..0000000000
--- a/changelog.d/8705.misc
+++ /dev/null
@@ -1 +0,0 @@
-Improve the error returned when a non-string displayname or avatar_url is used when updating a user's profile.
\ No newline at end of file
diff --git a/changelog.d/8706.doc b/changelog.d/8706.doc
deleted file mode 100644
index 96a0427e73..0000000000
--- a/changelog.d/8706.doc
+++ /dev/null
@@ -1 +0,0 @@
-Document experimental support for running multiple event persisters.
diff --git a/changelog.d/8708.misc b/changelog.d/8708.misc
deleted file mode 100644
index be679fb0f8..0000000000
--- a/changelog.d/8708.misc
+++ /dev/null
@@ -1 +0,0 @@
-Block attempts by clients to send server ACLs, or redactions of server ACLs, that would result in the local server being blocked from the room.
diff --git a/changelog.d/8713.misc b/changelog.d/8713.misc
deleted file mode 100644
index c5d3f3216b..0000000000
--- a/changelog.d/8713.misc
+++ /dev/null
@@ -1 +0,0 @@
-Consolidate duplicated lists of purged tables that are checked in tests.
\ No newline at end of file
diff --git a/changelog.d/8714.doc b/changelog.d/8714.doc
deleted file mode 100644
index bda22714e7..0000000000
--- a/changelog.d/8714.doc
+++ /dev/null
@@ -1 +0,0 @@
-Add information regarding the various sources of, and expected contributions to, Synapse's documentation to `CONTRIBUTING.md`.
\ No newline at end of file
diff --git a/changelog.d/8719.misc b/changelog.d/8719.misc
deleted file mode 100644
index 9aabef8fc3..0000000000
--- a/changelog.d/8719.misc
+++ /dev/null
@@ -1 +0,0 @@
-Improve the error message returned when a remote server incorrectly sets the `Content-Type` header in response to a JSON request.
diff --git a/changelog.d/8722.feature b/changelog.d/8722.feature
deleted file mode 100644
index 0413d8838b..0000000000
--- a/changelog.d/8722.feature
+++ /dev/null
@@ -1 +0,0 @@
-Add `displayname` to Shared-Secret Registration for admins.
\ No newline at end of file
diff --git a/changelog.d/8728.bugfix b/changelog.d/8728.bugfix
deleted file mode 100644
index 8064aad0ff..0000000000
--- a/changelog.d/8728.bugfix
+++ /dev/null
@@ -1 +0,0 @@
-Fix bug where the `/_synapse/admin/v1/send_server_notice` API could send notices to non-notice rooms.
diff --git a/docs/admin_api/event_reports.md b/docs/admin_api/event_reports.md
new file mode 100644
index 0000000000..0159098138
--- /dev/null
+++ b/docs/admin_api/event_reports.md
@@ -0,0 +1,172 @@
+# Show reported events
+
+This API returns information about reported events.
+
+The api is:
+```
+GET /_synapse/admin/v1/event_reports?from=0&limit=10
+```
+To use it, you will need to authenticate by providing an `access_token` for a
+server admin: see [README.rst](README.rst).
+
+It returns a JSON body like the following:
+
+```json
+{
+ "event_reports": [
+ {
+ "event_id": "$bNUFCwGzWca1meCGkjp-zwslF-GfVcXukvRLI1_FaVY",
+ "id": 2,
+ "reason": "foo",
+ "score": -100,
+ "received_ts": 1570897107409,
+ "canonical_alias": "#alias1:matrix.org",
+ "room_id": "!ERAgBpSOcCCuTJqQPk:matrix.org",
+ "name": "Matrix HQ",
+ "sender": "@foobar:matrix.org",
+ "user_id": "@foo:matrix.org"
+ },
+ {
+ "event_id": "$3IcdZsDaN_En-S1DF4EMCy3v4gNRKeOJs8W5qTOKj4I",
+ "id": 3,
+ "reason": "bar",
+ "score": -100,
+ "received_ts": 1598889612059,
+ "canonical_alias": "#alias2:matrix.org",
+ "room_id": "!eGvUQuTCkHGVwNMOjv:matrix.org",
+ "name": "Your room name here",
+ "sender": "@foobar:matrix.org",
+ "user_id": "@bar:matrix.org"
+ }
+ ],
+ "next_token": 2,
+ "total": 4
+}
+```
+
+To paginate, check for `next_token` and if present, call the endpoint again with `from`
+set to the value of `next_token`. This will return a new page.
+
+If the endpoint does not return a `next_token` then there are no more reports to
+paginate through.
+
+**URL parameters:**
+
+* `limit`: integer - Is optional but is used for pagination, denoting the maximum number
+ of items to return in this call. Defaults to `100`.
+* `from`: integer - Is optional but used for pagination, denoting the offset in the
+ returned results. This should be treated as an opaque value and not explicitly set to
+ anything other than the return value of `next_token` from a previous call. Defaults to `0`.
+* `dir`: string - Direction of event report order. Whether to fetch the most recent
+ first (`b`) or the oldest first (`f`). Defaults to `b`.
+* `user_id`: string - Is optional and filters to only return users with user IDs that
+ contain this value. This is the user who reported the event and wrote the reason.
+* `room_id`: string - Is optional and filters to only return rooms with room IDs that
+ contain this value.
+
+**Response**
+
+The following fields are returned in the JSON response body:
+
+* `id`: integer - ID of event report.
+* `received_ts`: integer - The timestamp (in milliseconds since the unix epoch) when this
+ report was sent.
+* `room_id`: string - The ID of the room in which the event being reported is located.
+* `name`: string - The name of the room.
+* `event_id`: string - The ID of the reported event.
+* `user_id`: string - This is the user who reported the event and wrote the reason.
+* `reason`: string - Comment made by the `user_id` in this report. May be blank.
+* `score`: integer - Content is reported based upon a negative score, where -100 is
+ "most offensive" and 0 is "inoffensive".
+* `sender`: string - This is the ID of the user who sent the original message/event that
+ was reported.
+* `canonical_alias`: string - The canonical alias of the room. `null` if the room does not
+ have a canonical alias set.
+* `next_token`: integer - Indication for pagination. See above.
+* `total`: integer - Total number of event reports related to the query
+ (`user_id` and `room_id`).
+
+# Show details of a specific event report
+
+This API returns information about a specific event report.
+
+The api is:
+```
+GET /_synapse/admin/v1/event_reports/<report_id>
+```
+To use it, you will need to authenticate by providing an `access_token` for a
+server admin: see [README.rst](README.rst).
+
+It returns a JSON body like the following:
+
+```jsonc
+{
+ "event_id": "$bNUFCwGzWca1meCGkjp-zwslF-GfVcXukvRLI1_FaVY",
+ "event_json": {
+ "auth_events": [
+ "$YK4arsKKcc0LRoe700pS8DSjOvUT4NDv0HfInlMFw2M",
+ "$oggsNXxzPFRE3y53SUNd7nsj69-QzKv03a1RucHu-ws"
+ ],
+ "content": {
+ "body": "matrix.org: This Week in Matrix",
+ "format": "org.matrix.custom.html",
+ "formatted_body": "<strong>matrix.org</strong>:<br><a href=\"https://matrix.org/blog/\"><strong>This Week in Matrix</strong></a>",
+ "msgtype": "m.notice"
+ },
+ "depth": 546,
+ "hashes": {
+ "sha256": "xK1//xnmvHJIOvbgXlkI8eEqdvoMmihVDJ9J4SNlsAw"
+ },
+ "origin": "matrix.org",
+ "origin_server_ts": 1592291711430,
+ "prev_events": [
+ "$YK4arsKKcc0LRoe700pS8DSjOvUT4NDv0HfInlMFw2M"
+ ],
+ "prev_state": [],
+ "room_id": "!ERAgBpSOcCCuTJqQPk:matrix.org",
+ "sender": "@foobar:matrix.org",
+ "signatures": {
+ "matrix.org": {
+ "ed25519:a_JaEG": "cs+OUKW/iHx5pEidbWxh0UiNNHwe46Ai9LwNz+Ah16aWDNszVIe2gaAcVZfvNsBhakQTew51tlKmL2kspXk/Dg"
+ }
+ },
+ "type": "m.room.message",
+ "unsigned": {
+ "age_ts": 1592291711430,
+ }
+ },
+ "id": <report_id>,
+ "reason": "foo",
+ "score": -100,
+ "received_ts": 1570897107409,
+ "canonical_alias": "#alias1:matrix.org",
+ "room_id": "!ERAgBpSOcCCuTJqQPk:matrix.org",
+ "name": "Matrix HQ",
+ "sender": "@foobar:matrix.org",
+ "user_id": "@foo:matrix.org"
+}
+```
+
+**URL parameters:**
+
+* `report_id`: string - The ID of the event report.
+
+**Response**
+
+The following fields are returned in the JSON response body:
+
+* `id`: integer - ID of event report.
+* `received_ts`: integer - The timestamp (in milliseconds since the unix epoch) when this
+ report was sent.
+* `room_id`: string - The ID of the room in which the event being reported is located.
+* `name`: string - The name of the room.
+* `event_id`: string - The ID of the reported event.
+* `user_id`: string - This is the user who reported the event and wrote the reason.
+* `reason`: string - Comment made by the `user_id` in this report. May be blank.
+* `score`: integer - Content is reported based upon a negative score, where -100 is
+ "most offensive" and 0 is "inoffensive".
+* `sender`: string - This is the ID of the user who sent the original message/event that
+ was reported.
+* `canonical_alias`: string - The canonical alias of the room. `null` if the room does not
+ have a canonical alias set.
+* `event_json`: object - Details of the original event that was reported.
diff --git a/docs/admin_api/event_reports.rst b/docs/admin_api/event_reports.rst
deleted file mode 100644
index 5f7b0fa6bb..0000000000
--- a/docs/admin_api/event_reports.rst
+++ /dev/null
@@ -1,165 +0,0 @@
-Show reported events
-====================
-
-This API returns information about reported events.
-
-The api is::
-
- GET /_synapse/admin/v1/event_reports?from=0&limit=10
-
-To use it, you will need to authenticate by providing an ``access_token`` for a
-server admin: see `README.rst <README.rst>`_.
-
-It returns a JSON body like the following:
-
-.. code:: jsonc
-
- {
- "event_reports": [
- {
- "event_id": "$bNUFCwGzWca1meCGkjp-zwslF-GfVcXukvRLI1_FaVY",
- "id": 2,
- "reason": "foo",
- "score": -100,
- "received_ts": 1570897107409,
- "canonical_alias": "#alias1:matrix.org",
- "room_id": "!ERAgBpSOcCCuTJqQPk:matrix.org",
- "name": "Matrix HQ",
- "sender": "@foobar:matrix.org",
- "user_id": "@foo:matrix.org"
- },
- {
- "event_id": "$3IcdZsDaN_En-S1DF4EMCy3v4gNRKeOJs8W5qTOKj4I",
- "id": 3,
- "reason": "bar",
- "score": -100,
- "received_ts": 1598889612059,
- "canonical_alias": "#alias2:matrix.org",
- "room_id": "!eGvUQuTCkHGVwNMOjv:matrix.org",
- "name": "Your room name here",
- "sender": "@foobar:matrix.org",
- "user_id": "@bar:matrix.org"
- }
- ],
- "next_token": 2,
- "total": 4
- }
-
-To paginate, check for ``next_token`` and if present, call the endpoint again
-with ``from`` set to the value of ``next_token``. This will return a new page.
-
-If the endpoint does not return a ``next_token`` then there are no more
-reports to paginate through.
-
-**URL parameters:**
-
-- ``limit``: integer - Is optional but is used for pagination,
- denoting the maximum number of items to return in this call. Defaults to ``100``.
-- ``from``: integer - Is optional but used for pagination,
- denoting the offset in the returned results. This should be treated as an opaque value and
- not explicitly set to anything other than the return value of ``next_token`` from a previous call.
- Defaults to ``0``.
-- ``dir``: string - Direction of event report order. Whether to fetch the most recent first (``b``) or the
- oldest first (``f``). Defaults to ``b``.
-- ``user_id``: string - Is optional and filters to only return users with user IDs that contain this value.
- This is the user who reported the event and wrote the reason.
-- ``room_id``: string - Is optional and filters to only return rooms with room IDs that contain this value.
-
-**Response**
-
-The following fields are returned in the JSON response body:
-
-- ``id``: integer - ID of event report.
-- ``received_ts``: integer - The timestamp (in milliseconds since the unix epoch) when this report was sent.
-- ``room_id``: string - The ID of the room in which the event being reported is located.
-- ``name``: string - The name of the room.
-- ``event_id``: string - The ID of the reported event.
-- ``user_id``: string - This is the user who reported the event and wrote the reason.
-- ``reason``: string - Comment made by the ``user_id`` in this report. May be blank.
-- ``score``: integer - Content is reported based upon a negative score, where -100 is "most offensive" and 0 is "inoffensive".
-- ``sender``: string - This is the ID of the user who sent the original message/event that was reported.
-- ``canonical_alias``: string - The canonical alias of the room. ``null`` if the room does not have a canonical alias set.
-- ``next_token``: integer - Indication for pagination. See above.
-- ``total``: integer - Total number of event reports related to the query (``user_id`` and ``room_id``).
-
-Show details of a specific event report
-=======================================
-
-This API returns information about a specific event report.
-
-The api is::
-
- GET /_synapse/admin/v1/event_reports/<report_id>
-
-To use it, you will need to authenticate by providing an ``access_token`` for a
-server admin: see `README.rst <README.rst>`_.
-
-It returns a JSON body like the following:
-
-.. code:: jsonc
-
- {
- "event_id": "$bNUFCwGzWca1meCGkjp-zwslF-GfVcXukvRLI1_FaVY",
- "event_json": {
- "auth_events": [
- "$YK4arsKKcc0LRoe700pS8DSjOvUT4NDv0HfInlMFw2M",
- "$oggsNXxzPFRE3y53SUNd7nsj69-QzKv03a1RucHu-ws"
- ],
- "content": {
- "body": "matrix.org: This Week in Matrix",
- "format": "org.matrix.custom.html",
- "formatted_body": "<strong>matrix.org</strong>:<br><a href=\"https://matrix.org/blog/\"><strong>This Week in Matrix</strong></a>",
- "msgtype": "m.notice"
- },
- "depth": 546,
- "hashes": {
- "sha256": "xK1//xnmvHJIOvbgXlkI8eEqdvoMmihVDJ9J4SNlsAw"
- },
- "origin": "matrix.org",
- "origin_server_ts": 1592291711430,
- "prev_events": [
- "$YK4arsKKcc0LRoe700pS8DSjOvUT4NDv0HfInlMFw2M"
- ],
- "prev_state": [],
- "room_id": "!ERAgBpSOcCCuTJqQPk:matrix.org",
- "sender": "@foobar:matrix.org",
- "signatures": {
- "matrix.org": {
- "ed25519:a_JaEG": "cs+OUKW/iHx5pEidbWxh0UiNNHwe46Ai9LwNz+Ah16aWDNszVIe2gaAcVZfvNsBhakQTew51tlKmL2kspXk/Dg"
- }
- },
- "type": "m.room.message",
- "unsigned": {
- "age_ts": 1592291711430,
- }
- },
- "id": <report_id>,
- "reason": "foo",
- "score": -100,
- "received_ts": 1570897107409,
- "canonical_alias": "#alias1:matrix.org",
- "room_id": "!ERAgBpSOcCCuTJqQPk:matrix.org",
- "name": "Matrix HQ",
- "sender": "@foobar:matrix.org",
- "user_id": "@foo:matrix.org"
- }
-
-**URL parameters:**
-
-- ``report_id``: string - The ID of the event report.
-
-**Response**
-
-The following fields are returned in the JSON response body:
-
-- ``id``: integer - ID of event report.
-- ``received_ts``: integer - The timestamp (in milliseconds since the unix epoch) when this report was sent.
-- ``room_id``: string - The ID of the room in which the event being reported is located.
-- ``name``: string - The name of the room.
-- ``event_id``: string - The ID of the reported event.
-- ``user_id``: string - This is the user who reported the event and wrote the reason.
-- ``reason``: string - Comment made by the ``user_id`` in this report. May be blank.
-- ``score``: integer - Content is reported based upon a negative score, where -100 is "most offensive" and 0 is "inoffensive".
-- ``sender``: string - This is the ID of the user who sent the original message/event that was reported.
-- ``canonical_alias``: string - The canonical alias of the room. ``null`` if the room does not have a canonical alias set.
-- ``event_json``: object - Details of the original event that was reported.
diff --git a/docs/sample_config.yaml b/docs/sample_config.yaml
index 6cd95707af..1a6fe10750 100644
--- a/docs/sample_config.yaml
+++ b/docs/sample_config.yaml
@@ -1738,6 +1738,28 @@ saml2_config:
#description: ["My awesome SP", "en"]
#name: ["Test SP", "en"]
+ #ui_info:
+ # display_name:
+ # - lang: en
+ # text: "Display Name is the descriptive name of your service."
+ # description:
+ # - lang: en
+ # text: "Description should be a short paragraph explaining the purpose of the service."
+ # information_url:
+ # - lang: en
+ # text: "https://example.com/terms-of-service"
+ # privacy_statement_url:
+ # - lang: en
+ # text: "https://example.com/privacy-policy"
+ # keywords:
+ # - lang: en
+ # text: ["Matrix", "Element"]
+ # logo:
+ # - lang: en
+ # text: "https://example.com/logo.svg"
+ # width: "200"
+ # height: "80"
+
#organization:
# name: Example com
# display_name:
diff --git a/scripts/synapse_port_db b/scripts/synapse_port_db
index a117e61586..c982ca9350 100755
--- a/scripts/synapse_port_db
+++ b/scripts/synapse_port_db
@@ -22,7 +22,7 @@ import logging
import sys
import time
import traceback
-from typing import Optional
+from typing import Dict, Optional, Set
import yaml
@@ -294,6 +294,34 @@ class Porter(object):
return table, already_ported, total_to_port, forward_chunk, backward_chunk
+ async def get_table_constraints(self) -> Dict[str, Set[str]]:
+ """Returns a map of tables that have foreign key constraints to tables they depend on.
+ """
+
+ def _get_constraints(txn):
+ # We can pull the information about foreign key constraints out from
+ # the postgres schema tables.
+ sql = """
+ SELECT DISTINCT
+ tc.table_name,
+ ccu.table_name AS foreign_table_name
+ FROM
+ information_schema.table_constraints AS tc
+ INNER JOIN information_schema.constraint_column_usage AS ccu
+ USING (table_schema, constraint_name)
+ WHERE tc.constraint_type = 'FOREIGN KEY';
+ """
+ txn.execute(sql)
+
+ results = {}
+ for table, foreign_table in txn:
+ results.setdefault(table, set()).add(foreign_table)
+ return results
+
+ return await self.postgres_store.db_pool.runInteraction(
+ "get_table_constraints", _get_constraints
+ )
+
async def handle_table(
self, table, postgres_size, table_size, forward_chunk, backward_chunk
):
@@ -593,7 +621,18 @@ class Porter(object):
"create_port_table", create_port_table
)
- # Step 2. Get tables.
+ # Step 2. Set up sequences
+ #
+ # We do this before porting the tables so that event if we fail half
+ # way through the postgres DB always have sequences that are greater
+ # than their respective tables. If we don't then creating the
+ # `DataStore` object will fail due to the inconsistency.
+ self.progress.set_state("Setting up sequence generators")
+ await self._setup_state_group_id_seq()
+ await self._setup_user_id_seq()
+ await self._setup_events_stream_seqs()
+
+ # Step 3. Get tables.
self.progress.set_state("Fetching tables")
sqlite_tables = await self.sqlite_store.db_pool.simple_select_onecol(
table="sqlite_master", keyvalues={"type": "table"}, retcol="name"
@@ -608,7 +647,7 @@ class Porter(object):
tables = set(sqlite_tables) & set(postgres_tables)
logger.info("Found %d tables", len(tables))
- # Step 3. Figure out what still needs copying
+ # Step 4. Figure out what still needs copying
self.progress.set_state("Checking on port progress")
setup_res = await make_deferred_yieldable(
defer.gatherResults(
@@ -621,21 +660,43 @@ class Porter(object):
consumeErrors=True,
)
)
-
- # Step 4. Do the copying.
+ # Map from table name to args passed to `handle_table`, i.e. a tuple
+ # of: `postgres_size`, `table_size`, `forward_chunk`, `backward_chunk`.
+ tables_to_port_info_map = {r[0]: r[1:] for r in setup_res}
+
+ # Step 5. Do the copying.
+ #
+ # This is slightly convoluted as we need to ensure tables are ported
+ # in the correct order due to foreign key constraints.
self.progress.set_state("Copying to postgres")
- await make_deferred_yieldable(
- defer.gatherResults(
- [run_in_background(self.handle_table, *res) for res in setup_res],
- consumeErrors=True,
+
+ constraints = await self.get_table_constraints()
+ tables_ported = set() # type: Set[str]
+
+ while tables_to_port_info_map:
+ # Pulls out all tables that are still to be ported and which
+ # only depend on tables that are already ported (if any).
+ tables_to_port = [
+ table
+ for table in tables_to_port_info_map
+ if not constraints.get(table, set()) - tables_ported
+ ]
+
+ await make_deferred_yieldable(
+ defer.gatherResults(
+ [
+ run_in_background(
+ self.handle_table,
+ table,
+ *tables_to_port_info_map.pop(table),
+ )
+ for table in tables_to_port
+ ],
+ consumeErrors=True,
+ )
)
- )
- # Step 5. Set up sequences
- self.progress.set_state("Setting up sequence generators")
- await self._setup_state_group_id_seq()
- await self._setup_user_id_seq()
- await self._setup_events_stream_seqs()
+ tables_ported.update(tables_to_port)
self.progress.done()
except Exception as e:
@@ -794,45 +855,62 @@ class Porter(object):
return done, remaining + done
- def _setup_state_group_id_seq(self):
+ async def _setup_state_group_id_seq(self):
+ curr_id = await self.sqlite_store.db_pool.simple_select_one_onecol(
+ table="state_groups", keyvalues={}, retcol="MAX(id)", allow_none=True
+ )
+
+ if not curr_id:
+ return
+
def r(txn):
- txn.execute("SELECT MAX(id) FROM state_groups")
- curr_id = txn.fetchone()[0]
- if not curr_id:
- return
next_id = curr_id + 1
txn.execute("ALTER SEQUENCE state_group_id_seq RESTART WITH %s", (next_id,))
- return self.postgres_store.db_pool.runInteraction("setup_state_group_id_seq", r)
+ await self.postgres_store.db_pool.runInteraction("setup_state_group_id_seq", r)
+
+ async def _setup_user_id_seq(self):
+ curr_id = await self.sqlite_store.db_pool.runInteraction(
+ "setup_user_id_seq", find_max_generated_user_id_localpart
+ )
- def _setup_user_id_seq(self):
def r(txn):
- next_id = find_max_generated_user_id_localpart(txn) + 1
+ next_id = curr_id + 1
txn.execute("ALTER SEQUENCE user_id_seq RESTART WITH %s", (next_id,))
return self.postgres_store.db_pool.runInteraction("setup_user_id_seq", r)
- def _setup_events_stream_seqs(self):
- def r(txn):
- txn.execute("SELECT MAX(stream_ordering) FROM events")
- curr_id = txn.fetchone()[0]
- if curr_id:
- next_id = curr_id + 1
- txn.execute(
- "ALTER SEQUENCE events_stream_seq RESTART WITH %s", (next_id,)
- )
+ async def _setup_events_stream_seqs(self):
+ """Set the event stream sequences to the correct values.
+ """
+
+ # We get called before we've ported the events table, so we need to
+ # fetch the current positions from the SQLite store.
+ curr_forward_id = await self.sqlite_store.db_pool.simple_select_one_onecol(
+ table="events", keyvalues={}, retcol="MAX(stream_ordering)", allow_none=True
+ )
- txn.execute("SELECT -MIN(stream_ordering) FROM events")
- curr_id = txn.fetchone()[0]
- if curr_id:
- next_id = curr_id + 1
+ curr_backward_id = await self.sqlite_store.db_pool.simple_select_one_onecol(
+ table="events",
+ keyvalues={},
+ retcol="MAX(-MIN(stream_ordering), 1)",
+ allow_none=True,
+ )
+
+ def _setup_events_stream_seqs_set_pos(txn):
+ if curr_forward_id:
txn.execute(
- "ALTER SEQUENCE events_backfill_stream_seq RESTART WITH %s",
- (next_id,),
+ "ALTER SEQUENCE events_stream_seq RESTART WITH %s",
+ (curr_forward_id + 1,),
)
- return self.postgres_store.db_pool.runInteraction(
- "_setup_events_stream_seqs", r
+ txn.execute(
+ "ALTER SEQUENCE events_backfill_stream_seq RESTART WITH %s",
+ (curr_backward_id + 1,),
+ )
+
+ return await self.postgres_store.db_pool.runInteraction(
+ "_setup_events_stream_seqs", _setup_events_stream_seqs_set_pos,
)
diff --git a/synapse/__init__.py b/synapse/__init__.py
index 3e1df2b035..537f2239e5 100644
--- a/synapse/__init__.py
+++ b/synapse/__init__.py
@@ -48,7 +48,7 @@ try:
except ImportError:
pass
-__version__ = "1.22.1"
+__version__ = "1.23.0rc1"
if bool(os.environ.get("SYNAPSE_TEST_PATCH_LOG_CONTEXTS", False)):
# We import here so that we don't have to install a bunch of deps when
diff --git a/synapse/config/saml2_config.py b/synapse/config/saml2_config.py
index 778750f43b..2ff7dfb311 100644
--- a/synapse/config/saml2_config.py
+++ b/synapse/config/saml2_config.py
@@ -271,6 +271,28 @@ class SAML2Config(Config):
#description: ["My awesome SP", "en"]
#name: ["Test SP", "en"]
+ #ui_info:
+ # display_name:
+ # - lang: en
+ # text: "Display Name is the descriptive name of your service."
+ # description:
+ # - lang: en
+ # text: "Description should be a short paragraph explaining the purpose of the service."
+ # information_url:
+ # - lang: en
+ # text: "https://example.com/terms-of-service"
+ # privacy_statement_url:
+ # - lang: en
+ # text: "https://example.com/privacy-policy"
+ # keywords:
+ # - lang: en
+ # text: ["Matrix", "Element"]
+ # logo:
+ # - lang: en
+ # text: "https://example.com/logo.svg"
+ # width: "200"
+ # height: "80"
+
#organization:
# name: Example com
# display_name:
diff --git a/synapse/metrics/__init__.py b/synapse/metrics/__init__.py
index b8d2a8e8a9..cbf0dbb871 100644
--- a/synapse/metrics/__init__.py
+++ b/synapse/metrics/__init__.py
@@ -502,6 +502,16 @@ build_info.labels(
last_ticked = time.time()
+# 3PID send info
+threepid_send_requests = Histogram(
+ "synapse_threepid_send_requests_with_tries",
+ documentation="Number of requests for a 3pid token by try count. Note if"
+ " there is a request with try count of 4, then there would have been one"
+ " each for 1, 2 and 3",
+ buckets=(1, 2, 3, 4, 5, 10),
+ labelnames=("type", "reason"),
+)
+
class ReactorLastSeenMetric:
def collect(self):
diff --git a/synapse/rest/client/v2_alpha/account.py b/synapse/rest/client/v2_alpha/account.py
index 17cd87f596..f5cd397493 100644
--- a/synapse/rest/client/v2_alpha/account.py
+++ b/synapse/rest/client/v2_alpha/account.py
@@ -39,6 +39,7 @@ from synapse.http.servlet import (
parse_json_object_from_request,
parse_string,
)
+from synapse.metrics import threepid_send_requests
from synapse.push.mailer import Mailer
from synapse.types import UserID
from synapse.util.msisdn import phone_number_to_msisdn
@@ -145,6 +146,10 @@ class EmailPasswordRequestTokenRestServlet(RestServlet):
# Wrap the session id in a JSON object
ret = {"sid": sid}
+ threepid_send_requests.labels(type="email", reason="password_reset").observe(
+ send_attempt
+ )
+
return 200, ret
@@ -441,6 +446,10 @@ class EmailThreepidRequestTokenRestServlet(RestServlet):
# Wrap the session id in a JSON object
ret = {"sid": sid}
+ threepid_send_requests.labels(type="email", reason="add_threepid").observe(
+ send_attempt
+ )
+
return 200, ret
@@ -511,6 +520,10 @@ class MsisdnThreepidRequestTokenRestServlet(RestServlet):
next_link,
)
+ threepid_send_requests.labels(type="msisdn", reason="add_threepid").observe(
+ send_attempt
+ )
+
return 200, ret
diff --git a/synapse/rest/client/v2_alpha/register.py b/synapse/rest/client/v2_alpha/register.py
index 0725aa4512..0e54489b5b 100644
--- a/synapse/rest/client/v2_alpha/register.py
+++ b/synapse/rest/client/v2_alpha/register.py
@@ -47,6 +47,7 @@ from synapse.http.servlet import (
parse_json_object_from_request,
parse_string,
)
+from synapse.metrics import threepid_send_requests
from synapse.push.mailer import Mailer
from synapse.util.msisdn import phone_number_to_msisdn
from synapse.util.ratelimitutils import FederationRateLimiter
@@ -165,6 +166,10 @@ class EmailRegisterRequestTokenRestServlet(RestServlet):
# Wrap the session id in a JSON object
ret = {"sid": sid}
+ threepid_send_requests.labels(type="email", reason="register").observe(
+ send_attempt
+ )
+
return 200, ret
@@ -238,6 +243,10 @@ class MsisdnRegisterRequestTokenRestServlet(RestServlet):
next_link,
)
+ threepid_send_requests.labels(type="msisdn", reason="register").observe(
+ send_attempt
+ )
+
return 200, ret
diff --git a/synapse/storage/database.py b/synapse/storage/database.py
index a0572b2952..d1b5760c2c 100644
--- a/synapse/storage/database.py
+++ b/synapse/storage/database.py
@@ -88,13 +88,18 @@ def make_pool(
"""Get the connection pool for the database.
"""
+ # By default enable `cp_reconnect`. We need to fiddle with db_args in case
+ # someone has explicitly set `cp_reconnect`.
+ db_args = dict(db_config.config.get("args", {}))
+ db_args.setdefault("cp_reconnect", True)
+
return adbapi.ConnectionPool(
db_config.config["name"],
cp_reactor=reactor,
cp_openfun=lambda conn: engine.on_new_connection(
LoggingDatabaseConnection(conn, engine, "on_new_connection")
),
- **db_config.config.get("args", {}),
+ **db_args,
)
diff --git a/synapse/storage/databases/main/event_federation.py b/synapse/storage/databases/main/event_federation.py
index a6279a6c13..2e07c37340 100644
--- a/synapse/storage/databases/main/event_federation.py
+++ b/synapse/storage/databases/main/event_federation.py
@@ -26,6 +26,7 @@ from synapse.storage.databases.main.events_worker import EventsWorkerStore
from synapse.storage.databases.main.signatures import SignatureWorkerStore
from synapse.types import Collection
from synapse.util.caches.descriptors import cached
+from synapse.util.caches.lrucache import LruCache
from synapse.util.iterutils import batch_iter
logger = logging.getLogger(__name__)
@@ -40,6 +41,11 @@ class EventFederationWorkerStore(EventsWorkerStore, SignatureWorkerStore, SQLBas
self._delete_old_forward_extrem_cache, 60 * 60 * 1000
)
+ # Cache of event ID to list of auth event IDs and their depths.
+ self._event_auth_cache = LruCache(
+ 500000, "_event_auth_cache", size_callback=len
+ ) # type: LruCache[str, List[Tuple[str, int]]]
+
async def get_auth_chain(
self, event_ids: Collection[str], include_given: bool = False
) -> List[EventBase]:
@@ -84,17 +90,45 @@ class EventFederationWorkerStore(EventsWorkerStore, SignatureWorkerStore, SQLBas
else:
results = set()
- base_sql = "SELECT DISTINCT auth_id FROM event_auth WHERE "
+ # We pull out the depth simply so that we can populate the
+ # `_event_auth_cache` cache.
+ base_sql = """
+ SELECT a.event_id, auth_id, depth
+ FROM event_auth AS a
+ INNER JOIN events AS e ON (e.event_id = a.auth_id)
+ WHERE
+ """
front = set(event_ids)
while front:
new_front = set()
for chunk in batch_iter(front, 100):
- clause, args = make_in_list_sql_clause(
- txn.database_engine, "event_id", chunk
- )
- txn.execute(base_sql + clause, args)
- new_front.update(r[0] for r in txn)
+ # Pull the auth events either from the cache or DB.
+ to_fetch = [] # Event IDs to fetch from DB # type: List[str]
+ for event_id in chunk:
+ res = self._event_auth_cache.get(event_id)
+ if res is None:
+ to_fetch.append(event_id)
+ else:
+ new_front.update(auth_id for auth_id, depth in res)
+
+ if to_fetch:
+ clause, args = make_in_list_sql_clause(
+ txn.database_engine, "a.event_id", to_fetch
+ )
+ txn.execute(base_sql + clause, args)
+
+ # Note we need to batch up the results by event ID before
+ # adding to the cache.
+ to_cache = {}
+ for event_id, auth_event_id, auth_event_depth in txn:
+ to_cache.setdefault(event_id, []).append(
+ (auth_event_id, auth_event_depth)
+ )
+ new_front.add(auth_event_id)
+
+ for event_id, auth_events in to_cache.items():
+ self._event_auth_cache.set(event_id, auth_events)
new_front -= results
@@ -213,14 +247,38 @@ class EventFederationWorkerStore(EventsWorkerStore, SignatureWorkerStore, SQLBas
break
# Fetch the auth events and their depths of the N last events we're
- # currently walking
+ # currently walking, either from cache or DB.
search, chunk = search[:-100], search[-100:]
- clause, args = make_in_list_sql_clause(
- txn.database_engine, "a.event_id", [e_id for _, e_id in chunk]
- )
- txn.execute(base_sql + clause, args)
- for event_id, auth_event_id, auth_event_depth in txn:
+ found = [] # Results found # type: List[Tuple[str, str, int]]
+ to_fetch = [] # Event IDs to fetch from DB # type: List[str]
+ for _, event_id in chunk:
+ res = self._event_auth_cache.get(event_id)
+ if res is None:
+ to_fetch.append(event_id)
+ else:
+ found.extend((event_id, auth_id, depth) for auth_id, depth in res)
+
+ if to_fetch:
+ clause, args = make_in_list_sql_clause(
+ txn.database_engine, "a.event_id", to_fetch
+ )
+ txn.execute(base_sql + clause, args)
+
+ # We parse the results and add the to the `found` set and the
+ # cache (note we need to batch up the results by event ID before
+ # adding to the cache).
+ to_cache = {}
+ for event_id, auth_event_id, auth_event_depth in txn:
+ to_cache.setdefault(event_id, []).append(
+ (auth_event_id, auth_event_depth)
+ )
+ found.append((event_id, auth_event_id, auth_event_depth))
+
+ for event_id, auth_events in to_cache.items():
+ self._event_auth_cache.set(event_id, auth_events)
+
+ for event_id, auth_event_id, auth_event_depth in found:
event_to_auth_events.setdefault(event_id, set()).add(auth_event_id)
sets = event_to_missing_sets.get(auth_event_id)
|
