diff --git a/.buildkite/scripts/test_old_deps.sh b/.buildkite/scripts/test_old_deps.sh
index cdb77b556c..9905c4bc4f 100755
--- a/.buildkite/scripts/test_old_deps.sh
+++ b/.buildkite/scripts/test_old_deps.sh
@@ -6,7 +6,7 @@
set -ex
apt-get update
-apt-get install -y python3.5 python3.5-dev python3-pip libxml2-dev libxslt-dev zlib1g-dev tox
+apt-get install -y python3.5 python3.5-dev python3-pip libxml2-dev libxslt-dev xmlsec1 zlib1g-dev tox
export LANG="C.UTF-8"
diff --git a/CHANGES.md b/CHANGES.md
index 4237550818..662762c07c 100644
--- a/CHANGES.md
+++ b/CHANGES.md
@@ -1,3 +1,75 @@
+Synapse 1.24.0rc1 (2020-12-02)
+==============================
+
+Features
+--------
+
+- Add admin API for logging in as a user. ([\#8617](https://github.com/matrix-org/synapse/issues/8617))
+- Allow specification of the SAML IdP if the metadata returns multiple IdPs. ([\#8630](https://github.com/matrix-org/synapse/issues/8630))
+- Add support for re-trying generation of a localpart for OpenID Connect mapping providers. ([\#8801](https://github.com/matrix-org/synapse/issues/8801), [\#8855](https://github.com/matrix-org/synapse/issues/8855))
+- Allow Date header through CORS. Contributed by Nicolas Chamo. ([\#8804](https://github.com/matrix-org/synapse/issues/8804))
+- Add a config option, `push.group_by_unread_count`, which controls whether unread message counts in push notifications are defined as "the number of rooms with unread messages" or "total unread messages". ([\#8820](https://github.com/matrix-org/synapse/issues/8820))
+- Add `force_purge` option to delete-room admin api. ([\#8843](https://github.com/matrix-org/synapse/issues/8843))
+
+
+Bugfixes
+--------
+
+- Fix a bug where appservices may be sent an excessive amount of read receipts and presence. Broke in v1.22.0. ([\#8744](https://github.com/matrix-org/synapse/issues/8744))
+- Fix a bug in some federation APIs which could lead to unexpected behaviour if different parameters were set in the URI and the request body. ([\#8776](https://github.com/matrix-org/synapse/issues/8776))
+- Fix a bug where synctl could spawn duplicate copies of a worker. Contributed by Waylon Cude. ([\#8798](https://github.com/matrix-org/synapse/issues/8798))
+- Allow per-room profiles to be used for the server notice user. ([\#8799](https://github.com/matrix-org/synapse/issues/8799))
+- Fix bug where logging could break after a call to SIGHUP. ([\#8817](https://github.com/matrix-org/synapse/issues/8817))
+- Fix `register_new_matrix_user` failing with "Bad Request" when trailing slash is included in server URL. Contributed by @angdraug. ([\#8823](https://github.com/matrix-org/synapse/issues/8823))
+- Fix minor long-standing bug in login, where we would offer the `password` login type if a custom auth provider supported it, even if password login was disabled. ([\#8835](https://github.com/matrix-org/synapse/issues/8835))
+- Fix a long-standing bug which caused Synapse to require unspecified parameters during user-interactive authentication. ([\#8848](https://github.com/matrix-org/synapse/issues/8848))
+
+
+Improved Documentation
+----------------------
+
+- Clarify the usecase for an msisdn delegate. Contributed by Adrian Wannenmacher. ([\#8734](https://github.com/matrix-org/synapse/issues/8734))
+- Remove extraneous comma from JSON example in User Admin API docs. ([\#8771](https://github.com/matrix-org/synapse/issues/8771))
+- Update `turn-howto.md` with troubleshooting notes. ([\#8779](https://github.com/matrix-org/synapse/issues/8779))
+- Fix the example on how to set the `Content-Type` header in nginx for the Client Well-Known URI. ([\#8793](https://github.com/matrix-org/synapse/issues/8793))
+- Improve the documentation for the admin API to list all media in a room with respect to encrypted events. ([\#8795](https://github.com/matrix-org/synapse/issues/8795))
+- Update the formatting of the `push` section of the homeserver config file to better align with the [code style guidelines](https://github.com/matrix-org/synapse/blob/develop/docs/code_style.md#configuration-file-format). ([\#8818](https://github.com/matrix-org/synapse/issues/8818))
+- Improve documentation how to configure prometheus for workers. ([\#8822](https://github.com/matrix-org/synapse/issues/8822))
+- Update example prometheus console. ([\#8824](https://github.com/matrix-org/synapse/issues/8824))
+
+
+Deprecations and Removals
+-------------------------
+
+- Remove old `/_matrix/client/*/admin` endpoints which was deprecated since Synapse 1.20.0. ([\#8785](https://github.com/matrix-org/synapse/issues/8785))
+- Disable pretty printing JSON responses for curl. Users who want pretty-printed output should use [jq](https://stedolan.github.io/jq/) in combination with curl. Contributed by @tulir. ([\#8833](https://github.com/matrix-org/synapse/issues/8833))
+
+
+Internal Changes
+----------------
+
+- Simplify the way the `HomeServer` object caches its internal attributes. ([\#8565](https://github.com/matrix-org/synapse/issues/8565), [\#8851](https://github.com/matrix-org/synapse/issues/8851))
+- Add an example and documentation for clock skew to the SAML2 sample configuration to allow for clock/time difference between the homserver and IdP. Contributed by @localguru. ([\#8731](https://github.com/matrix-org/synapse/issues/8731))
+- Generalise `RoomMemberHandler._locally_reject_invite` to apply to more flows than just invite. ([\#8751](https://github.com/matrix-org/synapse/issues/8751))
+- Generalise `RoomStore.maybe_store_room_on_invite` to handle other, non-invite membership events. ([\#8754](https://github.com/matrix-org/synapse/issues/8754))
+- Refactor test utilities for injecting HTTP requests. ([\#8757](https://github.com/matrix-org/synapse/issues/8757), [\#8758](https://github.com/matrix-org/synapse/issues/8758), [\#8759](https://github.com/matrix-org/synapse/issues/8759), [\#8760](https://github.com/matrix-org/synapse/issues/8760), [\#8761](https://github.com/matrix-org/synapse/issues/8761), [\#8777](https://github.com/matrix-org/synapse/issues/8777))
+- Consolidate logic between the OpenID Connect and SAML code. ([\#8765](https://github.com/matrix-org/synapse/issues/8765))
+- Use `TYPE_CHECKING` instead of magic `MYPY` variable. ([\#8770](https://github.com/matrix-org/synapse/issues/8770))
+- Add a commandline script to sign arbitrary json objects. ([\#8772](https://github.com/matrix-org/synapse/issues/8772))
+- Minor log line improvements for the SSO mapping code used to generate Matrix IDs from SSO IDs. ([\#8773](https://github.com/matrix-org/synapse/issues/8773))
+- Add additional error checking for OpenID Connect and SAML mapping providers. ([\#8774](https://github.com/matrix-org/synapse/issues/8774), [\#8800](https://github.com/matrix-org/synapse/issues/8800))
+- Fix a bug introduced in v1.20.0 where the user-agent and IP address reported during user registration for CAS, OpenID Connect, and SAML were of the wrong form. ([\#8784](https://github.com/matrix-org/synapse/issues/8784))
+- Add type hints to HTTP abstractions. ([\#8806](https://github.com/matrix-org/synapse/issues/8806), [\#8812](https://github.com/matrix-org/synapse/issues/8812))
+- Remove unnecessary function arguments and add typing to several membership replication classes. ([\#8809](https://github.com/matrix-org/synapse/issues/8809))
+- Optimise the lookup for an invite from another homeserver when trying to reject it. ([\#8815](https://github.com/matrix-org/synapse/issues/8815))
+- Add tests for `password_auth_provider`s. ([\#8819](https://github.com/matrix-org/synapse/issues/8819))
+- Drop redundant database index on `event_json`. ([\#8845](https://github.com/matrix-org/synapse/issues/8845))
+- Simplify `uk.half-shot.msc2778.login.application_service` login handler. ([\#8847](https://github.com/matrix-org/synapse/issues/8847))
+- Refactor `password_auth_provider` support code. ([\#8849](https://github.com/matrix-org/synapse/issues/8849))
+- Add missing `ordering` to background database updates. ([\#8850](https://github.com/matrix-org/synapse/issues/8850))
+- Allow for specifying a room version when creating a room in unit tests via `RestHelper.create_room_as`. ([\#8854](https://github.com/matrix-org/synapse/issues/8854))
+
+
Synapse 1.23.0 (2020-11-18)
===========================
diff --git a/changelog.d/8565.misc b/changelog.d/8565.misc
deleted file mode 100644
index 7bef422618..0000000000
--- a/changelog.d/8565.misc
+++ /dev/null
@@ -1 +0,0 @@
-Simplify the way the `HomeServer` object caches its internal attributes.
diff --git a/changelog.d/8617.feature b/changelog.d/8617.feature
deleted file mode 100644
index 4f1e788506..0000000000
--- a/changelog.d/8617.feature
+++ /dev/null
@@ -1 +0,0 @@
-Add admin API for logging in as a user.
diff --git a/changelog.d/8630.feature b/changelog.d/8630.feature
deleted file mode 100644
index 706051f131..0000000000
--- a/changelog.d/8630.feature
+++ /dev/null
@@ -1 +0,0 @@
-Allow specification of the SAML IdP if the metadata returns multiple IdPs.
diff --git a/changelog.d/8731.misc b/changelog.d/8731.misc
deleted file mode 100644
index df5882e960..0000000000
--- a/changelog.d/8731.misc
+++ /dev/null
@@ -1 +0,0 @@
-Add an example and documentation for clock skew to the SAML2 sample configuration to allow for clock/time difference between the homserver and IdP. Contributed by @localguru.
diff --git a/changelog.d/8734.doc b/changelog.d/8734.doc
deleted file mode 100644
index 3bff9021c7..0000000000
--- a/changelog.d/8734.doc
+++ /dev/null
@@ -1 +0,0 @@
-Clarify the usecase for an msisdn delegate. Contributed by Adrian Wannenmacher.
diff --git a/changelog.d/8744.bugfix b/changelog.d/8744.bugfix
deleted file mode 100644
index f8f9630bd6..0000000000
--- a/changelog.d/8744.bugfix
+++ /dev/null
@@ -1 +0,0 @@
-Fix a bug where appservices may be sent an excessive amount of read receipts and presence. Broke in v1.22.0.
diff --git a/changelog.d/8751.misc b/changelog.d/8751.misc
deleted file mode 100644
index 204c280c0e..0000000000
--- a/changelog.d/8751.misc
+++ /dev/null
@@ -1 +0,0 @@
-Generalise `RoomMemberHandler._locally_reject_invite` to apply to more flows than just invite.
\ No newline at end of file
diff --git a/changelog.d/8754.misc b/changelog.d/8754.misc
deleted file mode 100644
index 0436bb1be7..0000000000
--- a/changelog.d/8754.misc
+++ /dev/null
@@ -1 +0,0 @@
-Generalise `RoomStore.maybe_store_room_on_invite` to handle other, non-invite membership events.
\ No newline at end of file
diff --git a/changelog.d/8757.misc b/changelog.d/8757.misc
deleted file mode 100644
index 54502e9b90..0000000000
--- a/changelog.d/8757.misc
+++ /dev/null
@@ -1 +0,0 @@
-Refactor test utilities for injecting HTTP requests.
diff --git a/changelog.d/8758.misc b/changelog.d/8758.misc
deleted file mode 100644
index 54502e9b90..0000000000
--- a/changelog.d/8758.misc
+++ /dev/null
@@ -1 +0,0 @@
-Refactor test utilities for injecting HTTP requests.
diff --git a/changelog.d/8759.misc b/changelog.d/8759.misc
deleted file mode 100644
index 54502e9b90..0000000000
--- a/changelog.d/8759.misc
+++ /dev/null
@@ -1 +0,0 @@
-Refactor test utilities for injecting HTTP requests.
diff --git a/changelog.d/8760.misc b/changelog.d/8760.misc
deleted file mode 100644
index 54502e9b90..0000000000
--- a/changelog.d/8760.misc
+++ /dev/null
@@ -1 +0,0 @@
-Refactor test utilities for injecting HTTP requests.
diff --git a/changelog.d/8761.misc b/changelog.d/8761.misc
deleted file mode 100644
index e6da7d038d..0000000000
--- a/changelog.d/8761.misc
+++ /dev/null
@@ -1 +0,0 @@
- Refactor test utilities for injecting HTTP requests.
diff --git a/changelog.d/8765.misc b/changelog.d/8765.misc
deleted file mode 100644
index 053f9acc9c..0000000000
--- a/changelog.d/8765.misc
+++ /dev/null
@@ -1 +0,0 @@
-Consolidate logic between the OpenID Connect and SAML code.
diff --git a/changelog.d/8770.misc b/changelog.d/8770.misc
deleted file mode 100644
index b5876a82f9..0000000000
--- a/changelog.d/8770.misc
+++ /dev/null
@@ -1 +0,0 @@
-Use `TYPE_CHECKING` instead of magic `MYPY` variable.
diff --git a/changelog.d/8771.doc b/changelog.d/8771.doc
deleted file mode 100644
index 297cf61e98..0000000000
--- a/changelog.d/8771.doc
+++ /dev/null
@@ -1 +0,0 @@
-Remove extraneous comma from JSON example in User Admin API docs.
\ No newline at end of file
diff --git a/changelog.d/8772.misc b/changelog.d/8772.misc
deleted file mode 100644
index d74d0a3d5d..0000000000
--- a/changelog.d/8772.misc
+++ /dev/null
@@ -1 +0,0 @@
-Add a commandline script to sign arbitrary json objects.
diff --git a/changelog.d/8773.misc b/changelog.d/8773.misc
deleted file mode 100644
index 62778ba410..0000000000
--- a/changelog.d/8773.misc
+++ /dev/null
@@ -1 +0,0 @@
-Minor log line improvements for the SSO mapping code used to generate Matrix IDs from SSO IDs.
diff --git a/changelog.d/8774.misc b/changelog.d/8774.misc
deleted file mode 100644
index 57cca8fee5..0000000000
--- a/changelog.d/8774.misc
+++ /dev/null
@@ -1 +0,0 @@
-Add additional error checking for OpenID Connect and SAML mapping providers.
diff --git a/changelog.d/8776.bugfix b/changelog.d/8776.bugfix
deleted file mode 100644
index dd7ebbeb86..0000000000
--- a/changelog.d/8776.bugfix
+++ /dev/null
@@ -1 +0,0 @@
-Fix a bug in some federation APIs which could lead to unexpected behaviour if different parameters were set in the URI and the request body.
diff --git a/changelog.d/8777.misc b/changelog.d/8777.misc
deleted file mode 100644
index e6da7d038d..0000000000
--- a/changelog.d/8777.misc
+++ /dev/null
@@ -1 +0,0 @@
- Refactor test utilities for injecting HTTP requests.
diff --git a/changelog.d/8779.doc b/changelog.d/8779.doc
deleted file mode 100644
index 3641ae7f91..0000000000
--- a/changelog.d/8779.doc
+++ /dev/null
@@ -1 +0,0 @@
-Update `turn-howto.md` with troubleshooting notes.
diff --git a/changelog.d/8784.misc b/changelog.d/8784.misc
deleted file mode 100644
index 18a4263398..0000000000
--- a/changelog.d/8784.misc
+++ /dev/null
@@ -1 +0,0 @@
-Fix a bug introduced in v1.20.0 where the user-agent and IP address reported during user registration for CAS, OpenID Connect, and SAML were of the wrong form.
diff --git a/changelog.d/8785.removal b/changelog.d/8785.removal
deleted file mode 100644
index ee8ee32598..0000000000
--- a/changelog.d/8785.removal
+++ /dev/null
@@ -1 +0,0 @@
-Remove old `/_matrix/client/*/admin` endpoints which was deprecated since Synapse 1.20.0.
\ No newline at end of file
diff --git a/changelog.d/8793.doc b/changelog.d/8793.doc
deleted file mode 100644
index f6eee1ea73..0000000000
--- a/changelog.d/8793.doc
+++ /dev/null
@@ -1 +0,0 @@
-Fix the example on how to set the `Content-Type` header in nginx for the Client Well-Known URI.
diff --git a/changelog.d/8795.doc b/changelog.d/8795.doc
deleted file mode 100644
index f97a74efb5..0000000000
--- a/changelog.d/8795.doc
+++ /dev/null
@@ -1 +0,0 @@
-Improve the documentation for the admin API to list all media in a room with respect to encrypted events.
diff --git a/changelog.d/8798.bugfix b/changelog.d/8798.bugfix
deleted file mode 100644
index 9bdb2b51ea..0000000000
--- a/changelog.d/8798.bugfix
+++ /dev/null
@@ -1 +0,0 @@
-Fix a bug where synctl could spawn duplicate copies of a worker. Contributed by Waylon Cude.
diff --git a/changelog.d/8799.bugfix b/changelog.d/8799.bugfix
deleted file mode 100644
index a7e6b3556d..0000000000
--- a/changelog.d/8799.bugfix
+++ /dev/null
@@ -1 +0,0 @@
-Allow per-room profiles to be used for the server notice user.
diff --git a/changelog.d/8801.feature b/changelog.d/8801.feature
deleted file mode 100644
index 77f7fe4e5d..0000000000
--- a/changelog.d/8801.feature
+++ /dev/null
@@ -1 +0,0 @@
-Add support for re-trying generation of a localpart for OpenID Connect mapping providers.
diff --git a/changelog.d/8804.feature b/changelog.d/8804.feature
deleted file mode 100644
index a907c8106c..0000000000
--- a/changelog.d/8804.feature
+++ /dev/null
@@ -1 +0,0 @@
-Allow Date header through CORS. Contributed by Nicolas Chamo.
diff --git a/changelog.d/8806.misc b/changelog.d/8806.misc
deleted file mode 100644
index ee144846a5..0000000000
--- a/changelog.d/8806.misc
+++ /dev/null
@@ -1 +0,0 @@
-Add type hints to HTTP abstractions.
diff --git a/changelog.d/8809.misc b/changelog.d/8809.misc
deleted file mode 100644
index bbf83cf18d..0000000000
--- a/changelog.d/8809.misc
+++ /dev/null
@@ -1 +0,0 @@
-Remove unnecessary function arguments and add typing to several membership replication classes.
\ No newline at end of file
diff --git a/changelog.d/8812.misc b/changelog.d/8812.misc
deleted file mode 100644
index ee144846a5..0000000000
--- a/changelog.d/8812.misc
+++ /dev/null
@@ -1 +0,0 @@
-Add type hints to HTTP abstractions.
diff --git a/changelog.d/8815.misc b/changelog.d/8815.misc
deleted file mode 100644
index 647edeb568..0000000000
--- a/changelog.d/8815.misc
+++ /dev/null
@@ -1 +0,0 @@
-Optimise the lookup for an invite from another homeserver when trying to reject it.
\ No newline at end of file
diff --git a/changelog.d/8817.bugfix b/changelog.d/8817.bugfix
deleted file mode 100644
index e45dbd2ba4..0000000000
--- a/changelog.d/8817.bugfix
+++ /dev/null
@@ -1 +0,0 @@
-Fix bug where logging could break after a call to SIGHUP.
diff --git a/changelog.d/8818.doc b/changelog.d/8818.doc
deleted file mode 100644
index 571b0e3f60..0000000000
--- a/changelog.d/8818.doc
+++ /dev/null
@@ -1 +0,0 @@
-Update the formatting of the `push` section of the homeserver config file to better align with the [code style guidelines](https://github.com/matrix-org/synapse/blob/develop/docs/code_style.md#configuration-file-format).
\ No newline at end of file
diff --git a/changelog.d/8819.misc b/changelog.d/8819.misc
deleted file mode 100644
index a5793273a5..0000000000
--- a/changelog.d/8819.misc
+++ /dev/null
@@ -1 +0,0 @@
-Add tests for `password_auth_provider`s.
diff --git a/changelog.d/8820.feature b/changelog.d/8820.feature
deleted file mode 100644
index 9e35861b11..0000000000
--- a/changelog.d/8820.feature
+++ /dev/null
@@ -1 +0,0 @@
-Add a config option, `push.group_by_unread_count`, which controls whether unread message counts in push notifications are defined as "the number of rooms with unread messages" or "total unread messages".
diff --git a/changelog.d/8822.doc b/changelog.d/8822.doc
deleted file mode 100644
index 4299245990..0000000000
--- a/changelog.d/8822.doc
+++ /dev/null
@@ -1 +0,0 @@
-Improve documentation how to configure prometheus for workers.
\ No newline at end of file
diff --git a/changelog.d/8823.bugfix b/changelog.d/8823.bugfix
deleted file mode 100644
index 74af1c20b6..0000000000
--- a/changelog.d/8823.bugfix
+++ /dev/null
@@ -1 +0,0 @@
-Fix `register_new_matrix_user` failing with "Bad Request" when trailing slash is included in server URL. Contributed by @angdraug.
diff --git a/changelog.d/8824.doc b/changelog.d/8824.doc
deleted file mode 100644
index 683b436328..0000000000
--- a/changelog.d/8824.doc
+++ /dev/null
@@ -1 +0,0 @@
-Update example prometheus console.
\ No newline at end of file
diff --git a/changelog.d/8833.removal b/changelog.d/8833.removal
deleted file mode 100644
index 5c2d195f94..0000000000
--- a/changelog.d/8833.removal
+++ /dev/null
@@ -1 +0,0 @@
-Disable pretty printing JSON responses for curl. Users who want pretty-printed output should use [jq](https://stedolan.github.io/jq/) in combination with curl. Contributed by @tulir.
diff --git a/changelog.d/8835.bugfix b/changelog.d/8835.bugfix
deleted file mode 100644
index 446d04aa55..0000000000
--- a/changelog.d/8835.bugfix
+++ /dev/null
@@ -1 +0,0 @@
-Fix minor long-standing bug in login, where we would offer the `password` login type if a custom auth provider supported it, even if password login was disabled.
diff --git a/changelog.d/8843.feature b/changelog.d/8843.feature
deleted file mode 100644
index 824d46d5aa..0000000000
--- a/changelog.d/8843.feature
+++ /dev/null
@@ -1 +0,0 @@
-Add `force_purge` option to delete-room admin api.
diff --git a/changelog.d/8845.misc b/changelog.d/8845.misc
deleted file mode 100644
index 7db1c31520..0000000000
--- a/changelog.d/8845.misc
+++ /dev/null
@@ -1 +0,0 @@
-Drop redundant database index on `event_json`.
diff --git a/changelog.d/8847.misc b/changelog.d/8847.misc
deleted file mode 100644
index 5028997b04..0000000000
--- a/changelog.d/8847.misc
+++ /dev/null
@@ -1 +0,0 @@
-Simplify `uk.half-shot.msc2778.login.application_service` login handler.
diff --git a/changelog.d/8848.bugfix b/changelog.d/8848.bugfix
deleted file mode 100644
index 499e66f05b..0000000000
--- a/changelog.d/8848.bugfix
+++ /dev/null
@@ -1 +0,0 @@
-Fix a long-standing bug which caused Synapse to require unspecified parameters during user-interactive authentication.
diff --git a/changelog.d/8849.misc b/changelog.d/8849.misc
deleted file mode 100644
index 3dd496ce61..0000000000
--- a/changelog.d/8849.misc
+++ /dev/null
@@ -1 +0,0 @@
-Refactor `password_auth_provider` support code.
diff --git a/changelog.d/8850.misc b/changelog.d/8850.misc
deleted file mode 100644
index 4b54b8dd87..0000000000
--- a/changelog.d/8850.misc
+++ /dev/null
@@ -1 +0,0 @@
-Add missing `ordering` to background database updates.
diff --git a/changelog.d/8851.misc b/changelog.d/8851.misc
deleted file mode 100644
index 7bef422618..0000000000
--- a/changelog.d/8851.misc
+++ /dev/null
@@ -1 +0,0 @@
-Simplify the way the `HomeServer` object caches its internal attributes.
diff --git a/changelog.d/8854.misc b/changelog.d/8854.misc
deleted file mode 100644
index 5895df2d5c..0000000000
--- a/changelog.d/8854.misc
+++ /dev/null
@@ -1 +0,0 @@
-Allow for specifying a room version when creating a room in unit tests via `RestHelper.create_room_as`.
\ No newline at end of file
diff --git a/synapse/__init__.py b/synapse/__init__.py
index 65c1f5aa3f..d33a99f230 100644
--- a/synapse/__init__.py
+++ b/synapse/__init__.py
@@ -48,7 +48,7 @@ try:
except ImportError:
pass
-__version__ = "1.23.0"
+__version__ = "1.24.0rc1"
if bool(os.environ.get("SYNAPSE_TEST_PATCH_LOG_CONTEXTS", False)):
# We import here so that we don't have to install a bunch of deps when
diff --git a/synapse/handlers/oidc_handler.py b/synapse/handlers/oidc_handler.py
index 78c4e94a9d..55c4377890 100644
--- a/synapse/handlers/oidc_handler.py
+++ b/synapse/handlers/oidc_handler.py
@@ -39,7 +39,7 @@ from synapse.handlers._base import BaseHandler
from synapse.handlers.sso import MappingException, UserAttributes
from synapse.http.site import SynapseRequest
from synapse.logging.context import make_deferred_yieldable
-from synapse.types import JsonDict, map_username_to_mxid_localpart
+from synapse.types import JsonDict, UserID, map_username_to_mxid_localpart
from synapse.util import json_decoder
if TYPE_CHECKING:
@@ -898,13 +898,39 @@ class OidcHandler(BaseHandler):
return UserAttributes(**attributes)
+ async def grandfather_existing_users() -> Optional[str]:
+ if self._allow_existing_users:
+ # If allowing existing users we want to generate a single localpart
+ # and attempt to match it.
+ attributes = await oidc_response_to_user_attributes(failures=0)
+
+ user_id = UserID(attributes.localpart, self.server_name).to_string()
+ users = await self.store.get_users_by_id_case_insensitive(user_id)
+ if users:
+ # If an existing matrix ID is returned, then use it.
+ if len(users) == 1:
+ previously_registered_user_id = next(iter(users))
+ elif user_id in users:
+ previously_registered_user_id = user_id
+ else:
+ # Do not attempt to continue generating Matrix IDs.
+ raise MappingException(
+ "Attempted to login as '{}' but it matches more than one user inexactly: {}".format(
+ user_id, users
+ )
+ )
+
+ return previously_registered_user_id
+
+ return None
+
return await self._sso_handler.get_mxid_from_sso(
self._auth_provider_id,
remote_user_id,
user_agent,
ip_address,
oidc_response_to_user_attributes,
- self._allow_existing_users,
+ grandfather_existing_users,
)
diff --git a/synapse/handlers/saml_handler.py b/synapse/handlers/saml_handler.py
index 34db10ffe4..76d4169fe2 100644
--- a/synapse/handlers/saml_handler.py
+++ b/synapse/handlers/saml_handler.py
@@ -265,10 +265,10 @@ class SamlHandler(BaseHandler):
return UserAttributes(
localpart=result.get("mxid_localpart"),
display_name=result.get("displayname"),
- emails=result.get("emails"),
+ emails=result.get("emails", []),
)
- with (await self._mapping_lock.queue(self._auth_provider_id)):
+ async def grandfather_existing_users() -> Optional[str]:
# backwards-compatibility hack: see if there is an existing user with a
# suitable mapping from the uid
if (
@@ -290,17 +290,18 @@ class SamlHandler(BaseHandler):
if users:
registered_user_id = list(users.keys())[0]
logger.info("Grandfathering mapping to %s", registered_user_id)
- await self.store.record_user_external_id(
- self._auth_provider_id, remote_user_id, registered_user_id
- )
return registered_user_id
+ return None
+
+ with (await self._mapping_lock.queue(self._auth_provider_id)):
return await self._sso_handler.get_mxid_from_sso(
self._auth_provider_id,
remote_user_id,
user_agent,
ip_address,
saml_response_to_remapped_user_attributes,
+ grandfather_existing_users,
)
def expire_sessions(self):
diff --git a/synapse/handlers/sso.py b/synapse/handlers/sso.py
index d963082210..f42b90e1bc 100644
--- a/synapse/handlers/sso.py
+++ b/synapse/handlers/sso.py
@@ -116,7 +116,7 @@ class SsoHandler(BaseHandler):
user_agent: str,
ip_address: str,
sso_to_matrix_id_mapper: Callable[[int], Awaitable[UserAttributes]],
- allow_existing_users: bool = False,
+ grandfather_existing_users: Optional[Callable[[], Awaitable[Optional[str]]]],
) -> str:
"""
Given an SSO ID, retrieve the user ID for it and possibly register the user.
@@ -125,6 +125,10 @@ class SsoHandler(BaseHandler):
if it has that matrix ID is returned regardless of the current mapping
logic.
+ If a callable is provided for grandfathering users, it is called and can
+ potentially return a matrix ID to use. If it does, the SSO ID is linked to
+ this matrix ID for subsequent calls.
+
The mapping function is called (potentially multiple times) to generate
a localpart for the user.
@@ -132,17 +136,6 @@ class SsoHandler(BaseHandler):
given user-agent and IP address and the SSO ID is linked to this matrix
ID for subsequent calls.
- If allow_existing_users is true the mapping function is only called once
- and results in:
-
- 1. The use of a previously registered matrix ID. In this case, the
- SSO ID is linked to the matrix ID. (Note it is possible that
- other SSO IDs are linked to the same matrix ID.)
- 2. An unused localpart, in which case the user is registered (as
- discussed above).
- 3. An error if the generated localpart matches multiple pre-existing
- matrix IDs. Generally this should not happen.
-
Args:
auth_provider_id: A unique identifier for this SSO provider, e.g.
"oidc" or "saml".
@@ -152,8 +145,9 @@ class SsoHandler(BaseHandler):
sso_to_matrix_id_mapper: A callable to generate the user attributes.
The only parameter is an integer which represents the amount of
times the returned mxid localpart mapping has failed.
- allow_existing_users: True if the localpart returned from the
- mapping provider can be linked to an existing matrix ID.
+ grandfather_existing_users: A callable which can return an previously
+ existing matrix ID. The SSO ID is then linked to the returned
+ matrix ID.
Returns:
The user ID associated with the SSO response.
@@ -171,6 +165,16 @@ class SsoHandler(BaseHandler):
if previously_registered_user_id:
return previously_registered_user_id
+ # Check for grandfathering of users.
+ if grandfather_existing_users:
+ previously_registered_user_id = await grandfather_existing_users()
+ if previously_registered_user_id:
+ # Future logins should also match this user ID.
+ await self.store.record_user_external_id(
+ auth_provider_id, remote_user_id, previously_registered_user_id
+ )
+ return previously_registered_user_id
+
# Otherwise, generate a new user.
for i in range(self._MAP_USERNAME_RETRIES):
try:
@@ -194,33 +198,7 @@ class SsoHandler(BaseHandler):
# Check if this mxid already exists
user_id = UserID(attributes.localpart, self.server_name).to_string()
- users = await self.store.get_users_by_id_case_insensitive(user_id)
- # Note, if allow_existing_users is true then the loop is guaranteed
- # to end on the first iteration: either by matching an existing user,
- # raising an error, or registering a new user. See the docstring for
- # more in-depth an explanation.
- if users and allow_existing_users:
- # If an existing matrix ID is returned, then use it.
- if len(users) == 1:
- previously_registered_user_id = next(iter(users))
- elif user_id in users:
- previously_registered_user_id = user_id
- else:
- # Do not attempt to continue generating Matrix IDs.
- raise MappingException(
- "Attempted to login as '{}' but it matches more than one user inexactly: {}".format(
- user_id, users
- )
- )
-
- # Future logins should also match this user ID.
- await self.store.record_user_external_id(
- auth_provider_id, remote_user_id, previously_registered_user_id
- )
-
- return previously_registered_user_id
-
- elif not users:
+ if not await self.store.get_users_by_id_case_insensitive(user_id):
# This mxid is free
break
else:
diff --git a/tests/handlers/test_oidc.py b/tests/handlers/test_oidc.py
index e880d32be6..d485af52fd 100644
--- a/tests/handlers/test_oidc.py
+++ b/tests/handlers/test_oidc.py
@@ -23,7 +23,7 @@ import pymacaroons
from twisted.python.failure import Failure
from twisted.web._newclient import ResponseDone
-from synapse.handlers.oidc_handler import OidcError, OidcHandler, OidcMappingProvider
+from synapse.handlers.oidc_handler import OidcError, OidcMappingProvider
from synapse.handlers.sso import MappingException
from synapse.types import UserID
@@ -127,13 +127,8 @@ async def get_json(url):
class OidcHandlerTestCase(HomeserverTestCase):
- def make_homeserver(self, reactor, clock):
-
- self.http_client = Mock(spec=["get_json"])
- self.http_client.get_json.side_effect = get_json
- self.http_client.user_agent = "Synapse Test"
-
- config = self.default_config()
+ def default_config(self):
+ config = super().default_config()
config["public_baseurl"] = BASE_URL
oidc_config = {
"enabled": True,
@@ -149,19 +144,24 @@ class OidcHandlerTestCase(HomeserverTestCase):
oidc_config.update(config.get("oidc_config", {}))
config["oidc_config"] = oidc_config
- hs = self.setup_test_homeserver(
- http_client=self.http_client,
- proxied_http_client=self.http_client,
- config=config,
- )
+ return config
+
+ def make_homeserver(self, reactor, clock):
- self.handler = OidcHandler(hs)
+ self.http_client = Mock(spec=["get_json"])
+ self.http_client.get_json.side_effect = get_json
+ self.http_client.user_agent = "Synapse Test"
+
+ hs = self.setup_test_homeserver(proxied_http_client=self.http_client)
+
+ self.handler = hs.get_oidc_handler()
+ sso_handler = hs.get_sso_handler()
# Mock the render error method.
self.render_error = Mock(return_value=None)
- self.handler._sso_handler.render_error = self.render_error
+ sso_handler.render_error = self.render_error
# Reduce the number of attempts when generating MXIDs.
- self.handler._sso_handler._MAP_USERNAME_RETRIES = 3
+ sso_handler._MAP_USERNAME_RETRIES = 3
return hs
@@ -731,6 +731,14 @@ class OidcHandlerTestCase(HomeserverTestCase):
)
self.assertEqual(mxid, "@test_user:test")
+ # Subsequent calls should map to the same mxid.
+ mxid = self.get_success(
+ self.handler._map_userinfo_to_user(
+ userinfo, token, "user-agent", "10.10.10.10"
+ )
+ )
+ self.assertEqual(mxid, "@test_user:test")
+
# Note that a second SSO user can be mapped to the same Matrix ID. (This
# requires a unique sub, but something that maps to the same matrix ID,
# in this case we'll just use the same username. A more realistic example
@@ -832,7 +840,7 @@ class OidcHandlerTestCase(HomeserverTestCase):
# test_user is already taken, so test_user1 gets registered instead.
self.assertEqual(mxid, "@test_user1:test")
- # Register all of the potential users for a particular username.
+ # Register all of the potential mxids for a particular OIDC username.
self.get_success(
store.register_user(user_id="@tester:test", password_hash=None)
)
diff --git a/tests/handlers/test_saml.py b/tests/handlers/test_saml.py
new file mode 100644
index 0000000000..e1e13a5faf
--- /dev/null
+++ b/tests/handlers/test_saml.py
@@ -0,0 +1,168 @@
+# Copyright 2020 The Matrix.org Foundation C.I.C.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import attr
+
+from synapse.handlers.sso import MappingException
+
+from tests.unittest import HomeserverTestCase, override_config
+
+# These are a few constants that are used as config parameters in the tests.
+BASE_URL = "https://synapse/"
+
+
+@attr.s
+class FakeAuthnResponse:
+ ava = attr.ib(type=dict)
+
+
+class TestMappingProvider:
+ def __init__(self, config, module):
+ pass
+
+ @staticmethod
+ def parse_config(config):
+ return
+
+ @staticmethod
+ def get_saml_attributes(config):
+ return {"uid"}, {"displayName"}
+
+ def get_remote_user_id(self, saml_response, client_redirect_url):
+ return saml_response.ava["uid"]
+
+ def saml_response_to_user_attributes(
+ self, saml_response, failures, client_redirect_url
+ ):
+ localpart = saml_response.ava["username"] + (str(failures) if failures else "")
+ return {"mxid_localpart": localpart, "displayname": None}
+
+
+class SamlHandlerTestCase(HomeserverTestCase):
+ def default_config(self):
+ config = super().default_config()
+ config["public_baseurl"] = BASE_URL
+ saml_config = {
+ "sp_config": {"metadata": {}},
+ # Disable grandfathering.
+ "grandfathered_mxid_source_attribute": None,
+ "user_mapping_provider": {"module": __name__ + ".TestMappingProvider"},
+ }
+
+ # Update this config with what's in the default config so that
+ # override_config works as expected.
+ saml_config.update(config.get("saml2_config", {}))
+ config["saml2_config"] = saml_config
+
+ return config
+
+ def make_homeserver(self, reactor, clock):
+ hs = self.setup_test_homeserver()
+
+ self.handler = hs.get_saml_handler()
+
+ # Reduce the number of attempts when generating MXIDs.
+ sso_handler = hs.get_sso_handler()
+ sso_handler._MAP_USERNAME_RETRIES = 3
+
+ return hs
+
+ def test_map_saml_response_to_user(self):
+ """Ensure that mapping the SAML response returned from a provider to an MXID works properly."""
+ saml_response = FakeAuthnResponse({"uid": "test_user", "username": "test_user"})
+ # The redirect_url doesn't matter with the default user mapping provider.
+ redirect_url = ""
+ mxid = self.get_success(
+ self.handler._map_saml_response_to_user(
+ saml_response, redirect_url, "user-agent", "10.10.10.10"
+ )
+ )
+ self.assertEqual(mxid, "@test_user:test")
+
+ @override_config({"saml2_config": {"grandfathered_mxid_source_attribute": "mxid"}})
+ def test_map_saml_response_to_existing_user(self):
+ """Existing users can log in with SAML account."""
+ store = self.hs.get_datastore()
+ self.get_success(
+ store.register_user(user_id="@test_user:test", password_hash=None)
+ )
+
+ # Map a user via SSO.
+ saml_response = FakeAuthnResponse(
+ {"uid": "tester", "mxid": ["test_user"], "username": "test_user"}
+ )
+ redirect_url = ""
+ mxid = self.get_success(
+ self.handler._map_saml_response_to_user(
+ saml_response, redirect_url, "user-agent", "10.10.10.10"
+ )
+ )
+ self.assertEqual(mxid, "@test_user:test")
+
+ # Subsequent calls should map to the same mxid.
+ mxid = self.get_success(
+ self.handler._map_saml_response_to_user(
+ saml_response, redirect_url, "user-agent", "10.10.10.10"
+ )
+ )
+ self.assertEqual(mxid, "@test_user:test")
+
+ def test_map_saml_response_to_invalid_localpart(self):
+ """If the mapping provider generates an invalid localpart it should be rejected."""
+ saml_response = FakeAuthnResponse({"uid": "test", "username": "föö"})
+ redirect_url = ""
+ e = self.get_failure(
+ self.handler._map_saml_response_to_user(
+ saml_response, redirect_url, "user-agent", "10.10.10.10"
+ ),
+ MappingException,
+ )
+ self.assertEqual(str(e.value), "localpart is invalid: föö")
+
+ def test_map_saml_response_to_user_retries(self):
+ """The mapping provider can retry generating an MXID if the MXID is already in use."""
+ store = self.hs.get_datastore()
+ self.get_success(
+ store.register_user(user_id="@test_user:test", password_hash=None)
+ )
+ saml_response = FakeAuthnResponse({"uid": "test", "username": "test_user"})
+ redirect_url = ""
+ mxid = self.get_success(
+ self.handler._map_saml_response_to_user(
+ saml_response, redirect_url, "user-agent", "10.10.10.10"
+ )
+ )
+ # test_user is already taken, so test_user1 gets registered instead.
+ self.assertEqual(mxid, "@test_user1:test")
+
+ # Register all of the potential mxids for a particular SAML username.
+ self.get_success(
+ store.register_user(user_id="@tester:test", password_hash=None)
+ )
+ for i in range(1, 3):
+ self.get_success(
+ store.register_user(user_id="@tester%d:test" % i, password_hash=None)
+ )
+
+ # Now attempt to map to a username, this will fail since all potential usernames are taken.
+ saml_response = FakeAuthnResponse({"uid": "tester", "username": "tester"})
+ e = self.get_failure(
+ self.handler._map_saml_response_to_user(
+ saml_response, redirect_url, "user-agent", "10.10.10.10"
+ ),
+ MappingException,
+ )
+ self.assertEqual(
+ str(e.value), "Unable to generate a Matrix ID from the SSO response"
+ )
|
