Merge branch 'release-v1.8.0' of github.com:matrix-org/synapse into matrix-org-hotfixes

64 files changed, 255 insertions, 83 deletions

diff --git a/CHANGES.md b/CHANGES.md
index 361fd1fc6c..df94f742c0 100644
--- a/CHANGES.md
+++ b/CHANGES.md
@@ -1,3 +1,73 @@
+Synapse 1.8.0rc1 (2020-01-07)
+=============================
+
+Features
+--------
+
+- Add v2 APIs for the `send_join` and `send_leave` federation endpoints (as described in [MSC1802](https://github.com/matrix-org/matrix-doc/pull/1802)). ([\#6349](https://github.com/matrix-org/synapse/issues/6349))
+- Add a develop script to generate full SQL schemas. ([\#6394](https://github.com/matrix-org/synapse/issues/6394))
+- Add custom SAML username mapping functinality through an external provider plugin. ([\#6411](https://github.com/matrix-org/synapse/issues/6411))
+- Automatically delete empty groups/communities. ([\#6453](https://github.com/matrix-org/synapse/issues/6453))
+- Add option `limit_profile_requests_to_users_who_share_rooms` to prevent requirement of a local user sharing a room with another user to query their profile information. ([\#6523](https://github.com/matrix-org/synapse/issues/6523))
+- Add an `export_signing_key` script to extract the public part of signing keys when rotating them. ([\#6546](https://github.com/matrix-org/synapse/issues/6546))
+- Add experimental config option to specify multiple databases. ([\#6580](https://github.com/matrix-org/synapse/issues/6580))
+- Raise an error if someone tries to use the `log_file` config option. ([\#6626](https://github.com/matrix-org/synapse/issues/6626))
+
+
+Bugfixes
+--------
+
+- Prevent redacted events from being returned during message search. ([\#6377](https://github.com/matrix-org/synapse/issues/6377), [\#6522](https://github.com/matrix-org/synapse/issues/6522))
+- Prevent error on trying to search a upgraded room when the server is not in the predecessor room. ([\#6385](https://github.com/matrix-org/synapse/issues/6385))
+- Improve performance of looking up cross-signing keys. ([\#6486](https://github.com/matrix-org/synapse/issues/6486))
+- Fix race which occasionally caused deleted devices to reappear. ([\#6514](https://github.com/matrix-org/synapse/issues/6514))
+- Fix missing row in `device_max_stream_id` that could cause unable to decrypt errors after server restart. ([\#6555](https://github.com/matrix-org/synapse/issues/6555))
+- Fix a bug which meant that we did not send systemd notifications on startup if acme was enabled. ([\#6571](https://github.com/matrix-org/synapse/issues/6571))
+- Fix exception when fetching the `matrix.org:ed25519:auto` key. ([\#6625](https://github.com/matrix-org/synapse/issues/6625))
+- Fix bug where a moderator upgraded a room and became an admin in the new room. ([\#6633](https://github.com/matrix-org/synapse/issues/6633))
+- Fix an error which was thrown by the `PresenceHandler` `_on_shutdown` handler. ([\#6640](https://github.com/matrix-org/synapse/issues/6640))
+- Fix exceptions in the synchrotron worker log when events are rejected. ([\#6645](https://github.com/matrix-org/synapse/issues/6645))
+- Ensure that upgraded rooms are removed from the directory. ([\#6648](https://github.com/matrix-org/synapse/issues/6648))
+- Fix a bug causing Synapse not to fetch missing events when it believes it has every event in the room. ([\#6652](https://github.com/matrix-org/synapse/issues/6652))
+
+
+Improved Documentation
+----------------------
+
+- Document the Room Shutdown Admin API. ([\#6541](https://github.com/matrix-org/synapse/issues/6541))
+- Reword sections of [docs/federate.md](docs/federate.md) that explained delegation at time of Synapse 1.0 transition. ([\#6601](https://github.com/matrix-org/synapse/issues/6601))
+- Added the section 'Configuration' in [docs/turn-howto.md](docs/turn-howto.md). ([\#6614](https://github.com/matrix-org/synapse/issues/6614))
+
+
+Deprecations and Removals
+-------------------------
+
+- Remove redundant code from event authorisation implementation. ([\#6502](https://github.com/matrix-org/synapse/issues/6502))
+- Remove unused, undocumented `/_matrix/content` API. ([\#6628](https://github.com/matrix-org/synapse/issues/6628))
+
+
+Internal Changes
+----------------
+
+- Add *experimental* support for multiple physical databases and split out state storage to separate data store. ([\#6245](https://github.com/matrix-org/synapse/issues/6245), [\#6510](https://github.com/matrix-org/synapse/issues/6510), [\#6511](https://github.com/matrix-org/synapse/issues/6511), [\#6513](https://github.com/matrix-org/synapse/issues/6513), [\#6564](https://github.com/matrix-org/synapse/issues/6564), [\#6565](https://github.com/matrix-org/synapse/issues/6565))
+- Port sections of code base to async/await. ([\#6496](https://github.com/matrix-org/synapse/issues/6496), [\#6504](https://github.com/matrix-org/synapse/issues/6504), [\#6505](https://github.com/matrix-org/synapse/issues/6505), [\#6517](https://github.com/matrix-org/synapse/issues/6517), [\#6559](https://github.com/matrix-org/synapse/issues/6559), [\#6647](https://github.com/matrix-org/synapse/issues/6647), [\#6653](https://github.com/matrix-org/synapse/issues/6653))
+- Remove `SnapshotCache` in favour of `ResponseCache`. ([\#6506](https://github.com/matrix-org/synapse/issues/6506))
+- Silence mypy errors for files outside those specified. ([\#6512](https://github.com/matrix-org/synapse/issues/6512))
+- Clean up some logging when handling incoming events over federation. ([\#6515](https://github.com/matrix-org/synapse/issues/6515))
+- Test more folders against mypy. ([\#6534](https://github.com/matrix-org/synapse/issues/6534))
+- Update `mypy` to new version. ([\#6537](https://github.com/matrix-org/synapse/issues/6537))
+- Adjust the sytest blacklist for worker mode. ([\#6538](https://github.com/matrix-org/synapse/issues/6538))
+- Remove unused `get_pagination_rows` methods from `EventSource` classes. ([\#6557](https://github.com/matrix-org/synapse/issues/6557))
+- Clean up logs from the push notifier at startup. ([\#6558](https://github.com/matrix-org/synapse/issues/6558))
+- Improve diagnostics on database upgrade failure. ([\#6570](https://github.com/matrix-org/synapse/issues/6570))
+- Reduce the reconnect time when worker replication fails, to make it easier to catch up. ([\#6617](https://github.com/matrix-org/synapse/issues/6617))
+- Simplify http handling by removing redundant `SynapseRequestFactory`. ([\#6619](https://github.com/matrix-org/synapse/issues/6619))
+- Add a workaround for synapse raising exceptions when fetching the notary's own key from the notary. ([\#6620](https://github.com/matrix-org/synapse/issues/6620))
+- Automate generation of the sample log config. ([\#6627](https://github.com/matrix-org/synapse/issues/6627))
+- Simplify event creation code by removing redundant queries on the `event_reference_hashes` table. ([\#6629](https://github.com/matrix-org/synapse/issues/6629))
+- Fix errors when `frozen_dicts` are enabled. ([\#6642](https://github.com/matrix-org/synapse/issues/6642))
+
+
 Synapse 1.7.3 (2019-12-31)
 ==========================
 
diff --git a/changelog.d/6245.misc b/changelog.d/6245.misc
deleted file mode 100644
index a3e6b8296e..0000000000
--- a/changelog.d/6245.misc
+++ /dev/null
@@ -1 +0,0 @@
-Split out state storage into separate data store.
diff --git a/changelog.d/6349.feature b/changelog.d/6349.feature
deleted file mode 100644
index 56c4fbf78e..0000000000
--- a/changelog.d/6349.feature
+++ /dev/null
@@ -1 +0,0 @@
-Implement v2 APIs for the `send_join` and `send_leave` federation endpoints (as described in [MSC1802](https://github.com/matrix-org/matrix-doc/pull/1802)).
diff --git a/changelog.d/6377.bugfix b/changelog.d/6377.bugfix
deleted file mode 100644
index ccda96962f..0000000000
--- a/changelog.d/6377.bugfix
+++ /dev/null
@@ -1 +0,0 @@
-Prevent redacted events from being returned during message search.
\ No newline at end of file
diff --git a/changelog.d/6385.bugfix b/changelog.d/6385.bugfix
deleted file mode 100644
index 7a2bc02170..0000000000
--- a/changelog.d/6385.bugfix
+++ /dev/null
@@ -1 +0,0 @@
-Prevent error on trying to search a upgraded room when the server is not in the predecessor room.
\ No newline at end of file
diff --git a/changelog.d/6394.feature b/changelog.d/6394.feature
deleted file mode 100644
index 1a0e8845ad..0000000000
--- a/changelog.d/6394.feature
+++ /dev/null
@@ -1 +0,0 @@
-Add a develop script to generate full SQL schemas.
\ No newline at end of file
diff --git a/changelog.d/6411.feature b/changelog.d/6411.feature
deleted file mode 100644
index ebea4a208d..0000000000
--- a/changelog.d/6411.feature
+++ /dev/null
@@ -1 +0,0 @@
-Allow custom SAML username mapping functinality through an external provider plugin.
\ No newline at end of file
diff --git a/changelog.d/6453.feature b/changelog.d/6453.feature
deleted file mode 100644
index e7bb801c6a..0000000000
--- a/changelog.d/6453.feature
+++ /dev/null
@@ -1 +0,0 @@
-Automatically delete empty groups/communities.
diff --git a/changelog.d/6486.bugfix b/changelog.d/6486.bugfix
deleted file mode 100644
index b98c5a9ae5..0000000000
--- a/changelog.d/6486.bugfix
+++ /dev/null
@@ -1 +0,0 @@
-Improve performance of looking up cross-signing keys.
diff --git a/changelog.d/6496.misc b/changelog.d/6496.misc
deleted file mode 100644
index 19c6e926b8..0000000000
--- a/changelog.d/6496.misc
+++ /dev/null
@@ -1 +0,0 @@
-Port synapse.handlers.initial_sync to async/await.
diff --git a/changelog.d/6502.removal b/changelog.d/6502.removal
deleted file mode 100644
index 0b72261d58..0000000000
--- a/changelog.d/6502.removal
+++ /dev/null
@@ -1 +0,0 @@
-Remove redundant code from event authorisation implementation.
diff --git a/changelog.d/6504.misc b/changelog.d/6504.misc
deleted file mode 100644
index 7c873459af..0000000000
--- a/changelog.d/6504.misc
+++ /dev/null
@@ -1 +0,0 @@
-Port handlers.account_data and handlers.account_validity to async/await.
diff --git a/changelog.d/6505.misc b/changelog.d/6505.misc
deleted file mode 100644
index 3a75b2d9dd..0000000000
--- a/changelog.d/6505.misc
+++ /dev/null
@@ -1 +0,0 @@
-Make `make_deferred_yieldable` to work with async/await.
diff --git a/changelog.d/6506.misc b/changelog.d/6506.misc
deleted file mode 100644
index 99d7a70bcf..0000000000
--- a/changelog.d/6506.misc
+++ /dev/null
@@ -1 +0,0 @@
-Remove `SnapshotCache` in favour of `ResponseCache`.
diff --git a/changelog.d/6510.misc b/changelog.d/6510.misc
deleted file mode 100644
index 214f06539b..0000000000
--- a/changelog.d/6510.misc
+++ /dev/null
@@ -1 +0,0 @@
-Change phone home stats to not assume there is a single database and report information about the database used by the main data store.
diff --git a/changelog.d/6511.misc b/changelog.d/6511.misc
deleted file mode 100644
index 19ce435e68..0000000000
--- a/changelog.d/6511.misc
+++ /dev/null
@@ -1 +0,0 @@
-Move database config from apps into HomeServer object.
diff --git a/changelog.d/6512.misc b/changelog.d/6512.misc
deleted file mode 100644
index 37a8099eec..0000000000
--- a/changelog.d/6512.misc
+++ /dev/null
@@ -1 +0,0 @@
-Silence mypy errors for files outside those specified.
diff --git a/changelog.d/6513.misc b/changelog.d/6513.misc
deleted file mode 100644
index 36700f5657..0000000000
--- a/changelog.d/6513.misc
+++ /dev/null
@@ -1 +0,0 @@
-Remove all assumptions of there being a single phyiscal DB apart from the `synapse.config`.
diff --git a/changelog.d/6514.bugfix b/changelog.d/6514.bugfix
deleted file mode 100644
index 6dc1985c24..0000000000
--- a/changelog.d/6514.bugfix
+++ /dev/null
@@ -1 +0,0 @@
-Fix race which occasionally caused deleted devices to reappear.
diff --git a/changelog.d/6515.misc b/changelog.d/6515.misc
deleted file mode 100644
index a9c303ed1c..0000000000
--- a/changelog.d/6515.misc
+++ /dev/null
@@ -1 +0,0 @@
-Clean up some logging when handling incoming events over federation.
diff --git a/changelog.d/6517.misc b/changelog.d/6517.misc
deleted file mode 100644
index c6ffed9952..0000000000
--- a/changelog.d/6517.misc
+++ /dev/null
@@ -1 +0,0 @@
-Port some of FederationHandler to async/await.
\ No newline at end of file
diff --git a/changelog.d/6522.bugfix b/changelog.d/6522.bugfix
deleted file mode 100644
index ccda96962f..0000000000
--- a/changelog.d/6522.bugfix
+++ /dev/null
@@ -1 +0,0 @@
-Prevent redacted events from being returned during message search.
\ No newline at end of file
diff --git a/changelog.d/6523.feature b/changelog.d/6523.feature
deleted file mode 100644
index 798fa143df..0000000000
--- a/changelog.d/6523.feature
+++ /dev/null
@@ -1 +0,0 @@
-Add option `limit_profile_requests_to_users_who_share_rooms` to prevent requirement of a local user sharing a room with another user to query their profile information.
diff --git a/changelog.d/6534.misc b/changelog.d/6534.misc
deleted file mode 100644
index 7df6bb442a..0000000000
--- a/changelog.d/6534.misc
+++ /dev/null
@@ -1 +0,0 @@
-Test more folders against mypy.
diff --git a/changelog.d/6537.misc b/changelog.d/6537.misc
deleted file mode 100644
index 3543153584..0000000000
--- a/changelog.d/6537.misc
+++ /dev/null
@@ -1 +0,0 @@
-Update `mypy` to new version.
diff --git a/changelog.d/6538.misc b/changelog.d/6538.misc
deleted file mode 100644
index cb4fd56948..0000000000
--- a/changelog.d/6538.misc
+++ /dev/null
@@ -1 +0,0 @@
-Adjust the sytest blacklist for worker mode.
diff --git a/changelog.d/6541.doc b/changelog.d/6541.doc
deleted file mode 100644
index c20029edc0..0000000000
--- a/changelog.d/6541.doc
+++ /dev/null
@@ -1 +0,0 @@
-Document the Room Shutdown Admin API.
\ No newline at end of file
diff --git a/changelog.d/6546.feature b/changelog.d/6546.feature
deleted file mode 100644
index 954aacb0d0..0000000000
--- a/changelog.d/6546.feature
+++ /dev/null
@@ -1 +0,0 @@
-Add an export_signing_key script to extract the public part of signing keys when rotating them.
diff --git a/changelog.d/6555.bugfix b/changelog.d/6555.bugfix
deleted file mode 100644
index 86a5a56cf6..0000000000
--- a/changelog.d/6555.bugfix
+++ /dev/null
@@ -1 +0,0 @@
-Fix missing row in device_max_stream_id that could cause unable to decrypt errors after server restart.
\ No newline at end of file
diff --git a/changelog.d/6557.misc b/changelog.d/6557.misc
deleted file mode 100644
index 80e7eaedb8..0000000000
--- a/changelog.d/6557.misc
+++ /dev/null
@@ -1 +0,0 @@
-Remove unused `get_pagination_rows` methods from `EventSource` classes.
diff --git a/changelog.d/6558.misc b/changelog.d/6558.misc
deleted file mode 100644
index a7572f1a85..0000000000
--- a/changelog.d/6558.misc
+++ /dev/null
@@ -1 +0,0 @@
-Clean up logs from the push notifier at startup.
\ No newline at end of file
diff --git a/changelog.d/6559.misc b/changelog.d/6559.misc
deleted file mode 100644
index 8bca37457d..0000000000
--- a/changelog.d/6559.misc
+++ /dev/null
@@ -1 +0,0 @@
-Port `synapse.handlers.admin` and `synapse.handlers.deactivate_account` to async/await.
diff --git a/changelog.d/6563.bugfix b/changelog.d/6563.bugfix
new file mode 100644
index 0000000000..3325fb1dcf
--- /dev/null
+++ b/changelog.d/6563.bugfix
@@ -0,0 +1 @@
+Fix GET request on /_synapse/admin/v2/users endpoint. Contributed by Awesome Technologies Innovationslabor GmbH.
\ No newline at end of file
diff --git a/changelog.d/6564.misc b/changelog.d/6564.misc
deleted file mode 100644
index f644f5868b..0000000000
--- a/changelog.d/6564.misc
+++ /dev/null
@@ -1 +0,0 @@
-Change `EventContext` to use the `Storage` class, in preparation for moving state database queries to a separate data store.
diff --git a/changelog.d/6565.misc b/changelog.d/6565.misc
deleted file mode 100644
index e83f245bf0..0000000000
--- a/changelog.d/6565.misc
+++ /dev/null
@@ -1 +0,0 @@
-Add assertion that schema delta file names are unique.
diff --git a/changelog.d/6570.misc b/changelog.d/6570.misc
deleted file mode 100644
index e89955a51e..0000000000
--- a/changelog.d/6570.misc
+++ /dev/null
@@ -1 +0,0 @@
-Improve diagnostics on database upgrade failure.
diff --git a/changelog.d/6571.bugfix b/changelog.d/6571.bugfix
deleted file mode 100644
index e38ea7b4f7..0000000000
--- a/changelog.d/6571.bugfix
+++ /dev/null
@@ -1 +0,0 @@
-Fix a bug which meant that we did not send systemd notifications on startup if acme was enabled.
diff --git a/changelog.d/6580.feature b/changelog.d/6580.feature
deleted file mode 100644
index 233c589c66..0000000000
--- a/changelog.d/6580.feature
+++ /dev/null
@@ -1 +0,0 @@
-Add experimental config option to specify multiple databases.
diff --git a/changelog.d/6601.doc b/changelog.d/6601.doc
deleted file mode 100644
index 08c5b3d215..0000000000
--- a/changelog.d/6601.doc
+++ /dev/null
@@ -1 +0,0 @@
-Reword sections of federate.md that explained delegation at time of Synapse 1.0 transition.
\ No newline at end of file
diff --git a/changelog.d/6614.doc b/changelog.d/6614.doc
deleted file mode 100644
index 38b962b062..0000000000
--- a/changelog.d/6614.doc
+++ /dev/null
@@ -1 +0,0 @@
-Added the section 'Configuration' in /docs/turn-howto.md.
diff --git a/changelog.d/6617.misc b/changelog.d/6617.misc
deleted file mode 100644
index 94aa271d38..0000000000
--- a/changelog.d/6617.misc
+++ /dev/null
@@ -1 +0,0 @@
-Reduce the reconnect time when worker replication fails, to make it easier to catch up.
diff --git a/changelog.d/6619.misc b/changelog.d/6619.misc
deleted file mode 100644
index b608133219..0000000000
--- a/changelog.d/6619.misc
+++ /dev/null
@@ -1 +0,0 @@
-Simplify http handling by removing redundant SynapseRequestFactory.
diff --git a/changelog.d/6620.misc b/changelog.d/6620.misc
deleted file mode 100644
index 8bfb78fb20..0000000000
--- a/changelog.d/6620.misc
+++ /dev/null
@@ -1 +0,0 @@
-Add a workaround for synapse raising exceptions when fetching the notary's own key from the notary.
diff --git a/changelog.d/6625.bugfix b/changelog.d/6625.bugfix
deleted file mode 100644
index a8dc5587dc..0000000000
--- a/changelog.d/6625.bugfix
+++ /dev/null
@@ -1 +0,0 @@
-Fix exception when fetching the `matrix.org:ed25519:auto` key.
diff --git a/changelog.d/6626.feature b/changelog.d/6626.feature
deleted file mode 100644
index 15798fa59b..0000000000
--- a/changelog.d/6626.feature
+++ /dev/null
@@ -1 +0,0 @@
-Raise an error if someone tries to use the log_file config option.
diff --git a/changelog.d/6627.misc b/changelog.d/6627.misc
deleted file mode 100644
index 702f067070..0000000000
--- a/changelog.d/6627.misc
+++ /dev/null
@@ -1 +0,0 @@
-Automate generation of the sample log config.
diff --git a/changelog.d/6628.removal b/changelog.d/6628.removal
deleted file mode 100644
index 66cd6aeca4..0000000000
--- a/changelog.d/6628.removal
+++ /dev/null
@@ -1 +0,0 @@
-Remove unused, undocumented /_matrix/content API.
diff --git a/changelog.d/6629.misc b/changelog.d/6629.misc
deleted file mode 100644
index 68f77af05b..0000000000
--- a/changelog.d/6629.misc
+++ /dev/null
@@ -1 +0,0 @@
-Simplify event creation code by removing redundant queries on the event_reference_hashes table.
\ No newline at end of file
diff --git a/changelog.d/6633.bugfix b/changelog.d/6633.bugfix
deleted file mode 100644
index 4bacf26021..0000000000
--- a/changelog.d/6633.bugfix
+++ /dev/null
@@ -1 +0,0 @@
-Fix bug where a moderator upgraded a room and became an admin in the new room.
\ No newline at end of file
diff --git a/changelog.d/6640.bugfix b/changelog.d/6640.bugfix
deleted file mode 100644
index 8c2a129933..0000000000
--- a/changelog.d/6640.bugfix
+++ /dev/null
@@ -1 +0,0 @@
-Fix an error which was thrown by the PresenceHandler _on_shutdown handler.
diff --git a/changelog.d/6642.misc b/changelog.d/6642.misc
deleted file mode 100644
index a480bbd134..0000000000
--- a/changelog.d/6642.misc
+++ /dev/null
@@ -1 +0,0 @@
-Fix errors when frozen_dicts are enabled.
diff --git a/changelog.d/6645.bugfix b/changelog.d/6645.bugfix
deleted file mode 100644
index f648df3fc0..0000000000
--- a/changelog.d/6645.bugfix
+++ /dev/null
@@ -1 +0,0 @@
-Fix exceptions in the synchrotron worker log when events are rejected.
diff --git a/changelog.d/6647.misc b/changelog.d/6647.misc
deleted file mode 100644
index fbe7c0e7db..0000000000
--- a/changelog.d/6647.misc
+++ /dev/null
@@ -1 +0,0 @@
-Port core background update routines to async/await.
diff --git a/changelog.d/6648.bugfix b/changelog.d/6648.bugfix
deleted file mode 100644
index 39916de437..0000000000
--- a/changelog.d/6648.bugfix
+++ /dev/null
@@ -1 +0,0 @@
-Ensure that upgraded rooms are removed from the directory.
diff --git a/changelog.d/6652.bugfix b/changelog.d/6652.bugfix
deleted file mode 100644
index 7e9781d652..0000000000
--- a/changelog.d/6652.bugfix
+++ /dev/null
@@ -1 +0,0 @@
-Fix a bug causing Synapse not to fetch missing events when it believes it has every event in the room.
diff --git a/changelog.d/6653.misc b/changelog.d/6653.misc
deleted file mode 100644
index fbe7c0e7db..0000000000
--- a/changelog.d/6653.misc
+++ /dev/null
@@ -1 +0,0 @@
-Port core background update routines to async/await.
diff --git a/changelog.d/6657.bugfix b/changelog.d/6657.bugfix
new file mode 100644
index 0000000000..94e51a9896
--- /dev/null
+++ b/changelog.d/6657.bugfix
@@ -0,0 +1 @@
+Fix incorrect signing of responses from the key server implementation.
\ No newline at end of file
diff --git a/synapse/__init__.py b/synapse/__init__.py
index 71cb611820..a3bd855045 100644
--- a/synapse/__init__.py
+++ b/synapse/__init__.py
@@ -36,7 +36,7 @@ try:
 except ImportError:
     pass
 
-__version__ = "1.7.3"
+__version__ = "1.8.0rc1"
 
 if bool(os.environ.get("SYNAPSE_TEST_PATCH_LOG_CONTEXTS", False)):
     # We import here so that we don't have to install a bunch of deps when
diff --git a/synapse/rest/key/v2/remote_key_resource.py b/synapse/rest/key/v2/remote_key_resource.py
index bf5e0eb844..e7fc3f0431 100644
--- a/synapse/rest/key/v2/remote_key_resource.py
+++ b/synapse/rest/key/v2/remote_key_resource.py
@@ -15,7 +15,6 @@
 import logging
 
 from canonicaljson import encode_canonical_json, json
-from signedjson.key import encode_verify_key_base64
 from signedjson.sign import sign_json
 
 from twisted.internet import defer
@@ -217,28 +216,15 @@ class RemoteKey(DirectServeResource):
         if cache_misses and query_remote_on_cache_miss:
             yield self.fetcher.get_keys(cache_misses)
             yield self.query_keys(request, query, query_remote_on_cache_miss=False)
-            return
-
-        signed_keys = []
-        for key_json in json_results:
-            key_json = json.loads(key_json)
-
-            # backwards-compatibility hack for #6596: if the requested key belongs
-            # to us, make sure that all of the signing keys appear in the
-            # "verify_keys" section.
-            if key_json["server_name"] == self.config.server_name:
-                verify_keys = key_json["verify_keys"]
+        else:
+            signed_keys = []
+            for key_json in json_results:
+                key_json = json.loads(key_json)
                 for signing_key in self.config.key_server_signing_keys:
-                    key_id = "%s:%s" % (signing_key.alg, signing_key.version)
-                    verify_keys[key_id] = {
-                        "key": encode_verify_key_base64(signing_key.verify_key)
-                    }
-
-            for signing_key in self.config.key_server_signing_keys:
-                key_json = sign_json(key_json, self.config.server_name, signing_key)
+                    key_json = sign_json(key_json, self.config.server_name, signing_key)
 
-            signed_keys.append(key_json)
+                signed_keys.append(key_json)
 
-        results = {"server_keys": signed_keys}
+            results = {"server_keys": signed_keys}
 
-        respond_with_json_bytes(request, 200, encode_canonical_json(results))
+            respond_with_json_bytes(request, 200, encode_canonical_json(results))
diff --git a/synapse/storage/data_stores/main/__init__.py b/synapse/storage/data_stores/main/__init__.py
index c577c0df5f..2700cca822 100644
--- a/synapse/storage/data_stores/main/__init__.py
+++ b/synapse/storage/data_stores/main/__init__.py
@@ -526,9 +526,9 @@ class DataStore(
 
         attr_filter = {}
         if not guests:
-            attr_filter["is_guest"] = False
+            attr_filter["is_guest"] = 0
         if not deactivated:
-            attr_filter["deactivated"] = False
+            attr_filter["deactivated"] = 0
 
         return self.db.simple_select_list_paginate(
             desc="get_users_paginate",
diff --git a/tests/rest/admin/test_admin.py b/tests/rest/admin/test_admin.py
index 0ed2594381..325bd6a608 100644
--- a/tests/rest/admin/test_admin.py
+++ b/tests/rest/admin/test_admin.py
@@ -341,6 +341,47 @@ class UserRegisterTestCase(unittest.HomeserverTestCase):
         self.assertEqual("Invalid user type", channel.json_body["error"])
 
 
+class UsersListTestCase(unittest.HomeserverTestCase):
+
+    servlets = [
+        synapse.rest.admin.register_servlets,
+        login.register_servlets,
+    ]
+    url = "/_synapse/admin/v2/users"
+
+    def prepare(self, reactor, clock, hs):
+        self.admin_user = self.register_user("admin", "pass", admin=True)
+        self.admin_user_tok = self.login("admin", "pass")
+
+        self.register_user("user1", "pass1", admin=False)
+        self.register_user("user2", "pass2", admin=False)
+
+    def test_no_auth(self):
+        """
+        Try to list users without authentication.
+        """
+        request, channel = self.make_request("GET", self.url, b"{}")
+        self.render(request)
+
+        self.assertEqual(401, int(channel.result["code"]), msg=channel.result["body"])
+        self.assertEqual("M_MISSING_TOKEN", channel.json_body["errcode"])
+
+    def test_all_users(self):
+        """
+        List all users, including deactivated users.
+        """
+        request, channel = self.make_request(
+            "GET",
+            self.url + "?deactivated=true",
+            b"{}",
+            access_token=self.admin_user_tok,
+        )
+        self.render(request)
+
+        self.assertEqual(200, int(channel.result["code"]), msg=channel.result["body"])
+        self.assertEqual(3, len(channel.json_body["users"]))
+
+
 class ShutdownRoomTestCase(unittest.HomeserverTestCase):
     servlets = [
         synapse.rest.admin.register_servlets_for_client_rest_resource,
diff --git a/tests/rest/key/__init__.py b/tests/rest/key/__init__.py
new file mode 100644
index 0000000000..e69de29bb2
--- /dev/null
+++ b/tests/rest/key/__init__.py
diff --git a/tests/rest/key/v2/__init__.py b/tests/rest/key/v2/__init__.py
new file mode 100644
index 0000000000..e69de29bb2
--- /dev/null
+++ b/tests/rest/key/v2/__init__.py
diff --git a/tests/rest/key/v2/test_remote_key_resource.py b/tests/rest/key/v2/test_remote_key_resource.py
index d8246b4e78..6776a56cad 100644
--- a/tests/rest/key/v2/test_remote_key_resource.py
+++ b/tests/rest/key/v2/test_remote_key_resource.py
@@ -13,25 +13,30 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 import urllib.parse
-from io import BytesIO
+from io import BytesIO, StringIO
 
 from mock import Mock
 
 import signedjson.key
+from canonicaljson import encode_canonical_json
 from nacl.signing import SigningKey
 from signedjson.sign import sign_json
 
 from twisted.web.resource import NoResource
 
+from synapse.crypto.keyring import PerspectivesKeyFetcher
 from synapse.http.site import SynapseRequest
 from synapse.rest.key.v2 import KeyApiV2Resource
+from synapse.storage.keys import FetchKeyResult
 from synapse.util.httpresourcetree import create_resource_tree
+from synapse.util.stringutils import random_string
 
 from tests import unittest
 from tests.server import FakeChannel, wait_until_result
+from tests.utils import default_config
 
 
-class RemoteKeyResourceTestCase(unittest.HomeserverTestCase):
+class BaseRemoteKeyResourceTestCase(unittest.HomeserverTestCase):
     def make_homeserver(self, reactor, clock):
         self.http_client = Mock()
         return self.setup_test_homeserver(http_client=self.http_client)
@@ -73,6 +78,8 @@ class RemoteKeyResourceTestCase(unittest.HomeserverTestCase):
 
         self.http_client.get_json.side_effect = get_json
 
+
+class RemoteKeyResourceTestCase(BaseRemoteKeyResourceTestCase):
     def make_notary_request(self, server_name: str, key_id: str) -> dict:
         """Send a GET request to the test server requesting the given key.
 
@@ -125,6 +132,126 @@ class RemoteKeyResourceTestCase(unittest.HomeserverTestCase):
         oursigs = sigs[self.hs.hostname]
         self.assertEqual(len(oursigs), 2)
 
-        # and both keys should be present in the verify_keys section
+        # the requested key should be present in the verify_keys section
         self.assertIn("ed25519:ver1", keys[0]["verify_keys"])
-        self.assertIn("ed25519:a_lPym", keys[0]["verify_keys"])
+
+
+class EndToEndPerspectivesTests(BaseRemoteKeyResourceTestCase):
+    """End-to-end tests of the perspectives fetch case
+
+    The idea here is to actually wire up a PerspectivesKeyFetcher to the notary
+    endpoint, to check that the two implementations are compatible.
+    """
+
+    def default_config(self, *args, **kwargs):
+        config = super().default_config(*args, **kwargs)
+
+        # replace the signing key with our own
+        self.hs_signing_key = signedjson.key.generate_signing_key("kssk")
+        strm = StringIO()
+        signedjson.key.write_signing_keys(strm, [self.hs_signing_key])
+        config["signing_key"] = strm.getvalue()
+
+        return config
+
+    def prepare(self, reactor, clock, homeserver):
+        # make a second homeserver, configured to use the first one as a key notary
+        self.http_client2 = Mock()
+        config = default_config(name="keyclient")
+        config["trusted_key_servers"] = [
+            {
+                "server_name": self.hs.hostname,
+                "verify_keys": {
+                    "ed25519:%s"
+                    % (
+                        self.hs_signing_key.version,
+                    ): signedjson.key.encode_verify_key_base64(
+                        self.hs_signing_key.verify_key
+                    )
+                },
+            }
+        ]
+        self.hs2 = self.setup_test_homeserver(
+            http_client=self.http_client2, config=config
+        )
+
+        # wire up outbound POST /key/v2/query requests from hs2 so that they
+        # will be forwarded to hs1
+        def post_json(destination, path, data):
+            self.assertEqual(destination, self.hs.hostname)
+            self.assertEqual(
+                path, "/_matrix/key/v2/query",
+            )
+
+            channel = FakeChannel(self.site, self.reactor)
+            req = SynapseRequest(channel)
+            req.content = BytesIO(encode_canonical_json(data))
+
+            req.requestReceived(
+                b"POST", path.encode("utf-8"), b"1.1",
+            )
+            wait_until_result(self.reactor, req)
+            self.assertEqual(channel.code, 200)
+            resp = channel.json_body
+            return resp
+
+        self.http_client2.post_json.side_effect = post_json
+
+    def test_get_key(self):
+        """Fetch a key belonging to a random server"""
+        # make up a key to be fetched.
+        testkey = signedjson.key.generate_signing_key("abc")
+
+        # we expect hs1 to make a regular key request to the target server
+        self.expect_outgoing_key_request("targetserver", testkey)
+        keyid = "ed25519:%s" % (testkey.version,)
+
+        fetcher = PerspectivesKeyFetcher(self.hs2)
+        d = fetcher.get_keys({"targetserver": {keyid: 1000}})
+        res = self.get_success(d)
+        self.assertIn("targetserver", res)
+        keyres = res["targetserver"][keyid]
+        assert isinstance(keyres, FetchKeyResult)
+        self.assertEqual(
+            signedjson.key.encode_verify_key_base64(keyres.verify_key),
+            signedjson.key.encode_verify_key_base64(testkey.verify_key),
+        )
+
+    def test_get_notary_key(self):
+        """Fetch a key belonging to the notary server"""
+        # make up a key to be fetched. We randomise the keyid to try to get it to
+        # appear before the key server signing key sometimes (otherwise we bail out
+        # before fetching its signature)
+        testkey = signedjson.key.generate_signing_key(random_string(5))
+
+        # we expect hs1 to make a regular key request to itself
+        self.expect_outgoing_key_request(self.hs.hostname, testkey)
+        keyid = "ed25519:%s" % (testkey.version,)
+
+        fetcher = PerspectivesKeyFetcher(self.hs2)
+        d = fetcher.get_keys({self.hs.hostname: {keyid: 1000}})
+        res = self.get_success(d)
+        self.assertIn(self.hs.hostname, res)
+        keyres = res[self.hs.hostname][keyid]
+        assert isinstance(keyres, FetchKeyResult)
+        self.assertEqual(
+            signedjson.key.encode_verify_key_base64(keyres.verify_key),
+            signedjson.key.encode_verify_key_base64(testkey.verify_key),
+        )
+
+    def test_get_notary_keyserver_key(self):
+        """Fetch the notary's keyserver key"""
+        # we expect hs1 to make a regular key request to itself
+        self.expect_outgoing_key_request(self.hs.hostname, self.hs_signing_key)
+        keyid = "ed25519:%s" % (self.hs_signing_key.version,)
+
+        fetcher = PerspectivesKeyFetcher(self.hs2)
+        d = fetcher.get_keys({self.hs.hostname: {keyid: 1000}})
+        res = self.get_success(d)
+        self.assertIn(self.hs.hostname, res)
+        keyres = res[self.hs.hostname][keyid]
+        assert isinstance(keyres, FetchKeyResult)
+        self.assertEqual(
+            signedjson.key.encode_verify_key_base64(keyres.verify_key),
+            signedjson.key.encode_verify_key_base64(self.hs_signing_key.verify_key),
+        )