1.8.0rc1

56 files changed, 81 insertions, 55 deletions

diff --git a/CHANGES.md b/CHANGES.md
index 361fd1fc6c..2f1cd87e1a 100644
--- a/CHANGES.md
+++ b/CHANGES.md
@@ -1,3 +1,83 @@
+Synapse 1.8.0rc1 (2020-01-07)
+=============================
+
+Features
+--------
+
+- Implement v2 APIs for the `send_join` and `send_leave` federation endpoints (as described in [MSC1802](https://github.com/matrix-org/matrix-doc/pull/1802)). ([\#6349](https://github.com/matrix-org/synapse/issues/6349))
+- Add a develop script to generate full SQL schemas. ([\#6394](https://github.com/matrix-org/synapse/issues/6394))
+- Allow custom SAML username mapping functinality through an external provider plugin. ([\#6411](https://github.com/matrix-org/synapse/issues/6411))
+- Automatically delete empty groups/communities. ([\#6453](https://github.com/matrix-org/synapse/issues/6453))
+- Add option `limit_profile_requests_to_users_who_share_rooms` to prevent requirement of a local user sharing a room with another user to query their profile information. ([\#6523](https://github.com/matrix-org/synapse/issues/6523))
+- Add an export_signing_key script to extract the public part of signing keys when rotating them. ([\#6546](https://github.com/matrix-org/synapse/issues/6546))
+- Add experimental config option to specify multiple databases. ([\#6580](https://github.com/matrix-org/synapse/issues/6580))
+- Raise an error if someone tries to use the log_file config option. ([\#6626](https://github.com/matrix-org/synapse/issues/6626))
+
+
+Bugfixes
+--------
+
+- Prevent redacted events from being returned during message search. ([\#6377](https://github.com/matrix-org/synapse/issues/6377), [\#6522](https://github.com/matrix-org/synapse/issues/6522))
+- Prevent error on trying to search a upgraded room when the server is not in the predecessor room. ([\#6385](https://github.com/matrix-org/synapse/issues/6385))
+- Improve performance of looking up cross-signing keys. ([\#6486](https://github.com/matrix-org/synapse/issues/6486))
+- Fix race which occasionally caused deleted devices to reappear. ([\#6514](https://github.com/matrix-org/synapse/issues/6514))
+- Fix missing row in device_max_stream_id that could cause unable to decrypt errors after server restart. ([\#6555](https://github.com/matrix-org/synapse/issues/6555))
+- Fix a bug which meant that we did not send systemd notifications on startup if acme was enabled. ([\#6571](https://github.com/matrix-org/synapse/issues/6571))
+- Fix exception when fetching the `matrix.org:ed25519:auto` key. ([\#6625](https://github.com/matrix-org/synapse/issues/6625))
+- Fix bug where a moderator upgraded a room and became an admin in the new room. ([\#6633](https://github.com/matrix-org/synapse/issues/6633))
+- Fix an error which was thrown by the PresenceHandler _on_shutdown handler. ([\#6640](https://github.com/matrix-org/synapse/issues/6640))
+- Fix exceptions in the synchrotron worker log when events are rejected. ([\#6645](https://github.com/matrix-org/synapse/issues/6645))
+- Ensure that upgraded rooms are removed from the directory. ([\#6648](https://github.com/matrix-org/synapse/issues/6648))
+- Fix a bug causing Synapse not to fetch missing events when it believes it has every event in the room. ([\#6652](https://github.com/matrix-org/synapse/issues/6652))
+
+
+Improved Documentation
+----------------------
+
+- Document the Room Shutdown Admin API. ([\#6541](https://github.com/matrix-org/synapse/issues/6541))
+- Reword sections of federate.md that explained delegation at time of Synapse 1.0 transition. ([\#6601](https://github.com/matrix-org/synapse/issues/6601))
+- Added the section 'Configuration' in /docs/turn-howto.md. ([\#6614](https://github.com/matrix-org/synapse/issues/6614))
+
+
+Deprecations and Removals
+-------------------------
+
+- Remove redundant code from event authorisation implementation. ([\#6502](https://github.com/matrix-org/synapse/issues/6502))
+- Remove unused, undocumented /_matrix/content API. ([\#6628](https://github.com/matrix-org/synapse/issues/6628))
+
+
+Internal Changes
+----------------
+
+- Split out state storage into separate data store. ([\#6245](https://github.com/matrix-org/synapse/issues/6245))
+- Port synapse.handlers.initial_sync to async/await. ([\#6496](https://github.com/matrix-org/synapse/issues/6496))
+- Port handlers.account_data and handlers.account_validity to async/await. ([\#6504](https://github.com/matrix-org/synapse/issues/6504))
+- Make `make_deferred_yieldable` to work with async/await. ([\#6505](https://github.com/matrix-org/synapse/issues/6505))
+- Remove `SnapshotCache` in favour of `ResponseCache`. ([\#6506](https://github.com/matrix-org/synapse/issues/6506))
+- Change phone home stats to not assume there is a single database and report information about the database used by the main data store. ([\#6510](https://github.com/matrix-org/synapse/issues/6510))
+- Move database config from apps into HomeServer object. ([\#6511](https://github.com/matrix-org/synapse/issues/6511))
+- Silence mypy errors for files outside those specified. ([\#6512](https://github.com/matrix-org/synapse/issues/6512))
+- Remove all assumptions of there being a single phyiscal DB apart from the `synapse.config`. ([\#6513](https://github.com/matrix-org/synapse/issues/6513))
+- Clean up some logging when handling incoming events over federation. ([\#6515](https://github.com/matrix-org/synapse/issues/6515))
+- Port some of FederationHandler to async/await. ([\#6517](https://github.com/matrix-org/synapse/issues/6517))
+- Test more folders against mypy. ([\#6534](https://github.com/matrix-org/synapse/issues/6534))
+- Update `mypy` to new version. ([\#6537](https://github.com/matrix-org/synapse/issues/6537))
+- Adjust the sytest blacklist for worker mode. ([\#6538](https://github.com/matrix-org/synapse/issues/6538))
+- Remove unused `get_pagination_rows` methods from `EventSource` classes. ([\#6557](https://github.com/matrix-org/synapse/issues/6557))
+- Clean up logs from the push notifier at startup. ([\#6558](https://github.com/matrix-org/synapse/issues/6558))
+- Port `synapse.handlers.admin` and `synapse.handlers.deactivate_account` to async/await. ([\#6559](https://github.com/matrix-org/synapse/issues/6559))
+- Change `EventContext` to use the `Storage` class, in preparation for moving state database queries to a separate data store. ([\#6564](https://github.com/matrix-org/synapse/issues/6564))
+- Add assertion that schema delta file names are unique. ([\#6565](https://github.com/matrix-org/synapse/issues/6565))
+- Improve diagnostics on database upgrade failure. ([\#6570](https://github.com/matrix-org/synapse/issues/6570))
+- Reduce the reconnect time when worker replication fails, to make it easier to catch up. ([\#6617](https://github.com/matrix-org/synapse/issues/6617))
+- Simplify http handling by removing redundant SynapseRequestFactory. ([\#6619](https://github.com/matrix-org/synapse/issues/6619))
+- Add a workaround for synapse raising exceptions when fetching the notary's own key from the notary. ([\#6620](https://github.com/matrix-org/synapse/issues/6620))
+- Automate generation of the sample log config. ([\#6627](https://github.com/matrix-org/synapse/issues/6627))
+- Simplify event creation code by removing redundant queries on the event_reference_hashes table. ([\#6629](https://github.com/matrix-org/synapse/issues/6629))
+- Fix errors when frozen_dicts are enabled. ([\#6642](https://github.com/matrix-org/synapse/issues/6642))
+- Port core background update routines to async/await. ([\#6647](https://github.com/matrix-org/synapse/issues/6647), [\#6653](https://github.com/matrix-org/synapse/issues/6653))
+
+
 Synapse 1.7.3 (2019-12-31)
 ==========================
 
diff --git a/changelog.d/6245.misc b/changelog.d/6245.misc
deleted file mode 100644
index a3e6b8296e..0000000000
--- a/changelog.d/6245.misc
+++ /dev/null
@@ -1 +0,0 @@
-Split out state storage into separate data store.
diff --git a/changelog.d/6349.feature b/changelog.d/6349.feature
deleted file mode 100644
index 56c4fbf78e..0000000000
--- a/changelog.d/6349.feature
+++ /dev/null
@@ -1 +0,0 @@
-Implement v2 APIs for the `send_join` and `send_leave` federation endpoints (as described in [MSC1802](https://github.com/matrix-org/matrix-doc/pull/1802)).
diff --git a/changelog.d/6377.bugfix b/changelog.d/6377.bugfix
deleted file mode 100644
index ccda96962f..0000000000
--- a/changelog.d/6377.bugfix
+++ /dev/null
@@ -1 +0,0 @@
-Prevent redacted events from being returned during message search.
\ No newline at end of file
diff --git a/changelog.d/6385.bugfix b/changelog.d/6385.bugfix
deleted file mode 100644
index 7a2bc02170..0000000000
--- a/changelog.d/6385.bugfix
+++ /dev/null
@@ -1 +0,0 @@
-Prevent error on trying to search a upgraded room when the server is not in the predecessor room.
\ No newline at end of file
diff --git a/changelog.d/6394.feature b/changelog.d/6394.feature
deleted file mode 100644
index 1a0e8845ad..0000000000
--- a/changelog.d/6394.feature
+++ /dev/null
@@ -1 +0,0 @@
-Add a develop script to generate full SQL schemas.
\ No newline at end of file
diff --git a/changelog.d/6411.feature b/changelog.d/6411.feature
deleted file mode 100644
index ebea4a208d..0000000000
--- a/changelog.d/6411.feature
+++ /dev/null
@@ -1 +0,0 @@
-Allow custom SAML username mapping functinality through an external provider plugin.
\ No newline at end of file
diff --git a/changelog.d/6453.feature b/changelog.d/6453.feature
deleted file mode 100644
index e7bb801c6a..0000000000
--- a/changelog.d/6453.feature
+++ /dev/null
@@ -1 +0,0 @@
-Automatically delete empty groups/communities.
diff --git a/changelog.d/6486.bugfix b/changelog.d/6486.bugfix
deleted file mode 100644
index b98c5a9ae5..0000000000
--- a/changelog.d/6486.bugfix
+++ /dev/null
@@ -1 +0,0 @@
-Improve performance of looking up cross-signing keys.
diff --git a/changelog.d/6496.misc b/changelog.d/6496.misc
deleted file mode 100644
index 19c6e926b8..0000000000
--- a/changelog.d/6496.misc
+++ /dev/null
@@ -1 +0,0 @@
-Port synapse.handlers.initial_sync to async/await.
diff --git a/changelog.d/6502.removal b/changelog.d/6502.removal
deleted file mode 100644
index 0b72261d58..0000000000
--- a/changelog.d/6502.removal
+++ /dev/null
@@ -1 +0,0 @@
-Remove redundant code from event authorisation implementation.
diff --git a/changelog.d/6504.misc b/changelog.d/6504.misc
deleted file mode 100644
index 7c873459af..0000000000
--- a/changelog.d/6504.misc
+++ /dev/null
@@ -1 +0,0 @@
-Port handlers.account_data and handlers.account_validity to async/await.
diff --git a/changelog.d/6505.misc b/changelog.d/6505.misc
deleted file mode 100644
index 3a75b2d9dd..0000000000
--- a/changelog.d/6505.misc
+++ /dev/null
@@ -1 +0,0 @@
-Make `make_deferred_yieldable` to work with async/await.
diff --git a/changelog.d/6506.misc b/changelog.d/6506.misc
deleted file mode 100644
index 99d7a70bcf..0000000000
--- a/changelog.d/6506.misc
+++ /dev/null
@@ -1 +0,0 @@
-Remove `SnapshotCache` in favour of `ResponseCache`.
diff --git a/changelog.d/6510.misc b/changelog.d/6510.misc
deleted file mode 100644
index 214f06539b..0000000000
--- a/changelog.d/6510.misc
+++ /dev/null
@@ -1 +0,0 @@
-Change phone home stats to not assume there is a single database and report information about the database used by the main data store.
diff --git a/changelog.d/6511.misc b/changelog.d/6511.misc
deleted file mode 100644
index 19ce435e68..0000000000
--- a/changelog.d/6511.misc
+++ /dev/null
@@ -1 +0,0 @@
-Move database config from apps into HomeServer object.
diff --git a/changelog.d/6512.misc b/changelog.d/6512.misc
deleted file mode 100644
index 37a8099eec..0000000000
--- a/changelog.d/6512.misc
+++ /dev/null
@@ -1 +0,0 @@
-Silence mypy errors for files outside those specified.
diff --git a/changelog.d/6513.misc b/changelog.d/6513.misc
deleted file mode 100644
index 36700f5657..0000000000
--- a/changelog.d/6513.misc
+++ /dev/null
@@ -1 +0,0 @@
-Remove all assumptions of there being a single phyiscal DB apart from the `synapse.config`.
diff --git a/changelog.d/6514.bugfix b/changelog.d/6514.bugfix
deleted file mode 100644
index 6dc1985c24..0000000000
--- a/changelog.d/6514.bugfix
+++ /dev/null
@@ -1 +0,0 @@
-Fix race which occasionally caused deleted devices to reappear.
diff --git a/changelog.d/6515.misc b/changelog.d/6515.misc
deleted file mode 100644
index a9c303ed1c..0000000000
--- a/changelog.d/6515.misc
+++ /dev/null
@@ -1 +0,0 @@
-Clean up some logging when handling incoming events over federation.
diff --git a/changelog.d/6517.misc b/changelog.d/6517.misc
deleted file mode 100644
index c6ffed9952..0000000000
--- a/changelog.d/6517.misc
+++ /dev/null
@@ -1 +0,0 @@
-Port some of FederationHandler to async/await.
\ No newline at end of file
diff --git a/changelog.d/6522.bugfix b/changelog.d/6522.bugfix
deleted file mode 100644
index ccda96962f..0000000000
--- a/changelog.d/6522.bugfix
+++ /dev/null
@@ -1 +0,0 @@
-Prevent redacted events from being returned during message search.
\ No newline at end of file
diff --git a/changelog.d/6523.feature b/changelog.d/6523.feature
deleted file mode 100644
index 798fa143df..0000000000
--- a/changelog.d/6523.feature
+++ /dev/null
@@ -1 +0,0 @@
-Add option `limit_profile_requests_to_users_who_share_rooms` to prevent requirement of a local user sharing a room with another user to query their profile information.
diff --git a/changelog.d/6534.misc b/changelog.d/6534.misc
deleted file mode 100644
index 7df6bb442a..0000000000
--- a/changelog.d/6534.misc
+++ /dev/null
@@ -1 +0,0 @@
-Test more folders against mypy.
diff --git a/changelog.d/6537.misc b/changelog.d/6537.misc
deleted file mode 100644
index 3543153584..0000000000
--- a/changelog.d/6537.misc
+++ /dev/null
@@ -1 +0,0 @@
-Update `mypy` to new version.
diff --git a/changelog.d/6538.misc b/changelog.d/6538.misc
deleted file mode 100644
index cb4fd56948..0000000000
--- a/changelog.d/6538.misc
+++ /dev/null
@@ -1 +0,0 @@
-Adjust the sytest blacklist for worker mode.
diff --git a/changelog.d/6541.doc b/changelog.d/6541.doc
deleted file mode 100644
index c20029edc0..0000000000
--- a/changelog.d/6541.doc
+++ /dev/null
@@ -1 +0,0 @@
-Document the Room Shutdown Admin API.
\ No newline at end of file
diff --git a/changelog.d/6546.feature b/changelog.d/6546.feature
deleted file mode 100644
index 954aacb0d0..0000000000
--- a/changelog.d/6546.feature
+++ /dev/null
@@ -1 +0,0 @@
-Add an export_signing_key script to extract the public part of signing keys when rotating them.
diff --git a/changelog.d/6555.bugfix b/changelog.d/6555.bugfix
deleted file mode 100644
index 86a5a56cf6..0000000000
--- a/changelog.d/6555.bugfix
+++ /dev/null
@@ -1 +0,0 @@
-Fix missing row in device_max_stream_id that could cause unable to decrypt errors after server restart.
\ No newline at end of file
diff --git a/changelog.d/6557.misc b/changelog.d/6557.misc
deleted file mode 100644
index 80e7eaedb8..0000000000
--- a/changelog.d/6557.misc
+++ /dev/null
@@ -1 +0,0 @@
-Remove unused `get_pagination_rows` methods from `EventSource` classes.
diff --git a/changelog.d/6558.misc b/changelog.d/6558.misc
deleted file mode 100644
index a7572f1a85..0000000000
--- a/changelog.d/6558.misc
+++ /dev/null
@@ -1 +0,0 @@
-Clean up logs from the push notifier at startup.
\ No newline at end of file
diff --git a/changelog.d/6559.misc b/changelog.d/6559.misc
deleted file mode 100644
index 8bca37457d..0000000000
--- a/changelog.d/6559.misc
+++ /dev/null
@@ -1 +0,0 @@
-Port `synapse.handlers.admin` and `synapse.handlers.deactivate_account` to async/await.
diff --git a/changelog.d/6564.misc b/changelog.d/6564.misc
deleted file mode 100644
index f644f5868b..0000000000
--- a/changelog.d/6564.misc
+++ /dev/null
@@ -1 +0,0 @@
-Change `EventContext` to use the `Storage` class, in preparation for moving state database queries to a separate data store.
diff --git a/changelog.d/6565.misc b/changelog.d/6565.misc
deleted file mode 100644
index e83f245bf0..0000000000
--- a/changelog.d/6565.misc
+++ /dev/null
@@ -1 +0,0 @@
-Add assertion that schema delta file names are unique.
diff --git a/changelog.d/6570.misc b/changelog.d/6570.misc
deleted file mode 100644
index e89955a51e..0000000000
--- a/changelog.d/6570.misc
+++ /dev/null
@@ -1 +0,0 @@
-Improve diagnostics on database upgrade failure.
diff --git a/changelog.d/6571.bugfix b/changelog.d/6571.bugfix
deleted file mode 100644
index e38ea7b4f7..0000000000
--- a/changelog.d/6571.bugfix
+++ /dev/null
@@ -1 +0,0 @@
-Fix a bug which meant that we did not send systemd notifications on startup if acme was enabled.
diff --git a/changelog.d/6580.feature b/changelog.d/6580.feature
deleted file mode 100644
index 233c589c66..0000000000
--- a/changelog.d/6580.feature
+++ /dev/null
@@ -1 +0,0 @@
-Add experimental config option to specify multiple databases.
diff --git a/changelog.d/6601.doc b/changelog.d/6601.doc
deleted file mode 100644
index 08c5b3d215..0000000000
--- a/changelog.d/6601.doc
+++ /dev/null
@@ -1 +0,0 @@
-Reword sections of federate.md that explained delegation at time of Synapse 1.0 transition.
\ No newline at end of file
diff --git a/changelog.d/6614.doc b/changelog.d/6614.doc
deleted file mode 100644
index 38b962b062..0000000000
--- a/changelog.d/6614.doc
+++ /dev/null
@@ -1 +0,0 @@
-Added the section 'Configuration' in /docs/turn-howto.md.
diff --git a/changelog.d/6617.misc b/changelog.d/6617.misc
deleted file mode 100644
index 94aa271d38..0000000000
--- a/changelog.d/6617.misc
+++ /dev/null
@@ -1 +0,0 @@
-Reduce the reconnect time when worker replication fails, to make it easier to catch up.
diff --git a/changelog.d/6619.misc b/changelog.d/6619.misc
deleted file mode 100644
index b608133219..0000000000
--- a/changelog.d/6619.misc
+++ /dev/null
@@ -1 +0,0 @@
-Simplify http handling by removing redundant SynapseRequestFactory.
diff --git a/changelog.d/6620.misc b/changelog.d/6620.misc
deleted file mode 100644
index 8bfb78fb20..0000000000
--- a/changelog.d/6620.misc
+++ /dev/null
@@ -1 +0,0 @@
-Add a workaround for synapse raising exceptions when fetching the notary's own key from the notary.
diff --git a/changelog.d/6625.bugfix b/changelog.d/6625.bugfix
deleted file mode 100644
index a8dc5587dc..0000000000
--- a/changelog.d/6625.bugfix
+++ /dev/null
@@ -1 +0,0 @@
-Fix exception when fetching the `matrix.org:ed25519:auto` key.
diff --git a/changelog.d/6626.feature b/changelog.d/6626.feature
deleted file mode 100644
index 15798fa59b..0000000000
--- a/changelog.d/6626.feature
+++ /dev/null
@@ -1 +0,0 @@
-Raise an error if someone tries to use the log_file config option.
diff --git a/changelog.d/6627.misc b/changelog.d/6627.misc
deleted file mode 100644
index 702f067070..0000000000
--- a/changelog.d/6627.misc
+++ /dev/null
@@ -1 +0,0 @@
-Automate generation of the sample log config.
diff --git a/changelog.d/6628.removal b/changelog.d/6628.removal
deleted file mode 100644
index 66cd6aeca4..0000000000
--- a/changelog.d/6628.removal
+++ /dev/null
@@ -1 +0,0 @@
-Remove unused, undocumented /_matrix/content API.
diff --git a/changelog.d/6629.misc b/changelog.d/6629.misc
deleted file mode 100644
index 68f77af05b..0000000000
--- a/changelog.d/6629.misc
+++ /dev/null
@@ -1 +0,0 @@
-Simplify event creation code by removing redundant queries on the event_reference_hashes table.
\ No newline at end of file
diff --git a/changelog.d/6633.bugfix b/changelog.d/6633.bugfix
deleted file mode 100644
index 4bacf26021..0000000000
--- a/changelog.d/6633.bugfix
+++ /dev/null
@@ -1 +0,0 @@
-Fix bug where a moderator upgraded a room and became an admin in the new room.
\ No newline at end of file
diff --git a/changelog.d/6640.bugfix b/changelog.d/6640.bugfix
deleted file mode 100644
index 8c2a129933..0000000000
--- a/changelog.d/6640.bugfix
+++ /dev/null
@@ -1 +0,0 @@
-Fix an error which was thrown by the PresenceHandler _on_shutdown handler.
diff --git a/changelog.d/6642.misc b/changelog.d/6642.misc
deleted file mode 100644
index a480bbd134..0000000000
--- a/changelog.d/6642.misc
+++ /dev/null
@@ -1 +0,0 @@
-Fix errors when frozen_dicts are enabled.
diff --git a/changelog.d/6645.bugfix b/changelog.d/6645.bugfix
deleted file mode 100644
index f648df3fc0..0000000000
--- a/changelog.d/6645.bugfix
+++ /dev/null
@@ -1 +0,0 @@
-Fix exceptions in the synchrotron worker log when events are rejected.
diff --git a/changelog.d/6647.misc b/changelog.d/6647.misc
deleted file mode 100644
index fbe7c0e7db..0000000000
--- a/changelog.d/6647.misc
+++ /dev/null
@@ -1 +0,0 @@
-Port core background update routines to async/await.
diff --git a/changelog.d/6648.bugfix b/changelog.d/6648.bugfix
deleted file mode 100644
index 39916de437..0000000000
--- a/changelog.d/6648.bugfix
+++ /dev/null
@@ -1 +0,0 @@
-Ensure that upgraded rooms are removed from the directory.
diff --git a/changelog.d/6652.bugfix b/changelog.d/6652.bugfix
deleted file mode 100644
index 7e9781d652..0000000000
--- a/changelog.d/6652.bugfix
+++ /dev/null
@@ -1 +0,0 @@
-Fix a bug causing Synapse not to fetch missing events when it believes it has every event in the room.
diff --git a/changelog.d/6653.misc b/changelog.d/6653.misc
deleted file mode 100644
index fbe7c0e7db..0000000000
--- a/changelog.d/6653.misc
+++ /dev/null
@@ -1 +0,0 @@
-Port core background update routines to async/await.
diff --git a/synapse/__init__.py b/synapse/__init__.py
index 71cb611820..a3bd855045 100644
--- a/synapse/__init__.py
+++ b/synapse/__init__.py
@@ -36,7 +36,7 @@ try:
 except ImportError:
     pass
 
-__version__ = "1.7.3"
+__version__ = "1.8.0rc1"
 
 if bool(os.environ.get("SYNAPSE_TEST_PATCH_LOG_CONTEXTS", False)):
     # We import here so that we don't have to install a bunch of deps when