Merge tag 'v1.42.0rc1' into develop

Synapse 1.42.0rc1 (2021-09-01)
==============================

Server administrators are reminded to read [the upgrade notes](docs/upgrade.md#upgrading-to-v1420).

Features
--------

- Add support for [MSC3231](https://github.com/matrix-org/matrix-doc/pull/3231): Token authenticated registration. Users can be required to submit a token during registration to authenticate themselves. Contributed by Callum Brown. ([\#10142](https://github.com/matrix-org/synapse/issues/10142))
- Add support for [MSC3283](https://github.com/matrix-org/matrix-doc/pull/3283): Expose `enable_set_displayname` in capabilities. ([\#10452](https://github.com/matrix-org/synapse/issues/10452))
- Port the `PresenceRouter` module interface to the new generic interface. ([\#10524](https://github.com/matrix-org/synapse/issues/10524))
- Add pagination to the spaces summary based on updates to [MSC2946](https://github.com/matrix-org/matrix-doc/pull/2946). ([\#10613](https://github.com/matrix-org/synapse/issues/10613), [\#10725](https://github.com/matrix-org/synapse/issues/10725))

Bugfixes
--------

- Validate new `m.room.power_levels` events. Contributed by @aaronraimist. ([\#10232](https://github.com/matrix-org/synapse/issues/10232))
- Display an error on User-Interactive Authentication fallback pages when authentication fails. Contributed by Callum Brown. ([\#10561](https://github.com/matrix-org/synapse/issues/10561))
- Remove pushers when deleting an e-mail address from an account. Pushers for old unlinked emails will also be deleted. ([\#10581](https://github.com/matrix-org/synapse/issues/10581), [\#10734](https://github.com/matrix-org/synapse/issues/10734))
- Reject Client-Server `/keys/query` requests which provide `device_ids` incorrectly. ([\#10593](https://github.com/matrix-org/synapse/issues/10593))
- Rooms with unsupported room versions are no longer returned via `/sync`. ([\#10644](https://github.com/matrix-org/synapse/issues/10644))
- Enforce the maximum length for per-room display names and avatar URLs. ([\#10654](https://github.com/matrix-org/synapse/issues/10654))
- Fix a bug which caused the `synapse_user_logins_total` Prometheus metric not to be correctly initialised on restart. ([\#10677](https://github.com/matrix-org/synapse/issues/10677))
- Improve `ServerNoticeServlet` to avoid duplicate requests and add unit tests. ([\#10679](https://github.com/matrix-org/synapse/issues/10679))
- Fix long-standing issue which caused an error when a thumbnail is requested and there are multiple thumbnails with the same quality rating. ([\#10684](https://github.com/matrix-org/synapse/issues/10684))
- Fix a regression introduced in v1.41.0 which affected the performance of concurrent fetches of large sets of events, in extreme cases causing the process to hang. ([\#10703](https://github.com/matrix-org/synapse/issues/10703))
- Fix a regression introduced in Synapse 1.41 which broke email transmission on Systems using older versions of the Twisted library. ([\#10713](https://github.com/matrix-org/synapse/issues/10713))

Improved Documentation
----------------------

- Add documentation on how to connect Django with Synapse using OpenID Connect and django-oauth-toolkit. Contributed by @HugoDelval. ([\#10192](https://github.com/matrix-org/synapse/issues/10192))
- Advertise https://matrix-org.github.io/synapse documentation in the `README` and `CONTRIBUTING` files. ([\#10595](https://github.com/matrix-org/synapse/issues/10595))
- Fix some of the titles not rendering in the OpenID Connect documentation. ([\#10639](https://github.com/matrix-org/synapse/issues/10639))
- Minor clarifications to the documentation for reverse proxies. ([\#10708](https://github.com/matrix-org/synapse/issues/10708))
- Remove table of contents from the top of installation and contributing documentation pages. ([\#10711](https://github.com/matrix-org/synapse/issues/10711))

Deprecations and Removals
-------------------------

- Remove deprecated Shutdown Room and Purge Room Admin API. ([\#8830](https://github.com/matrix-org/synapse/issues/8830))

Internal Changes
----------------

- Improve type hints for the proxy agent and SRV resolver modules. Contributed by @dklimpel. ([\#10608](https://github.com/matrix-org/synapse/issues/10608))
- Clean up some of the federation event authentication code for clarity. ([\#10614](https://github.com/matrix-org/synapse/issues/10614), [\#10615](https://github.com/matrix-org/synapse/issues/10615), [\#10624](https://github.com/matrix-org/synapse/issues/10624), [\#10640](https://github.com/matrix-org/synapse/issues/10640))
- Add a comment asking developers to leave a reason when bumping the database schema version. ([\#10621](https://github.com/matrix-org/synapse/issues/10621))
- Remove not needed database updates in modify user admin API. ([\#10627](https://github.com/matrix-org/synapse/issues/10627))
- Convert room member storage tuples to `attrs` classes. ([\#10629](https://github.com/matrix-org/synapse/issues/10629), [\#10642](https://github.com/matrix-org/synapse/issues/10642))
- Use auto-attribs for the attrs classes used in sync. ([\#10630](https://github.com/matrix-org/synapse/issues/10630))
- Make `backfill` and `get_missing_events` use the same codepath. ([\#10645](https://github.com/matrix-org/synapse/issues/10645))
- Improve the performance of the `/hierarchy` API (from [MSC2946](https://github.com/matrix-org/matrix-doc/pull/2946)) by caching responses received over federation. ([\#10647](https://github.com/matrix-org/synapse/issues/10647))
- Run a nightly CI build against Twisted trunk. ([\#10651](https://github.com/matrix-org/synapse/issues/10651), [\#10672](https://github.com/matrix-org/synapse/issues/10672))
- Do not print out stack traces for network errors when fetching data over federation. ([\#10662](https://github.com/matrix-org/synapse/issues/10662))
- Simplify tests for device admin rest API. ([\#10664](https://github.com/matrix-org/synapse/issues/10664))
- Add missing type hints to REST servlets. ([\#10665](https://github.com/matrix-org/synapse/issues/10665), [\#10666](https://github.com/matrix-org/synapse/issues/10666), [\#10674](https://github.com/matrix-org/synapse/issues/10674))
- Flatten the `tests.synapse.rests` package by moving the contents of `v1` and `v2_alpha` into the parent. ([\#10667](https://github.com/matrix-org/synapse/issues/10667))
- Update `complement.sh` to rebuild the base Docker image when run with workers. ([\#10686](https://github.com/matrix-org/synapse/issues/10686))
- Split the event-processing methods in `FederationHandler` into a separate `FederationEventHandler`. ([\#10692](https://github.com/matrix-org/synapse/issues/10692))
- Remove unused `compare_digest` function. ([\#10706](https://github.com/matrix-org/synapse/issues/10706))

52 files changed, 123 insertions, 52 deletions

diff --git a/CHANGES.md b/CHANGES.md
index 7046c336a0..986efbba0d 100644
--- a/CHANGES.md
+++ b/CHANGES.md
@@ -1,4 +1,70 @@
-Users will stop receiving message updates via email for addresses that were previously linked to their account
+Synapse 1.42.0rc1 (2021-09-01)
+==============================
+
+Server administrators are reminded to read [the upgrade notes](docs/upgrade.md#upgrading-to-v1420).
+
+
+Features
+--------
+
+- Add support for [MSC3231](https://github.com/matrix-org/matrix-doc/pull/3231): Token authenticated registration. Users can be required to submit a token during registration to authenticate themselves. Contributed by Callum Brown. ([\#10142](https://github.com/matrix-org/synapse/issues/10142))
+- Add support for [MSC3283](https://github.com/matrix-org/matrix-doc/pull/3283): Expose `enable_set_displayname` in capabilities. ([\#10452](https://github.com/matrix-org/synapse/issues/10452))
+- Port the `PresenceRouter` module interface to the new generic interface. ([\#10524](https://github.com/matrix-org/synapse/issues/10524))
+- Add pagination to the spaces summary based on updates to [MSC2946](https://github.com/matrix-org/matrix-doc/pull/2946). ([\#10613](https://github.com/matrix-org/synapse/issues/10613), [\#10725](https://github.com/matrix-org/synapse/issues/10725))
+
+
+Bugfixes
+--------
+
+- Validate new `m.room.power_levels` events. Contributed by @aaronraimist. ([\#10232](https://github.com/matrix-org/synapse/issues/10232))
+- Display an error on User-Interactive Authentication fallback pages when authentication fails. Contributed by Callum Brown. ([\#10561](https://github.com/matrix-org/synapse/issues/10561))
+- Remove pushers when deleting an e-mail address from an account. Pushers for old unlinked emails will also be deleted. ([\#10581](https://github.com/matrix-org/synapse/issues/10581), [\#10734](https://github.com/matrix-org/synapse/issues/10734))
+- Reject Client-Server `/keys/query` requests which provide `device_ids` incorrectly. ([\#10593](https://github.com/matrix-org/synapse/issues/10593))
+- Rooms with unsupported room versions are no longer returned via `/sync`. ([\#10644](https://github.com/matrix-org/synapse/issues/10644))
+- Enforce the maximum length for per-room display names and avatar URLs. ([\#10654](https://github.com/matrix-org/synapse/issues/10654))
+- Fix a bug which caused the `synapse_user_logins_total` Prometheus metric not to be correctly initialised on restart. ([\#10677](https://github.com/matrix-org/synapse/issues/10677))
+- Improve `ServerNoticeServlet` to avoid duplicate requests and add unit tests. ([\#10679](https://github.com/matrix-org/synapse/issues/10679))
+- Fix long-standing issue which caused an error when a thumbnail is requested and there are multiple thumbnails with the same quality rating. ([\#10684](https://github.com/matrix-org/synapse/issues/10684))
+- Fix a regression introduced in v1.41.0 which affected the performance of concurrent fetches of large sets of events, in extreme cases causing the process to hang. ([\#10703](https://github.com/matrix-org/synapse/issues/10703))
+- Fix a regression introduced in Synapse 1.41 which broke email transmission on Systems using older versions of the Twisted library. ([\#10713](https://github.com/matrix-org/synapse/issues/10713))
+
+
+Improved Documentation
+----------------------
+
+- Add documentation on how to connect Django with Synapse using OpenID Connect and django-oauth-toolkit. Contributed by @HugoDelval. ([\#10192](https://github.com/matrix-org/synapse/issues/10192))
+- Advertise https://matrix-org.github.io/synapse documentation in the `README` and `CONTRIBUTING` files. ([\#10595](https://github.com/matrix-org/synapse/issues/10595))
+- Fix some of the titles not rendering in the OpenID Connect documentation. ([\#10639](https://github.com/matrix-org/synapse/issues/10639))
+- Minor clarifications to the documentation for reverse proxies. ([\#10708](https://github.com/matrix-org/synapse/issues/10708))
+- Remove table of contents from the top of installation and contributing documentation pages. ([\#10711](https://github.com/matrix-org/synapse/issues/10711))
+
+
+Deprecations and Removals
+-------------------------
+
+- Remove deprecated Shutdown Room and Purge Room Admin API. ([\#8830](https://github.com/matrix-org/synapse/issues/8830))
+
+
+Internal Changes
+----------------
+
+- Improve type hints for the proxy agent and SRV resolver modules. Contributed by @dklimpel. ([\#10608](https://github.com/matrix-org/synapse/issues/10608))
+- Clean up some of the federation event authentication code for clarity. ([\#10614](https://github.com/matrix-org/synapse/issues/10614), [\#10615](https://github.com/matrix-org/synapse/issues/10615), [\#10624](https://github.com/matrix-org/synapse/issues/10624), [\#10640](https://github.com/matrix-org/synapse/issues/10640))
+- Add a comment asking developers to leave a reason when bumping the database schema version. ([\#10621](https://github.com/matrix-org/synapse/issues/10621))
+- Remove not needed database updates in modify user admin API. ([\#10627](https://github.com/matrix-org/synapse/issues/10627))
+- Convert room member storage tuples to `attrs` classes. ([\#10629](https://github.com/matrix-org/synapse/issues/10629), [\#10642](https://github.com/matrix-org/synapse/issues/10642))
+- Use auto-attribs for the attrs classes used in sync. ([\#10630](https://github.com/matrix-org/synapse/issues/10630))
+- Make `backfill` and `get_missing_events` use the same codepath. ([\#10645](https://github.com/matrix-org/synapse/issues/10645))
+- Improve the performance of the `/hierarchy` API (from [MSC2946](https://github.com/matrix-org/matrix-doc/pull/2946)) by caching responses received over federation. ([\#10647](https://github.com/matrix-org/synapse/issues/10647))
+- Run a nightly CI build against Twisted trunk. ([\#10651](https://github.com/matrix-org/synapse/issues/10651), [\#10672](https://github.com/matrix-org/synapse/issues/10672))
+- Do not print out stack traces for network errors when fetching data over federation. ([\#10662](https://github.com/matrix-org/synapse/issues/10662))
+- Simplify tests for device admin rest API. ([\#10664](https://github.com/matrix-org/synapse/issues/10664))
+- Add missing type hints to REST servlets. ([\#10665](https://github.com/matrix-org/synapse/issues/10665), [\#10666](https://github.com/matrix-org/synapse/issues/10666), [\#10674](https://github.com/matrix-org/synapse/issues/10674))
+- Flatten the `tests.synapse.rests` package by moving the contents of `v1` and `v2_alpha` into the parent. ([\#10667](https://github.com/matrix-org/synapse/issues/10667))
+- Update `complement.sh` to rebuild the base Docker image when run with workers. ([\#10686](https://github.com/matrix-org/synapse/issues/10686))
+- Split the event-processing methods in `FederationHandler` into a separate `FederationEventHandler`. ([\#10692](https://github.com/matrix-org/synapse/issues/10692))
+- Remove unused `compare_digest` function. ([\#10706](https://github.com/matrix-org/synapse/issues/10706))
+
 
 Synapse 1.41.1 (2021-08-31)
 ===========================
diff --git a/changelog.d/10142.feature b/changelog.d/10142.feature
deleted file mode 100644
index 5353f6269d..0000000000
--- a/changelog.d/10142.feature
+++ /dev/null
@@ -1 +0,0 @@
-Add support for [MSC3231 - Token authenticated registration](https://github.com/matrix-org/matrix-doc/pull/3231). Users can be required to submit a token during registration to authenticate themselves. Contributed by Callum Brown.
diff --git a/changelog.d/10192.doc b/changelog.d/10192.doc
deleted file mode 100644
index 3dd00537e8..0000000000
--- a/changelog.d/10192.doc
+++ /dev/null
@@ -1 +0,0 @@
-Add documentation on how to connect Django with synapse using oidc and django-oauth-toolkit. Contributed by @HugoDelval.
diff --git a/changelog.d/10232.bugfix b/changelog.d/10232.bugfix
deleted file mode 100644
index 7be72271e0..0000000000
--- a/changelog.d/10232.bugfix
+++ /dev/null
@@ -1 +0,0 @@
-Validate new `m.room.power_levels` events. Contributed by @aaronraimist.
\ No newline at end of file
diff --git a/changelog.d/10452.feature b/changelog.d/10452.feature
deleted file mode 100644
index f332b383e3..0000000000
--- a/changelog.d/10452.feature
+++ /dev/null
@@ -1 +0,0 @@
-Add support for [MSC3283](https://github.com/matrix-org/matrix-doc/pull/3283): Expose enable_set_displayname in capabilities.
\ No newline at end of file
diff --git a/changelog.d/10524.feature b/changelog.d/10524.feature
deleted file mode 100644
index 288c9bd74e..0000000000
--- a/changelog.d/10524.feature
+++ /dev/null
@@ -1 +0,0 @@
-Port the PresenceRouter module interface to the new generic interface.
\ No newline at end of file
diff --git a/changelog.d/10561.bugfix b/changelog.d/10561.bugfix
deleted file mode 100644
index 2e4f53508c..0000000000
--- a/changelog.d/10561.bugfix
+++ /dev/null
@@ -1 +0,0 @@
-Display an error on User-Interactive Authentication fallback pages when authentication fails. Contributed by Callum Brown.
diff --git a/changelog.d/10581.bugfix b/changelog.d/10581.bugfix
deleted file mode 100644
index 15c7da4497..0000000000
--- a/changelog.d/10581.bugfix
+++ /dev/null
@@ -1 +0,0 @@
-Remove pushers when deleting a 3pid from an account. Pushers for old unlinked emails will also be deleted.
\ No newline at end of file
diff --git a/changelog.d/10593.bugfix b/changelog.d/10593.bugfix
deleted file mode 100644
index af910bfa4d..0000000000
--- a/changelog.d/10593.bugfix
+++ /dev/null
@@ -1 +0,0 @@
-Reject Client-Server `/keys/query` requests which provide `device_ids` incorrectly.
diff --git a/changelog.d/10595.doc b/changelog.d/10595.doc
deleted file mode 100644
index 4823146d6b..0000000000
--- a/changelog.d/10595.doc
+++ /dev/null
@@ -1 +0,0 @@
-Advertise https://matrix-org.github.io/synapse docs in README and CONTRIBUTING files.
diff --git a/changelog.d/10608.misc b/changelog.d/10608.misc
deleted file mode 100644
index 875bdd2fd0..0000000000
--- a/changelog.d/10608.misc
+++ /dev/null
@@ -1 +0,0 @@
-Improve type hints for the proxy agent and SRV resolver modules. Contributed by @dklimpel.
\ No newline at end of file
diff --git a/changelog.d/10613.feature b/changelog.d/10613.feature
deleted file mode 100644
index ffc4e4289c..0000000000
--- a/changelog.d/10613.feature
+++ /dev/null
@@ -1 +0,0 @@
-Add pagination to the spaces summary based on updates to [MSC2946](https://github.com/matrix-org/matrix-doc/pull/2946).
diff --git a/changelog.d/10614.misc b/changelog.d/10614.misc
deleted file mode 100644
index 9a765435db..0000000000
--- a/changelog.d/10614.misc
+++ /dev/null
@@ -1 +0,0 @@
-Clean up some of the federation event authentication code for clarity.
diff --git a/changelog.d/10615.misc b/changelog.d/10615.misc
deleted file mode 100644
index 9a765435db..0000000000
--- a/changelog.d/10615.misc
+++ /dev/null
@@ -1 +0,0 @@
-Clean up some of the federation event authentication code for clarity.
diff --git a/changelog.d/10621.misc b/changelog.d/10621.misc
deleted file mode 100644
index b8de2e1911..0000000000
--- a/changelog.d/10621.misc
+++ /dev/null
@@ -1 +0,0 @@
-Add a comment asking developers to leave a reason when bumping the database schema version.
\ No newline at end of file
diff --git a/changelog.d/10624.misc b/changelog.d/10624.misc
deleted file mode 100644
index 9a765435db..0000000000
--- a/changelog.d/10624.misc
+++ /dev/null
@@ -1 +0,0 @@
-Clean up some of the federation event authentication code for clarity.
diff --git a/changelog.d/10627.misc b/changelog.d/10627.misc
deleted file mode 100644
index e6d314976e..0000000000
--- a/changelog.d/10627.misc
+++ /dev/null
@@ -1 +0,0 @@
-Remove not needed database updates in modify user admin API.
\ No newline at end of file
diff --git a/changelog.d/10629.misc b/changelog.d/10629.misc
deleted file mode 100644
index cca1eb6c57..0000000000
--- a/changelog.d/10629.misc
+++ /dev/null
@@ -1 +0,0 @@
-Convert room member storage tuples to `attrs` classes.
diff --git a/changelog.d/10630.misc b/changelog.d/10630.misc
deleted file mode 100644
index 7d01e00e48..0000000000
--- a/changelog.d/10630.misc
+++ /dev/null
@@ -1 +0,0 @@
-Use auto-attribs for the attrs classes used in sync.
diff --git a/changelog.d/10639.doc b/changelog.d/10639.doc
deleted file mode 100644
index acbac4aad8..0000000000
--- a/changelog.d/10639.doc
+++ /dev/null
@@ -1 +0,0 @@
-Fix some of the titles not rendering in the OIDC documentation.
diff --git a/changelog.d/10640.misc b/changelog.d/10640.misc
deleted file mode 100644
index 9a765435db..0000000000
--- a/changelog.d/10640.misc
+++ /dev/null
@@ -1 +0,0 @@
-Clean up some of the federation event authentication code for clarity.
diff --git a/changelog.d/10642.misc b/changelog.d/10642.misc
deleted file mode 100644
index cca1eb6c57..0000000000
--- a/changelog.d/10642.misc
+++ /dev/null
@@ -1 +0,0 @@
-Convert room member storage tuples to `attrs` classes.
diff --git a/changelog.d/10644.bugfix b/changelog.d/10644.bugfix
deleted file mode 100644
index d88a81fd82..0000000000
--- a/changelog.d/10644.bugfix
+++ /dev/null
@@ -1 +0,0 @@
-Rooms with unsupported room versions are no longer returned via `/sync`.
diff --git a/changelog.d/10645.misc b/changelog.d/10645.misc
deleted file mode 100644
index ac19263cd8..0000000000
--- a/changelog.d/10645.misc
+++ /dev/null
@@ -1 +0,0 @@
-Make `backfill` and `get_missing_events` use the same codepath.
diff --git a/changelog.d/10647.misc b/changelog.d/10647.misc
deleted file mode 100644
index 4407a9030d..0000000000
--- a/changelog.d/10647.misc
+++ /dev/null
@@ -1 +0,0 @@
-Improve the performance of the `/hierarchy` API (from [MSC2946](https://github.com/matrix-org/matrix-doc/pull/2946)) by caching responses received over federation.
diff --git a/changelog.d/10651.misc b/changelog.d/10651.misc
deleted file mode 100644
index 7104c121e0..0000000000
--- a/changelog.d/10651.misc
+++ /dev/null
@@ -1 +0,0 @@
-Run a nightly CI build against Twisted trunk.
diff --git a/changelog.d/10654.bugfix b/changelog.d/10654.bugfix
deleted file mode 100644
index b0bd78453f..0000000000
--- a/changelog.d/10654.bugfix
+++ /dev/null
@@ -1 +0,0 @@
-Enforce the maximum length for per-room display names and avatar URLs.
\ No newline at end of file
diff --git a/changelog.d/10662.misc b/changelog.d/10662.misc
deleted file mode 100644
index 593f9ceaad..0000000000
--- a/changelog.d/10662.misc
+++ /dev/null
@@ -1 +0,0 @@
-Do not print out stack traces for network errors when fetching data over federation.
diff --git a/changelog.d/10664.misc b/changelog.d/10664.misc
deleted file mode 100644
index cebd5e9a96..0000000000
--- a/changelog.d/10664.misc
+++ /dev/null
@@ -1 +0,0 @@
-Simplify tests for device admin rest API.
\ No newline at end of file
diff --git a/changelog.d/10665.misc b/changelog.d/10665.misc
deleted file mode 100644
index 39a37b90b1..0000000000
--- a/changelog.d/10665.misc
+++ /dev/null
@@ -1 +0,0 @@
-Add missing type hints to REST servlets.
diff --git a/changelog.d/10666.misc b/changelog.d/10666.misc
deleted file mode 100644
index 39a37b90b1..0000000000
--- a/changelog.d/10666.misc
+++ /dev/null
@@ -1 +0,0 @@
-Add missing type hints to REST servlets.
diff --git a/changelog.d/10667.misc b/changelog.d/10667.misc
deleted file mode 100644
index c92846ae26..0000000000
--- a/changelog.d/10667.misc
+++ /dev/null
@@ -1 +0,0 @@
-Flatten the `tests.synapse.rests` package by moving the contents of `v1` and `v2_alpha` into the parent.
\ No newline at end of file
diff --git a/changelog.d/10672.misc b/changelog.d/10672.misc
deleted file mode 100644
index 7104c121e0..0000000000
--- a/changelog.d/10672.misc
+++ /dev/null
@@ -1 +0,0 @@
-Run a nightly CI build against Twisted trunk.
diff --git a/changelog.d/10674.misc b/changelog.d/10674.misc
deleted file mode 100644
index 39a37b90b1..0000000000
--- a/changelog.d/10674.misc
+++ /dev/null
@@ -1 +0,0 @@
-Add missing type hints to REST servlets.
diff --git a/changelog.d/10677.bugfix b/changelog.d/10677.bugfix
deleted file mode 100644
index 9964afaaee..0000000000
--- a/changelog.d/10677.bugfix
+++ /dev/null
@@ -1 +0,0 @@
-Fix a bug which caused the `synapse_user_logins_total` Prometheus metric not to be correctly initialised on restart.
diff --git a/changelog.d/10679.bugfix b/changelog.d/10679.bugfix
deleted file mode 100644
index 5c4061f6d5..0000000000
--- a/changelog.d/10679.bugfix
+++ /dev/null
@@ -1 +0,0 @@
-Improve ServerNoticeServlet to avoid duplicate requests and add unit tests.
\ No newline at end of file
diff --git a/changelog.d/10684.bugfix b/changelog.d/10684.bugfix
deleted file mode 100644
index 311b17601a..0000000000
--- a/changelog.d/10684.bugfix
+++ /dev/null
@@ -1 +0,0 @@
-Fix long-standing issue which caused an error when a thumbnail is requested and there are multiple thumbnails with the same quality rating.
diff --git a/changelog.d/10686.misc b/changelog.d/10686.misc
deleted file mode 100644
index b76908d74e..0000000000
--- a/changelog.d/10686.misc
+++ /dev/null
@@ -1 +0,0 @@
-Update `complement.sh` to rebuild the base Docker image when run with workers.
diff --git a/changelog.d/10692.misc b/changelog.d/10692.misc
deleted file mode 100644
index a1b0def76b..0000000000
--- a/changelog.d/10692.misc
+++ /dev/null
@@ -1 +0,0 @@
-Split the event-processing methods in `FederationHandler` into a separate `FederationEventHandler`.
diff --git a/changelog.d/10703.bugfix b/changelog.d/10703.bugfix
deleted file mode 100644
index a5a4ecf8ee..0000000000
--- a/changelog.d/10703.bugfix
+++ /dev/null
@@ -1 +0,0 @@
-Fix a regression introduced in v1.41.0 which affected the performance of concurrent fetches of large sets of events, in extreme cases causing the process to hang.
diff --git a/changelog.d/10706.misc b/changelog.d/10706.misc
deleted file mode 100644
index eed4aa58d6..0000000000
--- a/changelog.d/10706.misc
+++ /dev/null
@@ -1 +0,0 @@
-Remove unused `compare_digest` function.
diff --git a/changelog.d/10708.doc b/changelog.d/10708.doc
deleted file mode 100644
index 99f9d69288..0000000000
--- a/changelog.d/10708.doc
+++ /dev/null
@@ -1 +0,0 @@
-Minor clarifications to the documentation for reverse proxies.
diff --git a/changelog.d/10711.doc b/changelog.d/10711.doc
deleted file mode 100644
index c495f98be8..0000000000
--- a/changelog.d/10711.doc
+++ /dev/null
@@ -1 +0,0 @@
-Removed table of contents from the top of installation and contributing documentation pages.
\ No newline at end of file
diff --git a/changelog.d/10713.bugfix b/changelog.d/10713.bugfix
deleted file mode 100644
index e8caf3d23a..0000000000
--- a/changelog.d/10713.bugfix
+++ /dev/null
@@ -1 +0,0 @@
-Fix a regression introduced in Synapse 1.41 which broke email transmission on Systems using older versions of the Twisted library.
diff --git a/changelog.d/10723.bugfix b/changelog.d/10723.bugfix
deleted file mode 100644
index e6ffdc9512..0000000000
--- a/changelog.d/10723.bugfix
+++ /dev/null
@@ -1 +0,0 @@
-Fix unauthorised exposure of room metadata to communities.
diff --git a/changelog.d/10725.feature b/changelog.d/10725.feature
deleted file mode 100644
index ffc4e4289c..0000000000
--- a/changelog.d/10725.feature
+++ /dev/null
@@ -1 +0,0 @@
-Add pagination to the spaces summary based on updates to [MSC2946](https://github.com/matrix-org/matrix-doc/pull/2946).
diff --git a/changelog.d/8830.removal b/changelog.d/8830.removal
deleted file mode 100644
index b3a93a9af2..0000000000
--- a/changelog.d/8830.removal
+++ /dev/null
@@ -1 +0,0 @@
-Remove deprecated Shutdown Room and Purge Room Admin API.
\ No newline at end of file
diff --git a/debian/changelog b/debian/changelog
index 5f7a795b6e..0f7dbdf71e 100644
--- a/debian/changelog
+++ b/debian/changelog
@@ -1,3 +1,9 @@
+matrix-synapse-py3 (1.42.0~rc1) stable; urgency=medium
+
+  * New synapse release 1.42.0rc1.
+
+ -- Synapse Packaging team <packages@matrix.org>  Wed, 01 Sep 2021 11:37:48 +0100
+
 matrix-synapse-py3 (1.41.1) stable; urgency=high
 
   * New synapse release 1.41.1.
diff --git a/docs/upgrade.md b/docs/upgrade.md
index dcf0a7db5b..453dbbabe7 100644
--- a/docs/upgrade.md
+++ b/docs/upgrade.md
@@ -85,7 +85,7 @@ process, for example:
     dpkg -i matrix-synapse-py3_1.3.0+stretch1_amd64.deb
     ```
 
-# Upgrading to v1.xx.0
+# Upgrading to v1.42.0
 
 ## Removal of old Room Admin API
 
@@ -107,12 +107,12 @@ This may affect you if you make use of custom HTML templates for the
 The template is now provided an `error` variable if the authentication
 process failed. See the default templates linked above for an example.
 
-# Upgrading to v1.42.0
-
 ## Removal of out-of-date email pushers
+
 Users will stop receiving message updates via email for addresses that were
 once, but not still, linked to their account.
 
+
 # Upgrading to v1.41.0
 
 ## Add support for routing outbound HTTP requests via a proxy for federation
diff --git a/synapse/__init__.py b/synapse/__init__.py
index 06d80f79b3..e5b075c53b 100644
--- a/synapse/__init__.py
+++ b/synapse/__init__.py
@@ -47,7 +47,7 @@ try:
 except ImportError:
     pass
 
-__version__ = "1.41.1"
+__version__ = "1.42.0rc1"
 
 if bool(os.environ.get("SYNAPSE_TEST_PATCH_LOG_CONTEXTS", False)):
     # We import here so that we don't have to install a bunch of deps when
diff --git a/synapse/storage/databases/main/pusher.py b/synapse/storage/databases/main/pusher.py
index e47caa2125..63ac09c61d 100644
--- a/synapse/storage/databases/main/pusher.py
+++ b/synapse/storage/databases/main/pusher.py
@@ -430,10 +430,11 @@ class PusherWorkerStore(SQLBaseStore):
             """
 
             txn.execute(sql, (last_pusher, batch_size))
+            rows = txn.fetchall()
 
             last = None
             num_deleted = 0
-            for row in txn:
+            for row in rows:
                 last = row[0]
                 num_deleted += 1
                 self.db_pool.simple_delete_txn(
diff --git a/tests/push/test_email.py b/tests/push/test_email.py
index 2bed7302cf..fa8018e5a7 100644
--- a/tests/push/test_email.py
+++ b/tests/push/test_email.py
@@ -379,6 +379,50 @@ class EmailPusherTests(HomeserverTestCase):
         pushers = list(pushers)
         self.assertEqual(len(pushers), 0)
 
+    def test_remove_unlinked_pushers_background_job(self):
+        """Checks that all existing pushers associated with unlinked email addresses are removed
+        upon running the remove_deleted_email_pushers background update.
+        """
+        # disassociate the user's email address manually (without deleting the pusher).
+        # This resembles the old behaviour, which the background update below is intended
+        # to clean up.
+        self.get_success(
+            self.hs.get_datastore().user_delete_threepid(
+                self.user_id, "email", "a@example.com"
+            )
+        )
+
+        # Run the "remove_deleted_email_pushers" background job
+        self.get_success(
+            self.hs.get_datastore().db_pool.simple_insert(
+                table="background_updates",
+                values={
+                    "update_name": "remove_deleted_email_pushers",
+                    "progress_json": "{}",
+                    "depends_on": None,
+                },
+            )
+        )
+
+        # ... and tell the DataStore that it hasn't finished all updates yet
+        self.hs.get_datastore().db_pool.updates._all_done = False
+
+        # Now let's actually drive the updates to completion
+        while not self.get_success(
+            self.hs.get_datastore().db_pool.updates.has_completed_background_updates()
+        ):
+            self.get_success(
+                self.hs.get_datastore().db_pool.updates.do_next_background_update(100),
+                by=0.1,
+            )
+
+        # Check that all pushers with unlinked addresses were deleted
+        pushers = self.get_success(
+            self.hs.get_datastore().get_pushers_by({"user_name": self.user_id})
+        )
+        pushers = list(pushers)
+        self.assertEqual(len(pushers), 0)
+
     def _check_for_mail(self) -> Tuple[Sequence, Dict]:
         """
         Assert that synapse sent off exactly one email notification.