Merge branch 'develop' into madlittlemods/return-historical-events-in-order-from-backfill

Conflicts:
	scripts-dev/complement.sh
	synapse/handlers/room_batch.py

72 files changed, 7 insertions, 65 deletions

diff --git a/changelog.d/10097.bugfix b/changelog.d/10097.bugfix
deleted file mode 100644
index 5d3d9587c2..0000000000
--- a/changelog.d/10097.bugfix
+++ /dev/null
@@ -1 +0,0 @@
-Fix a long-standing bug which allowed hidden devices to receive to-device messages, resulting in unnecessary database bloat.
diff --git a/changelog.d/10520.misc b/changelog.d/10520.misc
new file mode 100644
index 0000000000..a911e165da
--- /dev/null
+++ b/changelog.d/10520.misc
@@ -0,0 +1 @@
+Send and handle cross-signing messages using the stable prefix.
diff --git a/changelog.d/10943.misc b/changelog.d/10943.misc
deleted file mode 100644
index 3ce28d1a67..0000000000
--- a/changelog.d/10943.misc
+++ /dev/null
@@ -1 +0,0 @@
-Add type annotations for the `log_function` decorator.
diff --git a/changelog.d/10969.bugfix b/changelog.d/10969.bugfix
deleted file mode 100644
index 89c299b8e8..0000000000
--- a/changelog.d/10969.bugfix
+++ /dev/null
@@ -1 +0,0 @@
-Fix a long-standing bug where messages in the `device_inbox` table for deleted devices would persist indefinitely. Contributed by @dklimpel and @JohannesKleine.
diff --git a/changelog.d/11033.bugfix b/changelog.d/11033.bugfix
deleted file mode 100644
index fa99f187b8..0000000000
--- a/changelog.d/11033.bugfix
+++ /dev/null
@@ -1 +0,0 @@
-Do not accept events if a third-party rule module API callback raises an exception.
diff --git a/changelog.d/11097.feature b/changelog.d/11097.feature
deleted file mode 100644
index d7563a406c..0000000000
--- a/changelog.d/11097.feature
+++ /dev/null
@@ -1 +0,0 @@
-Advertise support for Client-Server API r0.6.1.
\ No newline at end of file
diff --git a/changelog.d/11098.misc b/changelog.d/11098.misc
deleted file mode 100644
index 1e337bee54..0000000000
--- a/changelog.d/11098.misc
+++ /dev/null
@@ -1 +0,0 @@
-Add type hints to `synapse.events`.
diff --git a/changelog.d/11099.feature b/changelog.d/11099.feature
deleted file mode 100644
index c9126d4a9d..0000000000
--- a/changelog.d/11099.feature
+++ /dev/null
@@ -1 +0,0 @@
-Add search by room ID and room alias to List Room admin API.
\ No newline at end of file
diff --git a/changelog.d/11126.feature b/changelog.d/11126.feature
deleted file mode 100644
index c6078fe081..0000000000
--- a/changelog.d/11126.feature
+++ /dev/null
@@ -1 +0,0 @@
-Add an `on_new_event` third-party rules callback to allow Synapse modules to act after an event has been sent into a room.
diff --git a/changelog.d/11128.doc b/changelog.d/11128.doc
deleted file mode 100644
index c024679218..0000000000
--- a/changelog.d/11128.doc
+++ /dev/null
@@ -1 +0,0 @@
-Improve example HAProxy config in the docs to properly handle host headers with port information. This is required for federation over port 443 to work correctly.
diff --git a/changelog.d/11129.bugfix b/changelog.d/11129.bugfix
deleted file mode 100644
index 5e9aa538ec..0000000000
--- a/changelog.d/11129.bugfix
+++ /dev/null
@@ -1 +0,0 @@
-Fix long-standing bug where verification requests could fail in certain cases if whitelist was in place but did not include your own homeserver.
\ No newline at end of file
diff --git a/changelog.d/11137.misc b/changelog.d/11137.misc
deleted file mode 100644
index f0d6476f48..0000000000
--- a/changelog.d/11137.misc
+++ /dev/null
@@ -1 +0,0 @@
-Remove and document unnecessary `RoomStreamToken` checks in application service ephemeral event code.
\ No newline at end of file
diff --git a/changelog.d/11147.feature b/changelog.d/11147.feature
deleted file mode 100644
index af72d85c20..0000000000
--- a/changelog.d/11147.feature
+++ /dev/null
@@ -1 +0,0 @@
-Add a module API method to update a user's membership in a room.
diff --git a/changelog.d/11151.doc b/changelog.d/11151.doc
deleted file mode 100644
index 68cd99471f..0000000000
--- a/changelog.d/11151.doc
+++ /dev/null
@@ -1 +0,0 @@
-Add documentation for using Authentik as an OpenID Connect Identity Provider. Contributed by @samip5.
\ No newline at end of file
diff --git a/changelog.d/11157.misc b/changelog.d/11157.misc
deleted file mode 100644
index 75444c51d1..0000000000
--- a/changelog.d/11157.misc
+++ /dev/null
@@ -1 +0,0 @@
-Only allow old Element/Riot Android clients to send read receipts without a request body. All other clients must include a request body as required by the specification. Contributed by @rogersheu.
diff --git a/changelog.d/11164.misc b/changelog.d/11164.misc
deleted file mode 100644
index 751da49183..0000000000
--- a/changelog.d/11164.misc
+++ /dev/null
@@ -1 +0,0 @@
-Add type hints so that `synapse.http` passes `mypy` checks.
\ No newline at end of file
diff --git a/changelog.d/11166.misc b/changelog.d/11166.misc
deleted file mode 100644
index 79342e43d9..0000000000
--- a/changelog.d/11166.misc
+++ /dev/null
@@ -1 +0,0 @@
-Update scripts to pass Shellcheck lints.
diff --git a/changelog.d/11171.misc b/changelog.d/11171.misc
deleted file mode 100644
index b6a41a96da..0000000000
--- a/changelog.d/11171.misc
+++ /dev/null
@@ -1 +0,0 @@
-Add knock information in admin export. Contributed by Rafael Gonçalves.
diff --git a/changelog.d/11178.feature b/changelog.d/11178.feature
deleted file mode 100644
index 10b1cdffdc..0000000000
--- a/changelog.d/11178.feature
+++ /dev/null
@@ -1 +0,0 @@
-Add metrics for thread pool usage.
diff --git a/changelog.d/11179.misc b/changelog.d/11179.misc
deleted file mode 100644
index aded2e8367..0000000000
--- a/changelog.d/11179.misc
+++ /dev/null
@@ -1 +0,0 @@
-Add tests to check that `ClientIpStore.get_last_client_ip_by_device` and `get_user_ip_and_agents` combine database and in-memory data correctly.
diff --git a/changelog.d/11187.feature b/changelog.d/11187.feature
deleted file mode 100644
index dd28109030..0000000000
--- a/changelog.d/11187.feature
+++ /dev/null
@@ -1 +0,0 @@
-Support the stable room type field for [MSC3288](https://github.com/matrix-org/matrix-doc/pull/3288).
diff --git a/changelog.d/11188.bugfix b/changelog.d/11188.bugfix
deleted file mode 100644
index 0688743c00..0000000000
--- a/changelog.d/11188.bugfix
+++ /dev/null
@@ -1 +0,0 @@
-Allow an empty list of `state_events_at_start` to be sent when using the [MSC2716](https://github.com/matrix-org/matrix-doc/pull/2716) `/batch_send` endpoint and the author of the historical messages is already part of the current room state at the given `?prev_event_id`.
diff --git a/changelog.d/11191.bugfix b/changelog.d/11191.bugfix
deleted file mode 100644
index 9104db7f0e..0000000000
--- a/changelog.d/11191.bugfix
+++ /dev/null
@@ -1 +0,0 @@
-Fix a bug introduced in Synapse 1.45.0 which prevented the `synapse_review_recent_signups` script from running. Contributed by @samuel-p.
diff --git a/changelog.d/11194.misc b/changelog.d/11194.misc
deleted file mode 100644
index fc1d06ba89..0000000000
--- a/changelog.d/11194.misc
+++ /dev/null
@@ -1 +0,0 @@
-Refactor `Filter` to check different fields depending on the data type.
diff --git a/changelog.d/11198.doc b/changelog.d/11198.doc
deleted file mode 100644
index 54ec94acbc..0000000000
--- a/changelog.d/11198.doc
+++ /dev/null
@@ -1 +0,0 @@
-Clarify lack of support for Windows.
diff --git a/changelog.d/11199.bugfix b/changelog.d/11199.bugfix
deleted file mode 100644
index dc3ea8d515..0000000000
--- a/changelog.d/11199.bugfix
+++ /dev/null
@@ -1 +0,0 @@
-Delete `to_device` messages for hidden devices that will never be read, reducing database size.
\ No newline at end of file
diff --git a/changelog.d/11200.bugfix b/changelog.d/11200.bugfix
deleted file mode 100644
index c855081986..0000000000
--- a/changelog.d/11200.bugfix
+++ /dev/null
@@ -1 +0,0 @@
-Fix a long-standing bug wherein a missing `Content-Type` header when downloading remote media would cause Synapse to throw an error.
\ No newline at end of file
diff --git a/changelog.d/11204.feature b/changelog.d/11204.feature
deleted file mode 100644
index f58ed4b3dc..0000000000
--- a/changelog.d/11204.feature
+++ /dev/null
@@ -1 +0,0 @@
-Add a module API method to retrieve the current state of a room.
diff --git a/changelog.d/11205.misc b/changelog.d/11205.misc
deleted file mode 100644
index 62395c9432..0000000000
--- a/changelog.d/11205.misc
+++ /dev/null
@@ -1 +0,0 @@
-Improve type hints for the relations datastore.
diff --git a/changelog.d/11206.removal b/changelog.d/11206.removal
deleted file mode 100644
index cf05b16672..0000000000
--- a/changelog.d/11206.removal
+++ /dev/null
@@ -1 +0,0 @@
-The `user_may_create_room_with_invites` module callback is now deprecated. Please refer to the [upgrade notes](https://matrix-org.github.io/synapse/develop/upgrade#upgrading-to-v1470) for more information.
diff --git a/changelog.d/11207.bugfix b/changelog.d/11207.bugfix
deleted file mode 100644
index 7e98d565a1..0000000000
--- a/changelog.d/11207.bugfix
+++ /dev/null
@@ -1 +0,0 @@
-Fix a long-standing bug which could result in serialization errors and potentially duplicate transaction data when sending ephemeral events to application services. Contributed by @Fizzadar at Beeper.
diff --git a/changelog.d/11209.docker b/changelog.d/11209.docker
deleted file mode 100644
index 838b165ac9..0000000000
--- a/changelog.d/11209.docker
+++ /dev/null
@@ -1 +0,0 @@
-Avoid changing userid when started as a non-root user, and no explicit `UID` is set.
diff --git a/changelog.d/11210.feature b/changelog.d/11210.feature
deleted file mode 100644
index 8f8e386415..0000000000
--- a/changelog.d/11210.feature
+++ /dev/null
@@ -1 +0,0 @@
-Calculate a default value for `public_baseurl` based on `server_name`.
diff --git a/changelog.d/11211.feature b/changelog.d/11211.feature
deleted file mode 100644
index feeb0cf089..0000000000
--- a/changelog.d/11211.feature
+++ /dev/null
@@ -1 +0,0 @@
-Add support for serving `/.well-known/matrix/server` files, to redirect federation traffic to port 443.
diff --git a/changelog.d/11212.bugfix b/changelog.d/11212.bugfix
deleted file mode 100644
index ba6efab25b..0000000000
--- a/changelog.d/11212.bugfix
+++ /dev/null
@@ -1 +0,0 @@
-Fix a long-standing bug where messages in the `device_inbox` table for deleted devices would persist indefinitely. Contributed by @dklimpel and @JohannesKleine.
\ No newline at end of file
diff --git a/changelog.d/11213.removal b/changelog.d/11213.removal
deleted file mode 100644
index 9e5ec936e3..0000000000
--- a/changelog.d/11213.removal
+++ /dev/null
@@ -1 +0,0 @@
-Remove deprecated admin API to delete rooms (`POST /_synapse/admin/v1/rooms/<room_id>/delete`).
\ No newline at end of file
diff --git a/changelog.d/11217.bugfix b/changelog.d/11217.bugfix
deleted file mode 100644
index 67ebb0d0e3..0000000000
--- a/changelog.d/11217.bugfix
+++ /dev/null
@@ -1 +0,0 @@
-Fix a bug introduced in 1.35.0 which made it impossible to join rooms that return a `send_join` response containing floats.
\ No newline at end of file
diff --git a/changelog.d/11221.doc b/changelog.d/11221.doc
deleted file mode 100644
index 17010bac8b..0000000000
--- a/changelog.d/11221.doc
+++ /dev/null
@@ -1 +0,0 @@
-Improve code formatting and fix a few typos in docs. Contributed by @sumnerevans at Beeper.
diff --git a/changelog.d/11225.misc b/changelog.d/11225.misc
deleted file mode 100644
index f14f65f9d4..0000000000
--- a/changelog.d/11225.misc
+++ /dev/null
@@ -1 +0,0 @@
-Replace outdated links in the pull request checklist with links to the rendered documentation.
diff --git a/changelog.d/11226.misc b/changelog.d/11226.misc
deleted file mode 100644
index 9ed4760ae0..0000000000
--- a/changelog.d/11226.misc
+++ /dev/null
@@ -1 +0,0 @@
-Fix a bug in unit test `test_block_room_and_not_purge`.
diff --git a/changelog.d/11228.feature b/changelog.d/11228.feature
deleted file mode 100644
index 33c1756b50..0000000000
--- a/changelog.d/11228.feature
+++ /dev/null
@@ -1 +0,0 @@
-Allow the admin [Delete Room API](https://matrix-org.github.io/synapse/latest/admin_api/rooms.html#delete-room-api) to block a room without the need to join it.
diff --git a/changelog.d/11229.misc b/changelog.d/11229.misc
deleted file mode 100644
index 7bb01cf079..0000000000
--- a/changelog.d/11229.misc
+++ /dev/null
@@ -1 +0,0 @@
-`ObservableDeferred`: run registered observers in order.
diff --git a/changelog.d/11231.misc b/changelog.d/11231.misc
deleted file mode 100644
index c7fca7071e..0000000000
--- a/changelog.d/11231.misc
+++ /dev/null
@@ -1 +0,0 @@
-Minor speed up to start up times and getting updates for groups by adding missing index to `local_group_updates.stream_id`.
diff --git a/changelog.d/11233.misc b/changelog.d/11233.misc
deleted file mode 100644
index fdf9e5642e..0000000000
--- a/changelog.d/11233.misc
+++ /dev/null
@@ -1 +0,0 @@
-Add `twine` and `towncrier` as dev dependencies, as they're used by the release script.
diff --git a/changelog.d/11234.bugfix b/changelog.d/11234.bugfix
deleted file mode 100644
index c0c02a58f6..0000000000
--- a/changelog.d/11234.bugfix
+++ /dev/null
@@ -1 +0,0 @@
-Fix long-standing bug where cross signing keys were not included in the response to `/r0/keys/query` the first time a remote user was queried.
diff --git a/changelog.d/11236.feature b/changelog.d/11236.feature
deleted file mode 100644
index e7aeee2aa6..0000000000
--- a/changelog.d/11236.feature
+++ /dev/null
@@ -1 +0,0 @@
-Support filtering by relation senders & types per [MSC3440](https://github.com/matrix-org/matrix-doc/pull/3440).
diff --git a/changelog.d/11237.misc b/changelog.d/11237.misc
deleted file mode 100644
index b90efc6535..0000000000
--- a/changelog.d/11237.misc
+++ /dev/null
@@ -1 +0,0 @@
-Allow `stream_writers.typing` config to be a list of one worker.
diff --git a/changelog.d/11239.misc b/changelog.d/11239.misc
deleted file mode 100644
index 48a796bed0..0000000000
--- a/changelog.d/11239.misc
+++ /dev/null
@@ -1 +0,0 @@
-Remove debugging statement in tests.
diff --git a/changelog.d/11240.bugfix b/changelog.d/11240.bugfix
deleted file mode 100644
index 94d73f67e3..0000000000
--- a/changelog.d/11240.bugfix
+++ /dev/null
@@ -1 +0,0 @@
-Fix a long-standing bug where all requests that read events from the database could get stuck as a result of losing the database connection.
diff --git a/changelog.d/11242.misc b/changelog.d/11242.misc
deleted file mode 100644
index 3a98259edf..0000000000
--- a/changelog.d/11242.misc
+++ /dev/null
@@ -1 +0,0 @@
-Split out federated PDU retrieval function into a non-cached version.
diff --git a/changelog.d/11244.misc b/changelog.d/11244.misc
deleted file mode 100644
index c6e65df97f..0000000000
--- a/changelog.d/11244.misc
+++ /dev/null
@@ -1 +0,0 @@
-Fix [MSC2716](https://github.com/matrix-org/matrix-doc/pull/2716) historical messages backfilling in random order on remote homeservers.
diff --git a/changelog.d/11246.misc b/changelog.d/11246.misc
deleted file mode 100644
index e5e912c1b0..0000000000
--- a/changelog.d/11246.misc
+++ /dev/null
@@ -1 +0,0 @@
-Add an additional test for the `cachedList` method decorator.
diff --git a/changelog.d/11247.misc b/changelog.d/11247.misc
deleted file mode 100644
index 5ce701560e..0000000000
--- a/changelog.d/11247.misc
+++ /dev/null
@@ -1 +0,0 @@
-Clean up code relating to to-device messages and sending ephemeral events to application services.
\ No newline at end of file
diff --git a/changelog.d/11253.misc b/changelog.d/11253.misc
deleted file mode 100644
index 71c55a2751..0000000000
--- a/changelog.d/11253.misc
+++ /dev/null
@@ -1 +0,0 @@
-Make minor correction to the type of `auth_checkers` callbacks.
diff --git a/changelog.d/11255.bugfix b/changelog.d/11255.bugfix
deleted file mode 100644
index ce72592624..0000000000
--- a/changelog.d/11255.bugfix
+++ /dev/null
@@ -1 +0,0 @@
-Fix rolling back Synapse version when using workers.
diff --git a/changelog.d/11257.doc b/changelog.d/11257.doc
deleted file mode 100644
index 1205be2add..0000000000
--- a/changelog.d/11257.doc
+++ /dev/null
@@ -1 +0,0 @@
-Add documentation for using LemonLDAP as an OpenID Connect Identity Provider. Contributed by @l00ptr.
diff --git a/changelog.d/11262.bugfix b/changelog.d/11262.bugfix
deleted file mode 100644
index 768fbb8973..0000000000
--- a/changelog.d/11262.bugfix
+++ /dev/null
@@ -1 +0,0 @@
-Fix a bug where if a remote event is being processed by a worker when it gets killed then it won't get processed on restart. Introduced in v1.37.1.
diff --git a/changelog.d/11263.feature b/changelog.d/11263.feature
deleted file mode 100644
index 831e76ec9f..0000000000
--- a/changelog.d/11263.feature
+++ /dev/null
@@ -1 +0,0 @@
-Add some background update admin APIs.
diff --git a/changelog.d/11269.misc b/changelog.d/11269.misc
deleted file mode 100644
index a2149c2d2d..0000000000
--- a/changelog.d/11269.misc
+++ /dev/null
@@ -1 +0,0 @@
-Clean up trivial aspects of the Debian package build tooling.
diff --git a/changelog.d/11270.misc b/changelog.d/11270.misc
deleted file mode 100644
index e2181b9b2a..0000000000
--- a/changelog.d/11270.misc
+++ /dev/null
@@ -1 +0,0 @@
-Blacklist new SyTest that checks that key uploads are valid pending the validation being implemented in Synapse.
diff --git a/changelog.d/11273.misc b/changelog.d/11273.misc
deleted file mode 100644
index a2149c2d2d..0000000000
--- a/changelog.d/11273.misc
+++ /dev/null
@@ -1 +0,0 @@
-Clean up trivial aspects of the Debian package build tooling.
diff --git a/changelog.d/11276.bugfix b/changelog.d/11276.bugfix
deleted file mode 100644
index ce72592624..0000000000
--- a/changelog.d/11276.bugfix
+++ /dev/null
@@ -1 +0,0 @@
-Fix rolling back Synapse version when using workers.
diff --git a/changelog.d/11278.misc b/changelog.d/11278.misc
deleted file mode 100644
index 9b014bc8b4..0000000000
--- a/changelog.d/11278.misc
+++ /dev/null
@@ -1 +0,0 @@
-Fix a small typo in the error response when a relation type other than 'm.annotation' is passed to `GET /rooms/{room_id}/aggregations/{event_id}`.
\ No newline at end of file
diff --git a/changelog.d/11282.misc b/changelog.d/11282.misc
deleted file mode 100644
index 4720519cbc..0000000000
--- a/changelog.d/11282.misc
+++ /dev/null
@@ -1 +0,0 @@
-Require all files in synapse/ and tests/ to pass mypy unless specifically excluded.
diff --git a/changelog.d/11285.misc b/changelog.d/11285.misc
deleted file mode 100644
index 4720519cbc..0000000000
--- a/changelog.d/11285.misc
+++ /dev/null
@@ -1 +0,0 @@
-Require all files in synapse/ and tests/ to pass mypy unless specifically excluded.
diff --git a/changelog.d/11286.doc b/changelog.d/11286.doc
deleted file mode 100644
index 890d7b4ee4..0000000000
--- a/changelog.d/11286.doc
+++ /dev/null
@@ -1 +0,0 @@
-Fix typo in the word `available` and fix HTTP method (should be `GET`) for the `username_available` admin API. Contributed by Stanislav Motylkov.
diff --git a/changelog.d/11331.misc b/changelog.d/11331.misc
new file mode 100644
index 0000000000..1ab3a6a975
--- /dev/null
+++ b/changelog.d/11331.misc
@@ -0,0 +1 @@
+A test helper (`wait_for_background_updates`) no longer depends on classes defining a `store` property.
diff --git a/changelog.d/11427.doc b/changelog.d/11427.doc
new file mode 100644
index 0000000000..01cdfcf2b7
--- /dev/null
+++ b/changelog.d/11427.doc
@@ -0,0 +1 @@
+Document the usage of refresh tokens.
\ No newline at end of file
diff --git a/changelog.d/11520.misc b/changelog.d/11520.misc
new file mode 100644
index 0000000000..2d84120e19
--- /dev/null
+++ b/changelog.d/11520.misc
@@ -0,0 +1 @@
+Use HTTPStatus constants in place of literals in `tests.rest.client.test_auth`.
\ No newline at end of file
diff --git a/changelog.d/11531.misc b/changelog.d/11531.misc
new file mode 100644
index 0000000000..ed6ef3bb3e
--- /dev/null
+++ b/changelog.d/11531.misc
@@ -0,0 +1 @@
+Add a receipt types constant for `m.read`.
diff --git a/changelog.d/11535.misc b/changelog.d/11535.misc
new file mode 100644
index 0000000000..580ac354ab
--- /dev/null
+++ b/changelog.d/11535.misc
@@ -0,0 +1 @@
+Clean up `synapse.rest.admin`.
\ No newline at end of file
diff --git a/changelog.d/11536.misc b/changelog.d/11536.misc
new file mode 100644
index 0000000000..b9191c111b
--- /dev/null
+++ b/changelog.d/11536.misc
@@ -0,0 +1 @@
+Improvements to log messages around handling stream ids.