Merge branch 'develop' of github.com:matrix-org/synapse into anoa/user_param_ui_auth github/anoa/user_param_ui_auth anoa/user_param_ui_auth

* 'develop' of github.com:matrix-org/synapse: (369 commits)
  Add functions to `MultiWriterIdGen` used by events stream (#8164)
  Do not allow send_nonmember_event to be called with shadow-banned users. (#8158)
  Changelog fixes
  1.19.1rc1
  Make StreamIdGen `get_next` and `get_next_mult` async  (#8161)
  Wording fixes to 'name' user admin api filter (#8163)
  Fix missing double-backtick in RST document
  Search in columns 'name' and 'displayname' in the admin users endpoint (#7377)
  Add type hints for state. (#8140)
  Stop shadow-banned users from sending non-member events. (#8142)
  Allow capping a room's retention policy (#8104)
  Add healthcheck for default localhost 8008 port on /health endpoint. (#8147)
  Fix flaky shadow-ban tests. (#8152)
  Fix join ratelimiter breaking profile updates and idempotency (#8153)
  Do not apply ratelimiting on joins to appservices (#8139)
  Don't fail /submit_token requests on incorrect session ID if request_token_inhibit_3pid_errors is turned on (#7991)
  Do not apply ratelimiting on joins to appservices (#8139)
  Micro-optimisations to get_auth_chain_ids (#8132)
  Allow denying or shadow banning registrations via the spam checker (#8034)
  Stop shadow-banned users from sending invites. (#8095)
  ...

54 files changed, 48 insertions, 6 deletions

diff --git a/changelog.d/7377.misc b/changelog.d/7377.misc
new file mode 100644

index 0000000000..b3ec08855b
--- /dev/null
+++ b/changelog.d/7377.misc
@@ -0,0 +1 @@
+Add filter `name` to the `/users` admin API, which filters by user ID or displayname. Contributed by Awesome Technologies Innovationslabor GmbH.
diff --git a/changelog.d/7606.bugfix b/changelog.d/7606.bugfix
deleted file mode 100644

index 6c488c3996..0000000000
--- a/changelog.d/7606.bugfix
+++ /dev/null
@@ -1 +0,0 @@
-Remove `user_id` from the response to `GET /_matrix/client/r0/presence/{userId}/status` to match the specification.
diff --git a/changelog.d/7639.feature b/changelog.d/7639.feature
deleted file mode 100644

index ca80fc0ca2..0000000000
--- a/changelog.d/7639.feature
+++ /dev/null
@@ -1 +0,0 @@
-Add an option to enable encryption by default for new rooms.
diff --git a/changelog.d/7652.doc b/changelog.d/7652.doc
deleted file mode 100644

index c3ebbeb262..0000000000
--- a/changelog.d/7652.doc
+++ /dev/null
@@ -1 +0,0 @@
-Spelling correction in sample_config.yaml.
diff --git a/changelog.d/7657.misc b/changelog.d/7657.misc
deleted file mode 100644

index 384c05d2cd..0000000000
--- a/changelog.d/7657.misc
+++ /dev/null
@@ -1 +0,0 @@
-Clean-up the login fallback code.
diff --git a/changelog.d/7663.bugfix b/changelog.d/7663.bugfix
deleted file mode 100644

index b58316b34f..0000000000
--- a/changelog.d/7663.bugfix
+++ /dev/null
@@ -1 +0,0 @@
-Fix intermittent exception during startup, introduced in Synapse 1.14.0.
diff --git a/changelog.d/7664.misc b/changelog.d/7664.misc
deleted file mode 100644

index bbf0908109..0000000000
--- a/changelog.d/7664.misc
+++ /dev/null
@@ -1 +0,0 @@
-Increase the default SAML session expirary time to 15 minutes.
diff --git a/changelog.d/7864.bugfix b/changelog.d/7864.bugfix
new file mode 100644

index 0000000000..8623355fe9
--- /dev/null
+++ b/changelog.d/7864.bugfix
@@ -0,0 +1 @@
+Fix a memory leak by limiting the length of time that messages will be queued for a remote server that has been unreachable.
diff --git a/changelog.d/7991.misc b/changelog.d/7991.misc
new file mode 100644

index 0000000000..1562e3af9e
--- /dev/null
+++ b/changelog.d/7991.misc
@@ -0,0 +1 @@
+Don't fail `/submit_token` requests on incorrect session ID if `request_token_inhibit_3pid_errors` is turned on.
diff --git a/changelog.d/8013.feature b/changelog.d/8013.feature
new file mode 100644

index 0000000000..b1eaf1e78a
--- /dev/null
+++ b/changelog.d/8013.feature
@@ -0,0 +1 @@
+Iteratively encode JSON to avoid blocking the reactor.
diff --git a/changelog.d/8034.feature b/changelog.d/8034.feature
new file mode 100644

index 0000000000..813e6d0903
--- /dev/null
+++ b/changelog.d/8034.feature
@@ -0,0 +1 @@
+Add support for shadow-banning users (ignoring any message send requests).
diff --git a/changelog.d/8037.feature b/changelog.d/8037.feature
new file mode 100644

index 0000000000..2e5127477d
--- /dev/null
+++ b/changelog.d/8037.feature
@@ -0,0 +1 @@
+Use the default template file when its equivalent is not found in a custom template directory.
\ No newline at end of file
diff --git a/changelog.d/8071.misc b/changelog.d/8071.misc
new file mode 100644

index 0000000000..dfe4c03171
--- /dev/null
+++ b/changelog.d/8071.misc
@@ -0,0 +1 @@
+Convert various parts of the codebase to async/await.
diff --git a/changelog.d/8072.misc b/changelog.d/8072.misc
new file mode 100644

index 0000000000..dfe4c03171
--- /dev/null
+++ b/changelog.d/8072.misc
@@ -0,0 +1 @@
+Convert various parts of the codebase to async/await.
diff --git a/changelog.d/8074.misc b/changelog.d/8074.misc
new file mode 100644

index 0000000000..dfe4c03171
--- /dev/null
+++ b/changelog.d/8074.misc
@@ -0,0 +1 @@
+Convert various parts of the codebase to async/await.
diff --git a/changelog.d/8075.misc b/changelog.d/8075.misc
new file mode 100644

index 0000000000..dfe4c03171
--- /dev/null
+++ b/changelog.d/8075.misc
@@ -0,0 +1 @@
+Convert various parts of the codebase to async/await.
diff --git a/changelog.d/8076.misc b/changelog.d/8076.misc
new file mode 100644

index 0000000000..dfe4c03171
--- /dev/null
+++ b/changelog.d/8076.misc
@@ -0,0 +1 @@
+Convert various parts of the codebase to async/await.
diff --git a/changelog.d/8081.bugfix b/changelog.d/8081.bugfix
new file mode 100644

index 0000000000..9ebcbf5b84
--- /dev/null
+++ b/changelog.d/8081.bugfix
@@ -0,0 +1 @@
+Fix `Re-starting finished log context PUT-nnnn` warning when event persistence failed.
diff --git a/changelog.d/8085.misc b/changelog.d/8085.misc
new file mode 100644

index 0000000000..c3da1e297c
--- /dev/null
+++ b/changelog.d/8085.misc
@@ -0,0 +1 @@
+Remove some unused database functions.
diff --git a/changelog.d/8087.misc b/changelog.d/8087.misc
new file mode 100644

index 0000000000..dfe4c03171
--- /dev/null
+++ b/changelog.d/8087.misc
@@ -0,0 +1 @@
+Convert various parts of the codebase to async/await.
diff --git a/changelog.d/8090.misc b/changelog.d/8090.misc
new file mode 100644

index 0000000000..725a03ae88
--- /dev/null
+++ b/changelog.d/8090.misc
@@ -0,0 +1 @@
+Add type hints to `synapse.handlers.room`.
diff --git a/changelog.d/8092.feature b/changelog.d/8092.feature
new file mode 100644

index 0000000000..813e6d0903
--- /dev/null
+++ b/changelog.d/8092.feature
@@ -0,0 +1 @@
+Add support for shadow-banning users (ignoring any message send requests).
diff --git a/changelog.d/8093.misc b/changelog.d/8093.misc
new file mode 100644

index 0000000000..80045dde1a
--- /dev/null
+++ b/changelog.d/8093.misc
@@ -0,0 +1 @@
+Return the previous stream token if a non-member event is a duplicate.
diff --git a/changelog.d/8095.feature b/changelog.d/8095.feature
new file mode 100644

index 0000000000..813e6d0903
--- /dev/null
+++ b/changelog.d/8095.feature
@@ -0,0 +1 @@
+Add support for shadow-banning users (ignoring any message send requests).
diff --git a/changelog.d/8100.misc b/changelog.d/8100.misc
new file mode 100644

index 0000000000..dfe4c03171
--- /dev/null
+++ b/changelog.d/8100.misc
@@ -0,0 +1 @@
+Convert various parts of the codebase to async/await.
diff --git a/changelog.d/8101.bugfix b/changelog.d/8101.bugfix
new file mode 100644

index 0000000000..703bba4234
--- /dev/null
+++ b/changelog.d/8101.bugfix
@@ -0,0 +1 @@
+Synapse now correctly enforces the valid characters in the `client_secret` parameter used in various endpoints.
diff --git a/changelog.d/8104.bugfix b/changelog.d/8104.bugfix
new file mode 100644

index 0000000000..e32e2996c4
--- /dev/null
+++ b/changelog.d/8104.bugfix
@@ -0,0 +1 @@
+Fix a bug introduced in v1.7.2 impacting message retention policies that would allow federated homeservers to dictate a retention period that's lower than the configured minimum allowed duration in the configuration file.
diff --git a/changelog.d/8106.bugfix b/changelog.d/8106.bugfix
new file mode 100644

index 0000000000..c46c60448f
--- /dev/null
+++ b/changelog.d/8106.bugfix
@@ -0,0 +1 @@
+Fix a long-standing bug where invalid JSON would be accepted by Synapse.
diff --git a/changelog.d/8107.feature b/changelog.d/8107.feature
new file mode 100644

index 0000000000..2e5127477d
--- /dev/null
+++ b/changelog.d/8107.feature
@@ -0,0 +1 @@
+Use the default template file when its equivalent is not found in a custom template directory.
\ No newline at end of file
diff --git a/changelog.d/8110.bugfix b/changelog.d/8110.bugfix
new file mode 100644

index 0000000000..5269a232e1
--- /dev/null
+++ b/changelog.d/8110.bugfix
@@ -0,0 +1 @@
+Fix a bug introduced in Synapse 1.12.0 which could cause `/sync` requests to fail with a 404 if you had a very old outstanding room invite.
diff --git a/changelog.d/8111.doc b/changelog.d/8111.doc
new file mode 100644

index 0000000000..d3f7435452
--- /dev/null
+++ b/changelog.d/8111.doc
@@ -0,0 +1 @@
+Link to matrix-synapse-rest-password-provider in the password provider documentation.
diff --git a/changelog.d/8112.misc b/changelog.d/8112.misc
new file mode 100644

index 0000000000..80045dde1a
--- /dev/null
+++ b/changelog.d/8112.misc
@@ -0,0 +1 @@
+Return the previous stream token if a non-member event is a duplicate.
diff --git a/changelog.d/8113.misc b/changelog.d/8113.misc
new file mode 100644

index 0000000000..00bec4f8ef
--- /dev/null
+++ b/changelog.d/8113.misc
@@ -0,0 +1 @@
+Separate `get_current_token` into two since there are two different use cases for it.
diff --git a/changelog.d/8116.feature b/changelog.d/8116.feature
new file mode 100644

index 0000000000..b1eaf1e78a
--- /dev/null
+++ b/changelog.d/8116.feature
@@ -0,0 +1 @@
+Iteratively encode JSON to avoid blocking the reactor.
diff --git a/changelog.d/8119.misc b/changelog.d/8119.misc
new file mode 100644

index 0000000000..dfe4c03171
--- /dev/null
+++ b/changelog.d/8119.misc
@@ -0,0 +1 @@
+Convert various parts of the codebase to async/await.
diff --git a/changelog.d/8120.doc b/changelog.d/8120.doc
new file mode 100644

index 0000000000..877ef79fd2
--- /dev/null
+++ b/changelog.d/8120.doc
@@ -0,0 +1 @@
+Updated documentation to note that Synapse does not follow `HTTP 308` redirects due to an upstream library not supporting them. Contributed by Ryan Cole.
\ No newline at end of file
diff --git a/changelog.d/8121.misc b/changelog.d/8121.misc
new file mode 100644

index 0000000000..dfe4c03171
--- /dev/null
+++ b/changelog.d/8121.misc
@@ -0,0 +1 @@
+Convert various parts of the codebase to async/await.
diff --git a/changelog.d/8123.misc b/changelog.d/8123.misc
new file mode 100644

index 0000000000..7245122896
--- /dev/null
+++ b/changelog.d/8123.misc
@@ -0,0 +1 @@
+Remove `ChainedIdGenerator`.
diff --git a/changelog.d/8124.misc b/changelog.d/8124.misc
new file mode 100644

index 0000000000..9fac710205
--- /dev/null
+++ b/changelog.d/8124.misc
@@ -0,0 +1 @@
+Reduce the amount of whitespace in JSON stored and sent in responses.
diff --git a/changelog.d/8127.misc b/changelog.d/8127.misc
new file mode 100644

index 0000000000..cb557122aa
--- /dev/null
+++ b/changelog.d/8127.misc
@@ -0,0 +1 @@
+Add type hints to `synapse.storage.database`.
diff --git a/changelog.d/8129.bugfix b/changelog.d/8129.bugfix
new file mode 100644

index 0000000000..79eae9db6b
--- /dev/null
+++ b/changelog.d/8129.bugfix
@@ -0,0 +1 @@
+Return a proper error code when the rooms of an invalid group are requested.
diff --git a/changelog.d/8131.bugfix b/changelog.d/8131.bugfix
new file mode 100644

index 0000000000..5110f235d1
--- /dev/null
+++ b/changelog.d/8131.bugfix
@@ -0,0 +1 @@
+Fix a bug which could cause a leaked postgres connection if synapse was set to daemonize.
diff --git a/changelog.d/8132.misc b/changelog.d/8132.misc
new file mode 100644

index 0000000000..7afa267c69
--- /dev/null
+++ b/changelog.d/8132.misc
@@ -0,0 +1 @@
+Micro-optimisations to get_auth_chain_ids.
diff --git a/changelog.d/8133.misc b/changelog.d/8133.misc
new file mode 100644

index 0000000000..dfe4c03171
--- /dev/null
+++ b/changelog.d/8133.misc
@@ -0,0 +1 @@
+Convert various parts of the codebase to async/await.
diff --git a/changelog.d/8135.bugfix b/changelog.d/8135.bugfix
new file mode 100644

index 0000000000..9d5c60ea00
--- /dev/null
+++ b/changelog.d/8135.bugfix
@@ -0,0 +1 @@
+Clarify the error code if a user tries to register with a numeric ID. This bug was introduced in v1.15.0.
diff --git a/changelog.d/8139.bugfix b/changelog.d/8139.bugfix
new file mode 100644

index 0000000000..21f65d87b7
--- /dev/null
+++ b/changelog.d/8139.bugfix
@@ -0,0 +1 @@
+Fixes a bug where appservices with ratelimiting disabled would still be ratelimited when joining rooms. This bug was introduced in v1.19.0. 
diff --git a/changelog.d/8140.misc b/changelog.d/8140.misc
new file mode 100644

index 0000000000..78d8834328
--- /dev/null
+++ b/changelog.d/8140.misc
@@ -0,0 +1 @@
+Add type hints to `synapse.state`.
diff --git a/changelog.d/8142.feature b/changelog.d/8142.feature
new file mode 100644

index 0000000000..813e6d0903
--- /dev/null
+++ b/changelog.d/8142.feature
@@ -0,0 +1 @@
+Add support for shadow-banning users (ignoring any message send requests).
diff --git a/changelog.d/8147.docker b/changelog.d/8147.docker
new file mode 100644

index 0000000000..dcc951d8f5
--- /dev/null
+++ b/changelog.d/8147.docker
@@ -0,0 +1 @@
+Added curl for healthcheck support and readme updates for the change. Contributed by @maquis196.
diff --git a/changelog.d/8152.feature b/changelog.d/8152.feature
new file mode 100644

index 0000000000..813e6d0903
--- /dev/null
+++ b/changelog.d/8152.feature
@@ -0,0 +1 @@
+Add support for shadow-banning users (ignoring any message send requests).
diff --git a/changelog.d/8158.feature b/changelog.d/8158.feature
new file mode 100644

index 0000000000..47c4c39167
--- /dev/null
+++ b/changelog.d/8158.feature
@@ -0,0 +1 @@
+ Add support for shadow-banning users (ignoring any message send requests).
diff --git a/changelog.d/8161.misc b/changelog.d/8161.misc
new file mode 100644

index 0000000000..89ff274de3
--- /dev/null
+++ b/changelog.d/8161.misc
@@ -0,0 +1 @@
+Refactor `StreamIdGenerator` and `MultiWriterIdGenerator` to have the same interface.
diff --git a/changelog.d/8163.misc b/changelog.d/8163.misc
new file mode 100644

index 0000000000..b3ec08855b
--- /dev/null
+++ b/changelog.d/8163.misc
@@ -0,0 +1 @@
+Add filter `name` to the `/users` admin API, which filters by user ID or displayname. Contributed by Awesome Technologies Innovationslabor GmbH.
diff --git a/changelog.d/8164.misc b/changelog.d/8164.misc
new file mode 100644

index 0000000000..55bc079cdb
--- /dev/null
+++ b/changelog.d/8164.misc
@@ -0,0 +1 @@
+Add functions to `MultiWriterIdGen` used by events stream.