Merge branch 'release-v1.83' into develop

42 files changed, 77 insertions, 64 deletions

diff --git a/CHANGES.md b/CHANGES.md
index b2cc138ee3..f055772ca0 100644
--- a/CHANGES.md
+++ b/CHANGES.md
@@ -1,3 +1,56 @@
+Synapse 1.83.0rc1 (2023-05-02)
+==============================
+
+Features
+--------
+
+- Experimental support to recursively provide relations per [MSC3981](https://github.com/matrix-org/matrix-spec-proposals/pull/3981). ([\#15315](https://github.com/matrix-org/synapse/issues/15315))
+- Experimental support for [MSC3970](https://github.com/matrix-org/matrix-spec-proposals/pull/3970): Scope transaction IDs to devices. ([\#15318](https://github.com/matrix-org/synapse/issues/15318))
+- Add an [admin API endpoint](https://matrix-org.github.io/synapse/v1.83/admin_api/experimental_features.html) to support per-user feature flags. ([\#15344](https://github.com/matrix-org/synapse/issues/15344))
+- Add a module API to send an HTTP push notification. ([\#15387](https://github.com/matrix-org/synapse/issues/15387))
+- Add an [admin API endpoint](https://matrix-org.github.io/synapse/v1.83/admin_api/statistics.html#get-largest-rooms-by-size-in-database) to query the largest rooms by disk space used in the database. ([\#15482](https://github.com/matrix-org/synapse/issues/15482))
+
+
+Bugfixes
+--------
+
+- Disable push rule evaluation for rooms excluded from sync. ([\#15361](https://github.com/matrix-org/synapse/issues/15361))
+- Fix a long-standing bug where cached server key results which were directly fetched would not be properly re-used. ([\#15417](https://github.com/matrix-org/synapse/issues/15417))
+- Fix a bug introduced in Synapse 1.73.0 where some experimental push rules were returned by default. ([\#15494](https://github.com/matrix-org/synapse/issues/15494))
+
+
+Improved Documentation
+----------------------
+
+- Add Nginx loadbalancing example with sticky mxid for workers. ([\#15411](https://github.com/matrix-org/synapse/issues/15411))
+- Update outdated development docs that mention restrictions in versions of SQLite that we no longer support. ([\#15498](https://github.com/matrix-org/synapse/issues/15498))
+
+
+Internal Changes
+----------------
+
+- Speedup tests by caching HomeServerConfig instances. ([\#15284](https://github.com/matrix-org/synapse/issues/15284))
+- Add denormalised event stream ordering column to membership state tables for future use. Contributed by Nick @ Beeper (@fizzadar). ([\#15356](https://github.com/matrix-org/synapse/issues/15356))
+- Always use multi-user device resync replication endpoints. ([\#15418](https://github.com/matrix-org/synapse/issues/15418))
+- Add column `full_user_id` to tables `profiles` and `user_filters`. ([\#15458](https://github.com/matrix-org/synapse/issues/15458))
+- Update support for [MSC3983](https://github.com/matrix-org/matrix-spec-proposals/pull/3983) to allow always returning fallback-keys in a `/keys/claim` request. ([\#15462](https://github.com/matrix-org/synapse/issues/15462))
+- Improve type hints. ([\#15465](https://github.com/matrix-org/synapse/issues/15465), [\#15496](https://github.com/matrix-org/synapse/issues/15496), [\#15497](https://github.com/matrix-org/synapse/issues/15497))
+- Support claiming more than one OTK at a time. ([\#15468](https://github.com/matrix-org/synapse/issues/15468))
+- Bump types-pyyaml from 6.0.12.8 to 6.0.12.9. ([\#15471](https://github.com/matrix-org/synapse/issues/15471))
+- Bump pyasn1-modules from 0.2.8 to 0.3.0. ([\#15473](https://github.com/matrix-org/synapse/issues/15473))
+- Bump cryptography from 40.0.1 to 40.0.2. ([\#15474](https://github.com/matrix-org/synapse/issues/15474))
+- Bump types-netaddr from 0.8.0.7 to 0.8.0.8. ([\#15475](https://github.com/matrix-org/synapse/issues/15475))
+- Bump types-jsonschema from 4.17.0.6 to 4.17.0.7. ([\#15476](https://github.com/matrix-org/synapse/issues/15476))
+- Ask bug reporters to provide logs as text. ([\#15479](https://github.com/matrix-org/synapse/issues/15479))
+- Add a Nix flake for use as a development environment. ([\#15495](https://github.com/matrix-org/synapse/issues/15495))
+- Bump anyhow from 1.0.70 to 1.0.71. ([\#15507](https://github.com/matrix-org/synapse/issues/15507))
+- Bump types-pillow from 9.4.0.19 to 9.5.0.2. ([\#15508](https://github.com/matrix-org/synapse/issues/15508))
+- Bump packaging from 23.0 to 23.1. ([\#15510](https://github.com/matrix-org/synapse/issues/15510))
+- Bump types-requests from 2.28.11.16 to 2.29.0.0. ([\#15511](https://github.com/matrix-org/synapse/issues/15511))
+- Bump setuptools-rust from 1.5.2 to 1.6.0. ([\#15512](https://github.com/matrix-org/synapse/issues/15512))
+- Update the check_schema_delta script to account for when the schema version has been bumped locally. ([\#15466](https://github.com/matrix-org/synapse/issues/15466))
+
+
 Synapse 1.82.0 (2023-04-25)
 ===========================
 
diff --git a/changelog.d/15284.misc b/changelog.d/15284.misc
deleted file mode 100644
index 99d753f8f0..0000000000
--- a/changelog.d/15284.misc
+++ /dev/null
@@ -1 +0,0 @@
-Speedup tests by caching HomeServerConfig instances.
diff --git a/changelog.d/15315.feature b/changelog.d/15315.feature
deleted file mode 100644
index 30b2abdc62..0000000000
--- a/changelog.d/15315.feature
+++ /dev/null
@@ -1 +0,0 @@
-Experimental support to recursively provide relations per [MSC3981](https://github.com/matrix-org/matrix-spec-proposals/pull/3981).
diff --git a/changelog.d/15318.feature b/changelog.d/15318.feature
deleted file mode 100644
index 47bb2e17a7..0000000000
--- a/changelog.d/15318.feature
+++ /dev/null
@@ -1 +0,0 @@
-Experimental support for MSC3970: Scope transaction IDs to devices.
diff --git a/changelog.d/15344.feature b/changelog.d/15344.feature
deleted file mode 100644
index 44262e9bd8..0000000000
--- a/changelog.d/15344.feature
+++ /dev/null
@@ -1 +0,0 @@
-Add an admin API endpoint to support per-user feature flags.
diff --git a/changelog.d/15356.misc b/changelog.d/15356.misc
deleted file mode 100644
index c09911e48d..0000000000
--- a/changelog.d/15356.misc
+++ /dev/null
@@ -1 +0,0 @@
-Add denormalised event stream ordering column to membership state tables for future use. Contributed by Nick @ Beeper (@fizzadar).
diff --git a/changelog.d/15361.bugfix b/changelog.d/15361.bugfix
deleted file mode 100644
index 2cd795e576..0000000000
--- a/changelog.d/15361.bugfix
+++ /dev/null
@@ -1 +0,0 @@
-Disable push rule evaluation for rooms excluded from sync.
\ No newline at end of file
diff --git a/changelog.d/15387.feature b/changelog.d/15387.feature
deleted file mode 100644
index b36e331520..0000000000
--- a/changelog.d/15387.feature
+++ /dev/null
@@ -1 +0,0 @@
-Add a module API to send an HTTP push notification.
diff --git a/changelog.d/15411.doc b/changelog.d/15411.doc
deleted file mode 100644
index c23a8df04a..0000000000
--- a/changelog.d/15411.doc
+++ /dev/null
@@ -1 +0,0 @@
-Docs: Add Nginx loadbalancing example with sticky mxid for workers.
diff --git a/changelog.d/15417.bugfix b/changelog.d/15417.bugfix
deleted file mode 100644
index 300635cbdc..0000000000
--- a/changelog.d/15417.bugfix
+++ /dev/null
@@ -1 +0,0 @@
-Fix a long-standing bug where cached key results which were directly fetched would not be properly re-used.
diff --git a/changelog.d/15418.misc b/changelog.d/15418.misc
deleted file mode 100644
index ca6f995a9c..0000000000
--- a/changelog.d/15418.misc
+++ /dev/null
@@ -1 +0,0 @@
-Always use multi-user device resync replication endpoints.
\ No newline at end of file
diff --git a/changelog.d/15458.misc b/changelog.d/15458.misc
deleted file mode 100644
index 5183161d25..0000000000
--- a/changelog.d/15458.misc
+++ /dev/null
@@ -1 +0,0 @@
-Add column `full_user_id` to tables `profiles` and `user_filters`.
diff --git a/changelog.d/15462.misc b/changelog.d/15462.misc
deleted file mode 100644
index 36e4bffbc8..0000000000
--- a/changelog.d/15462.misc
+++ /dev/null
@@ -1 +0,0 @@
-Update support for [MSC3983](https://github.com/matrix-org/matrix-spec-proposals/pull/3983) to allow always returning fallback-keys in a `/keys/claim` request.
diff --git a/changelog.d/15465.misc b/changelog.d/15465.misc
deleted file mode 100644
index 93ceaeafc9..0000000000
--- a/changelog.d/15465.misc
+++ /dev/null
@@ -1 +0,0 @@
-Improve type hints.
diff --git a/changelog.d/15466.bugfix b/changelog.d/15466.bugfix
deleted file mode 100644
index 299a7f0f0e..0000000000
--- a/changelog.d/15466.bugfix
+++ /dev/null
@@ -1 +0,0 @@
-Update the check_schema_delta script to account for when the schema version has been bumped locally.
diff --git a/changelog.d/15468.misc b/changelog.d/15468.misc
deleted file mode 100644
index e0a94f36fd..0000000000
--- a/changelog.d/15468.misc
+++ /dev/null
@@ -1 +0,0 @@
-Support claiming more than one OTK at a time.
diff --git a/changelog.d/15471.misc b/changelog.d/15471.misc
deleted file mode 100644
index c5b16174c5..0000000000
--- a/changelog.d/15471.misc
+++ /dev/null
@@ -1 +0,0 @@
-Bump types-pyyaml from 6.0.12.8 to 6.0.12.9.
diff --git a/changelog.d/15473.misc b/changelog.d/15473.misc
deleted file mode 100644
index 9540fbc1d2..0000000000
--- a/changelog.d/15473.misc
+++ /dev/null
@@ -1 +0,0 @@
-Bump pyasn1-modules from 0.2.8 to 0.3.0.
diff --git a/changelog.d/15474.misc b/changelog.d/15474.misc
deleted file mode 100644
index 61f4abd515..0000000000
--- a/changelog.d/15474.misc
+++ /dev/null
@@ -1 +0,0 @@
-Bump cryptography from 40.0.1 to 40.0.2.
diff --git a/changelog.d/15475.misc b/changelog.d/15475.misc
deleted file mode 100644
index c86462652c..0000000000
--- a/changelog.d/15475.misc
+++ /dev/null
@@ -1 +0,0 @@
-Bump types-netaddr from 0.8.0.7 to 0.8.0.8.
diff --git a/changelog.d/15476.misc b/changelog.d/15476.misc
deleted file mode 100644
index 2ff909dffb..0000000000
--- a/changelog.d/15476.misc
+++ /dev/null
@@ -1 +0,0 @@
-Bump types-jsonschema from 4.17.0.6 to 4.17.0.7.
diff --git a/changelog.d/15479.misc b/changelog.d/15479.misc
deleted file mode 100644
index 482addfeb7..0000000000
--- a/changelog.d/15479.misc
+++ /dev/null
@@ -1 +0,0 @@
-Ask bug reporters to provide logs as text.
diff --git a/changelog.d/15482.feature b/changelog.d/15482.feature
deleted file mode 100644
index f3e9f2a5b2..0000000000
--- a/changelog.d/15482.feature
+++ /dev/null
@@ -1 +0,0 @@
-Add admin endpoint to query the largest rooms by disk space used in the database.
diff --git a/changelog.d/15494.bugfix b/changelog.d/15494.bugfix
deleted file mode 100644
index 89f9262200..0000000000
--- a/changelog.d/15494.bugfix
+++ /dev/null
@@ -1 +0,0 @@
-Fix a bug introduced in Synapse 1.73.0 where some experimental push rules were returned by default.
diff --git a/changelog.d/15495.misc b/changelog.d/15495.misc
deleted file mode 100644
index ff7b5cbddf..0000000000
--- a/changelog.d/15495.misc
+++ /dev/null
@@ -1 +0,0 @@
-Add a Nix flake for use as a development environment.
\ No newline at end of file
diff --git a/changelog.d/15496.misc b/changelog.d/15496.misc
deleted file mode 100644
index 93ceaeafc9..0000000000
--- a/changelog.d/15496.misc
+++ /dev/null
@@ -1 +0,0 @@
-Improve type hints.
diff --git a/changelog.d/15497.misc b/changelog.d/15497.misc
deleted file mode 100644
index 93ceaeafc9..0000000000
--- a/changelog.d/15497.misc
+++ /dev/null
@@ -1 +0,0 @@
-Improve type hints.
diff --git a/changelog.d/15498.doc b/changelog.d/15498.doc
deleted file mode 100644
index 78715b8a3e..0000000000
--- a/changelog.d/15498.doc
+++ /dev/null
@@ -1 +0,0 @@
-Update outdated development docs that mention restrictions in versions of SQLite that we no longer support.
diff --git a/changelog.d/15507.misc b/changelog.d/15507.misc
deleted file mode 100644
index 061f19e1c9..0000000000
--- a/changelog.d/15507.misc
+++ /dev/null
@@ -1 +0,0 @@
-Bump anyhow from 1.0.70 to 1.0.71.
diff --git a/changelog.d/15508.misc b/changelog.d/15508.misc
deleted file mode 100644
index 3f3f5d98f9..0000000000
--- a/changelog.d/15508.misc
+++ /dev/null
@@ -1 +0,0 @@
-Bump types-pillow from 9.4.0.19 to 9.5.0.2.
diff --git a/changelog.d/15510.misc b/changelog.d/15510.misc
deleted file mode 100644
index bff25e824b..0000000000
--- a/changelog.d/15510.misc
+++ /dev/null
@@ -1 +0,0 @@
-Bump packaging from 23.0 to 23.1.
diff --git a/changelog.d/15511.misc b/changelog.d/15511.misc
deleted file mode 100644
index 183e0c69d2..0000000000
--- a/changelog.d/15511.misc
+++ /dev/null
@@ -1 +0,0 @@
-Bump types-requests from 2.28.11.16 to 2.29.0.0.
diff --git a/changelog.d/15512.misc b/changelog.d/15512.misc
deleted file mode 100644
index 90bdf170d3..0000000000
--- a/changelog.d/15512.misc
+++ /dev/null
@@ -1 +0,0 @@
-Bump setuptools-rust from 1.5.2 to 1.6.0.
diff --git a/changelog.d/15514.misc b/changelog.d/15514.misc
deleted file mode 100644
index f24c18cf47..0000000000
--- a/changelog.d/15514.misc
+++ /dev/null
@@ -1 +0,0 @@
-Reduce the size of the HTTP connection pool for non-pushers.
diff --git a/debian/changelog b/debian/changelog
index f6e8720e58..a50fda69a8 100644
--- a/debian/changelog
+++ b/debian/changelog
@@ -1,3 +1,9 @@
+matrix-synapse-py3 (1.83.0~rc1) stable; urgency=medium
+
+  * New Synapse release 1.83.0rc1.
+
+ -- Synapse Packaging team <packages@matrix.org>  Tue, 02 May 2023 15:56:38 +0100
+
 matrix-synapse-py3 (1.82.0) stable; urgency=medium
 
   * New Synapse release 1.82.0.
diff --git a/docs/SUMMARY.md b/docs/SUMMARY.md
index ade77d4926..a8e5ddad9d 100644
--- a/docs/SUMMARY.md
+++ b/docs/SUMMARY.md
@@ -57,6 +57,7 @@
       - [Account Validity](admin_api/account_validity.md)
       - [Background Updates](usage/administration/admin_api/background_updates.md)
       - [Event Reports](admin_api/event_reports.md)
+      - [Experimental Features](admin_api/experimental_features.md)
       - [Media](admin_api/media_admin_api.md)
       - [Purge History](admin_api/purge_history_api.md)
       - [Register Users](admin_api/register_api.md)
diff --git a/pyproject.toml b/pyproject.toml
index c08352e4d3..caf69cc53f 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -89,7 +89,7 @@ manifest-path = "rust/Cargo.toml"
 
 [tool.poetry]
 name = "matrix-synapse"
-version = "1.82.0"
+version = "1.83.0rc1"
 description = "Homeserver for the Matrix decentralised comms protocol"
 authors = ["Matrix.org Team and Contributors <packages@matrix.org>"]
 license = "Apache-2.0"
diff --git a/synapse/http/client.py b/synapse/http/client.py
index 164abe9fc7..91fe474f36 100644
--- a/synapse/http/client.py
+++ b/synapse/http/client.py
@@ -768,7 +768,6 @@ class SimpleHttpClient(BaseHttpClient):
            request if it were otherwise caught in a blacklist.
         use_proxy: Whether proxy settings should be discovered and used
             from conventional environment variables.
-        connection_pool: The connection pool to use for this client's agent.
     """
 
     def __init__(
@@ -778,7 +777,6 @@ class SimpleHttpClient(BaseHttpClient):
         ip_whitelist: Optional[IPSet] = None,
         ip_blacklist: Optional[IPSet] = None,
         use_proxy: bool = False,
-        connection_pool: Optional[HTTPConnectionPool] = None,
     ):
         super().__init__(hs, treq_args=treq_args)
         self._ip_whitelist = ip_whitelist
@@ -791,12 +789,22 @@ class SimpleHttpClient(BaseHttpClient):
                 self.reactor, self._ip_whitelist, self._ip_blacklist
             )
 
+        # the pusher makes lots of concurrent SSL connections to Sygnal, and tends to
+        # do so in batches, so we need to allow the pool to keep lots of idle
+        # connections around.
+        pool = HTTPConnectionPool(self.reactor)
+        # XXX: The justification for using the cache factor here is that larger
+        # instances will need both more cache and more connections.
+        # Still, this should probably be a separate dial
+        pool.maxPersistentPerHost = max(int(100 * hs.config.caches.global_factor), 5)
+        pool.cachedConnectionTimeout = 2 * 60
+
         self.agent: IAgent = ProxyAgent(
             self.reactor,
             hs.get_reactor(),
             connectTimeout=15,
             contextFactory=self.hs.get_http_client_context_factory(),
-            pool=connection_pool,
+            pool=pool,
             use_proxy=use_proxy,
         )
 
diff --git a/synapse/push/httppusher.py b/synapse/push/httppusher.py
index e628b484a9..e91ee05e99 100644
--- a/synapse/push/httppusher.py
+++ b/synapse/push/httppusher.py
@@ -143,8 +143,7 @@ class HttpPusher(Pusher):
             )
 
         self.url = url
-        self.http_client = hs.get_pusher_http_client()
-
+        self.http_client = hs.get_proxied_blacklisted_http_client()
         self.data_minus_url = {}
         self.data_minus_url.update(self.data)
         del self.data_minus_url["url"]
diff --git a/synapse/server.py b/synapse/server.py
index a0036578b1..e597627a6d 100644
--- a/synapse/server.py
+++ b/synapse/server.py
@@ -27,7 +27,6 @@ from typing_extensions import TypeAlias
 
 from twisted.internet.interfaces import IOpenSSLContextFactory
 from twisted.internet.tcp import Port
-from twisted.web.client import HTTPConnectionPool
 from twisted.web.iweb import IPolicyForHTTPS
 from twisted.web.resource import Resource
 
@@ -460,26 +459,6 @@ class HomeServer(metaclass=abc.ABCMeta):
         )
 
     @cache_in_self
-    def get_pusher_http_client(self) -> SimpleHttpClient:
-        # the pusher makes lots of concurrent SSL connections to Sygnal, and tends to
-        # do so in batches, so we need to allow the pool to keep lots of idle
-        # connections around.
-        pool = HTTPConnectionPool(self.get_reactor())
-        # XXX: The justification for using the cache factor here is that larger
-        # instances will need both more cache and more connections.
-        # Still, this should probably be a separate dial
-        pool.maxPersistentPerHost = max(int(100 * self.config.caches.global_factor), 5)
-        pool.cachedConnectionTimeout = 2 * 60
-
-        return SimpleHttpClient(
-            self,
-            ip_whitelist=self.config.server.ip_range_whitelist,
-            ip_blacklist=self.config.server.ip_range_blacklist,
-            use_proxy=True,
-            connection_pool=pool,
-        )
-
-    @cache_in_self
     def get_federation_http_client(self) -> MatrixFederationHttpClient:
         """
         An HTTP client for federation.
diff --git a/tests/push/test_http.py b/tests/push/test_http.py
index 4f811bb9c0..54f558742d 100644
--- a/tests/push/test_http.py
+++ b/tests/push/test_http.py
@@ -52,7 +52,7 @@ class HTTPPusherTests(HomeserverTestCase):
 
         m.post_json_get_json = post_json_get_json
 
-        hs = self.setup_test_homeserver(pusher_http_client=m)
+        hs = self.setup_test_homeserver(proxied_blacklisted_http_client=m)
 
         return hs
 
diff --git a/tests/replication/test_pusher_shard.py b/tests/replication/test_pusher_shard.py
index b9bb1a6497..dcb3e6669b 100644
--- a/tests/replication/test_pusher_shard.py
+++ b/tests/replication/test_pusher_shard.py
@@ -93,7 +93,7 @@ class PusherShardTestCase(BaseMultiWorkerStreamTestCase):
         self.make_worker_hs(
             "synapse.app.generic_worker",
             {"worker_name": "pusher1", "pusher_instances": ["pusher1"]},
-            pusher_http_client=http_client_mock,
+            proxied_blacklisted_http_client=http_client_mock,
         )
 
         event_id = self._create_pusher_and_send_msg("user")
@@ -126,7 +126,7 @@ class PusherShardTestCase(BaseMultiWorkerStreamTestCase):
                 "worker_name": "pusher1",
                 "pusher_instances": ["pusher1", "pusher2"],
             },
-            pusher_http_client=http_client_mock1,
+            proxied_blacklisted_http_client=http_client_mock1,
         )
 
         http_client_mock2 = Mock(spec_set=["post_json_get_json"])
@@ -140,7 +140,7 @@ class PusherShardTestCase(BaseMultiWorkerStreamTestCase):
                 "worker_name": "pusher2",
                 "pusher_instances": ["pusher1", "pusher2"],
             },
-            pusher_http_client=http_client_mock2,
+            proxied_blacklisted_http_client=http_client_mock2,
         )
 
         # We choose a user name that we know should go to pusher1.