Merge tag 'v0.33.3'

Features
--------

- Add support for the SNI extension to federation TLS connections. Thanks to @vojeroen! ([\#3439](https://github.com/matrix-org/synapse/issues/3439))
- Add /_media/r0/config ([\#3184](https://github.com/matrix-org/synapse/issues/3184))
- speed up /members API and add `at` and `membership` params as per MSC1227 ([\#3568](https://github.com/matrix-org/synapse/issues/3568))
- implement `summary` block in /sync response as per MSC688 ([\#3574](https://github.com/matrix-org/synapse/issues/3574))
- Add lazy-loading support to /messages as per MSC1227 ([\#3589](https://github.com/matrix-org/synapse/issues/3589))
- Add ability to limit number of monthly active users on the server ([\#3633](https://github.com/matrix-org/synapse/issues/3633))
- Support more federation endpoints on workers ([\#3653](https://github.com/matrix-org/synapse/issues/3653))
- Basic support for room versioning ([\#3654](https://github.com/matrix-org/synapse/issues/3654))
- Ability to disable client/server Synapse via conf toggle ([\#3655](https://github.com/matrix-org/synapse/issues/3655))
- Ability to whitelist specific threepids against monthly active user limiting ([\#3662](https://github.com/matrix-org/synapse/issues/3662))
- Add some metrics for the appservice and federation event sending loops ([\#3664](https://github.com/matrix-org/synapse/issues/3664))
- Where server is disabled, block ability for locked out users to read new messages ([\#3670](https://github.com/matrix-org/synapse/issues/3670))
- set admin uri via config, to be used in error messages where the user should contact the administrator ([\#3687](https://github.com/matrix-org/synapse/issues/3687))
- Synapse's presence functionality can now be disabled with the "use_presence" configuration option. ([\#3694](https://github.com/matrix-org/synapse/issues/3694))
- For resource limit blocked users, prevent writing into rooms ([\#3708](https://github.com/matrix-org/synapse/issues/3708))

Bugfixes
--------

- Fix occasional glitches in the synapse_event_persisted_position metric ([\#3658](https://github.com/matrix-org/synapse/issues/3658))
- Fix bug on deleting 3pid when using identity servers that don't support unbind API ([\#3661](https://github.com/matrix-org/synapse/issues/3661))
- Make the tests pass on Twisted < 18.7.0 ([\#3676](https://github.com/matrix-org/synapse/issues/3676))
- Don’t ship recaptcha_ajax.js, use it directly from Google ([\#3677](https://github.com/matrix-org/synapse/issues/3677))
- Fixes test_reap_monthly_active_users so it passes under postgres ([\#3681](https://github.com/matrix-org/synapse/issues/3681))
- Fix mau blocking calulation bug on login ([\#3689](https://github.com/matrix-org/synapse/issues/3689))
- Fix missing yield in synapse.storage.monthly_active_users.initialise_reserved_users ([\#3692](https://github.com/matrix-org/synapse/issues/3692))
- Improve HTTP request logging to include all requests ([\#3700](https://github.com/matrix-org/synapse/issues/3700))
- Avoid timing out requests while we are streaming back the response ([\#3701](https://github.com/matrix-org/synapse/issues/3701))
- Support more federation endpoints on workers ([\#3705](https://github.com/matrix-org/synapse/issues/3705), [\#3713](https://github.com/matrix-org/synapse/issues/3713))
- Fix "Starting db txn 'get_all_updated_receipts' from sentinel context" warning ([\#3710](https://github.com/matrix-org/synapse/issues/3710))
- Fix bug where `state_cache` cache factor ignored environment variables ([\#3719](https://github.com/matrix-org/synapse/issues/3719))
- Fix bug in v0.33.3rc1 which caused infinite loops and OOMs ([\#3723](https://github.com/matrix-org/synapse/issues/3723))
- Fix bug introduced in v0.33.3rc1 which made the ToS give a 500 error ([\#3732](https://github.com/matrix-org/synapse/issues/3732))

Deprecations and Removals
-------------------------

- The Shared-Secret registration method of the legacy v1/register REST endpoint has been removed. For a replacement, please see [the admin/register API documentation](https://github.com/matrix-org/synapse/blob/master/docs/admin_api/register_api.rst). ([\#3703](https://github.com/matrix-org/synapse/issues/3703))

Internal Changes
----------------

- The test suite now can run under PostgreSQL. ([\#3423](https://github.com/matrix-org/synapse/issues/3423))
- Refactor HTTP replication endpoints to reduce code duplication ([\#3632](https://github.com/matrix-org/synapse/issues/3632))
- Tests now correctly execute on Python 3. ([\#3647](https://github.com/matrix-org/synapse/issues/3647))
- Sytests can now be run inside a Docker container. ([\#3660](https://github.com/matrix-org/synapse/issues/3660))
- Port over enough to Python 3 to allow the sytests to start. ([\#3668](https://github.com/matrix-org/synapse/issues/3668))
- Update docker base image from alpine 3.7 to 3.8. ([\#3669](https://github.com/matrix-org/synapse/issues/3669))
- Rename synapse.util.async to synapse.util.async_helpers to mitigate async becoming a keyword on Python 3.7. ([\#3678](https://github.com/matrix-org/synapse/issues/3678))
- Synapse's tests are now formatted with the black autoformatter. ([\#3679](https://github.com/matrix-org/synapse/issues/3679))
- Implemented a new testing base class to reduce test boilerplate. ([\#3684](https://github.com/matrix-org/synapse/issues/3684))
- Rename MAU prometheus metrics ([\#3690](https://github.com/matrix-org/synapse/issues/3690))
- add new error type ResourceLimit ([\#3707](https://github.com/matrix-org/synapse/issues/3707))
- Logcontexts for replication command handlers ([\#3709](https://github.com/matrix-org/synapse/issues/3709))
- Update admin register API documentation to reference a real user ID. ([\#3712](https://github.com/matrix-org/synapse/issues/3712))

1 files changed, 75 insertions, 72 deletions

diff --git a/tests/test_visibility.py b/tests/test_visibility.py
index 0dc1a924d3..45a78338d6 100644
--- a/tests/test_visibility.py
+++ b/tests/test_visibility.py
@@ -31,7 +31,7 @@ TEST_ROOM_ID = "!TEST:ROOM"
 class FilterEventsForServerTestCase(tests.unittest.TestCase):
     @defer.inlineCallbacks
     def setUp(self):
-        self.hs = yield setup_test_homeserver()
+        self.hs = yield setup_test_homeserver(self.addCleanup)
         self.event_creation_handler = self.hs.get_event_creation_handler()
         self.event_builder_factory = self.hs.get_event_builder_factory()
         self.store = self.hs.get_datastore()
@@ -54,14 +54,12 @@ class FilterEventsForServerTestCase(tests.unittest.TestCase):
         events_to_filter = []
 
         for i in range(0, 10):
-            user = "@user%i:%s" % (
-                i, "test_server" if i == 5 else "other_server"
-            )
+            user = "@user%i:%s" % (i, "test_server" if i == 5 else "other_server")
             evt = yield self.inject_room_member(user, extra_content={"a": "b"})
             events_to_filter.append(evt)
 
         filtered = yield filter_events_for_server(
-            self.store, "test_server", events_to_filter,
+            self.store, "test_server", events_to_filter
         )
 
         # the result should be 5 redacted events, and 5 unredacted events.
@@ -100,19 +98,21 @@ class FilterEventsForServerTestCase(tests.unittest.TestCase):
 
         # ... and the filtering happens.
         filtered = yield filter_events_for_server(
-            self.store, "test_server", events_to_filter,
+            self.store, "test_server", events_to_filter
         )
 
         for i in range(0, len(events_to_filter)):
             self.assertEqual(
-                events_to_filter[i].event_id, filtered[i].event_id,
-                "Unexpected event at result position %i" % (i, )
+                events_to_filter[i].event_id,
+                filtered[i].event_id,
+                "Unexpected event at result position %i" % (i,),
             )
 
         for i in (0, 3):
             self.assertEqual(
-                events_to_filter[i].content["body"], filtered[i].content["body"],
-                "Unexpected event content at result position %i" % (i,)
+                events_to_filter[i].content["body"],
+                filtered[i].content["body"],
+                "Unexpected event content at result position %i" % (i,),
             )
 
         for i in (1, 4):
@@ -121,13 +121,15 @@ class FilterEventsForServerTestCase(tests.unittest.TestCase):
     @defer.inlineCallbacks
     def inject_visibility(self, user_id, visibility):
         content = {"history_visibility": visibility}
-        builder = self.event_builder_factory.new({
-            "type": "m.room.history_visibility",
-            "sender": user_id,
-            "state_key": "",
-            "room_id": TEST_ROOM_ID,
-            "content": content,
-        })
+        builder = self.event_builder_factory.new(
+            {
+                "type": "m.room.history_visibility",
+                "sender": user_id,
+                "state_key": "",
+                "room_id": TEST_ROOM_ID,
+                "content": content,
+            }
+        )
 
         event, context = yield self.event_creation_handler.create_new_client_event(
             builder
@@ -139,13 +141,15 @@ class FilterEventsForServerTestCase(tests.unittest.TestCase):
     def inject_room_member(self, user_id, membership="join", extra_content={}):
         content = {"membership": membership}
         content.update(extra_content)
-        builder = self.event_builder_factory.new({
-            "type": "m.room.member",
-            "sender": user_id,
-            "state_key": user_id,
-            "room_id": TEST_ROOM_ID,
-            "content": content,
-        })
+        builder = self.event_builder_factory.new(
+            {
+                "type": "m.room.member",
+                "sender": user_id,
+                "state_key": user_id,
+                "room_id": TEST_ROOM_ID,
+                "content": content,
+            }
+        )
 
         event, context = yield self.event_creation_handler.create_new_client_event(
             builder
@@ -158,12 +162,14 @@ class FilterEventsForServerTestCase(tests.unittest.TestCase):
     def inject_message(self, user_id, content=None):
         if content is None:
             content = {"body": "testytest"}
-        builder = self.event_builder_factory.new({
-            "type": "m.room.message",
-            "sender": user_id,
-            "room_id": TEST_ROOM_ID,
-            "content": content,
-        })
+        builder = self.event_builder_factory.new(
+            {
+                "type": "m.room.message",
+                "sender": user_id,
+                "room_id": TEST_ROOM_ID,
+                "content": content,
+            }
+        )
 
         event, context = yield self.event_creation_handler.create_new_client_event(
             builder
@@ -192,56 +198,54 @@ class FilterEventsForServerTestCase(tests.unittest.TestCase):
         # history_visibility event.
         room_state = []
 
-        history_visibility_evt = FrozenEvent({
-            "event_id": "$history_vis",
-            "type": "m.room.history_visibility",
-            "sender": "@resident_user_0:test.com",
-            "state_key": "",
-            "room_id": TEST_ROOM_ID,
-            "content": {"history_visibility": "joined"},
-        })
+        history_visibility_evt = FrozenEvent(
+            {
+                "event_id": "$history_vis",
+                "type": "m.room.history_visibility",
+                "sender": "@resident_user_0:test.com",
+                "state_key": "",
+                "room_id": TEST_ROOM_ID,
+                "content": {"history_visibility": "joined"},
+            }
+        )
         room_state.append(history_visibility_evt)
         test_store.add_event(history_visibility_evt)
 
         for i in range(0, 100000):
-            user = "@resident_user_%i:test.com" % (i, )
-            evt = FrozenEvent({
-                "event_id": "$res_event_%i" % (i, ),
-                "type": "m.room.member",
-                "state_key": user,
-                "sender": user,
-                "room_id": TEST_ROOM_ID,
-                "content": {
-                    "membership": "join",
-                    "extra": "zzz,"
-                },
-            })
+            user = "@resident_user_%i:test.com" % (i,)
+            evt = FrozenEvent(
+                {
+                    "event_id": "$res_event_%i" % (i,),
+                    "type": "m.room.member",
+                    "state_key": user,
+                    "sender": user,
+                    "room_id": TEST_ROOM_ID,
+                    "content": {"membership": "join", "extra": "zzz,"},
+                }
+            )
             room_state.append(evt)
             test_store.add_event(evt)
 
         events_to_filter = []
         for i in range(0, 10):
-            user = "@user%i:%s" % (
-                i, "test_server" if i == 5 else "other_server"
+            user = "@user%i:%s" % (i, "test_server" if i == 5 else "other_server")
+            evt = FrozenEvent(
+                {
+                    "event_id": "$evt%i" % (i,),
+                    "type": "m.room.member",
+                    "state_key": user,
+                    "sender": user,
+                    "room_id": TEST_ROOM_ID,
+                    "content": {"membership": "join", "extra": "zzz"},
+                }
             )
-            evt = FrozenEvent({
-                "event_id": "$evt%i" % (i, ),
-                "type": "m.room.member",
-                "state_key": user,
-                "sender": user,
-                "room_id": TEST_ROOM_ID,
-                "content": {
-                    "membership": "join",
-                    "extra": "zzz",
-                },
-            })
             events_to_filter.append(evt)
             room_state.append(evt)
 
             test_store.add_event(evt)
-            test_store.set_state_ids_for_event(evt, {
-                (e.type, e.state_key): e.event_id for e in room_state
-            })
+            test_store.set_state_ids_for_event(
+                evt, {(e.type, e.state_key): e.event_id for e in room_state}
+            )
 
         pr = cProfile.Profile()
         pr.enable()
@@ -249,7 +253,7 @@ class FilterEventsForServerTestCase(tests.unittest.TestCase):
         logger.info("Starting filtering")
         start = time.time()
         filtered = yield filter_events_for_server(
-            test_store, "test_server", events_to_filter,
+            test_store, "test_server", events_to_filter
         )
         logger.info("Filtering took %f seconds", time.time() - start)
 
@@ -275,6 +279,7 @@ class _TestStore(object):
     filter_events_for_server
 
     """
+
     def __init__(self):
         # data for get_events: a map from event_id to event
         self.events = {}
@@ -298,8 +303,8 @@ class _TestStore(object):
                 continue
             if type != "m.room.member" or state_key is not None:
                 raise RuntimeError(
-                    "Unimplemented: get_state_ids with type (%s, %s)" %
-                    (type, state_key),
+                    "Unimplemented: get_state_ids with type (%s, %s)"
+                    % (type, state_key)
                 )
             include_memberships = True
 
@@ -316,9 +321,7 @@ class _TestStore(object):
         return succeed(res)
 
     def get_events(self, events):
-        return succeed({
-            event_id: self.events[event_id] for event_id in events
-        })
+        return succeed({event_id: self.events[event_id] for event_id in events})
 
     def are_users_erased(self, users):
         return succeed({u: False for u in users})