Merge tag 'v0.33.3'

Features
--------

- Add support for the SNI extension to federation TLS connections. Thanks to @vojeroen! ([\#3439](https://github.com/matrix-org/synapse/issues/3439))
- Add /_media/r0/config ([\#3184](https://github.com/matrix-org/synapse/issues/3184))
- speed up /members API and add `at` and `membership` params as per MSC1227 ([\#3568](https://github.com/matrix-org/synapse/issues/3568))
- implement `summary` block in /sync response as per MSC688 ([\#3574](https://github.com/matrix-org/synapse/issues/3574))
- Add lazy-loading support to /messages as per MSC1227 ([\#3589](https://github.com/matrix-org/synapse/issues/3589))
- Add ability to limit number of monthly active users on the server ([\#3633](https://github.com/matrix-org/synapse/issues/3633))
- Support more federation endpoints on workers ([\#3653](https://github.com/matrix-org/synapse/issues/3653))
- Basic support for room versioning ([\#3654](https://github.com/matrix-org/synapse/issues/3654))
- Ability to disable client/server Synapse via conf toggle ([\#3655](https://github.com/matrix-org/synapse/issues/3655))
- Ability to whitelist specific threepids against monthly active user limiting ([\#3662](https://github.com/matrix-org/synapse/issues/3662))
- Add some metrics for the appservice and federation event sending loops ([\#3664](https://github.com/matrix-org/synapse/issues/3664))
- Where server is disabled, block ability for locked out users to read new messages ([\#3670](https://github.com/matrix-org/synapse/issues/3670))
- set admin uri via config, to be used in error messages where the user should contact the administrator ([\#3687](https://github.com/matrix-org/synapse/issues/3687))
- Synapse's presence functionality can now be disabled with the "use_presence" configuration option. ([\#3694](https://github.com/matrix-org/synapse/issues/3694))
- For resource limit blocked users, prevent writing into rooms ([\#3708](https://github.com/matrix-org/synapse/issues/3708))

Bugfixes
--------

- Fix occasional glitches in the synapse_event_persisted_position metric ([\#3658](https://github.com/matrix-org/synapse/issues/3658))
- Fix bug on deleting 3pid when using identity servers that don't support unbind API ([\#3661](https://github.com/matrix-org/synapse/issues/3661))
- Make the tests pass on Twisted < 18.7.0 ([\#3676](https://github.com/matrix-org/synapse/issues/3676))
- Don’t ship recaptcha_ajax.js, use it directly from Google ([\#3677](https://github.com/matrix-org/synapse/issues/3677))
- Fixes test_reap_monthly_active_users so it passes under postgres ([\#3681](https://github.com/matrix-org/synapse/issues/3681))
- Fix mau blocking calulation bug on login ([\#3689](https://github.com/matrix-org/synapse/issues/3689))
- Fix missing yield in synapse.storage.monthly_active_users.initialise_reserved_users ([\#3692](https://github.com/matrix-org/synapse/issues/3692))
- Improve HTTP request logging to include all requests ([\#3700](https://github.com/matrix-org/synapse/issues/3700))
- Avoid timing out requests while we are streaming back the response ([\#3701](https://github.com/matrix-org/synapse/issues/3701))
- Support more federation endpoints on workers ([\#3705](https://github.com/matrix-org/synapse/issues/3705), [\#3713](https://github.com/matrix-org/synapse/issues/3713))
- Fix "Starting db txn 'get_all_updated_receipts' from sentinel context" warning ([\#3710](https://github.com/matrix-org/synapse/issues/3710))
- Fix bug where `state_cache` cache factor ignored environment variables ([\#3719](https://github.com/matrix-org/synapse/issues/3719))
- Fix bug in v0.33.3rc1 which caused infinite loops and OOMs ([\#3723](https://github.com/matrix-org/synapse/issues/3723))
- Fix bug introduced in v0.33.3rc1 which made the ToS give a 500 error ([\#3732](https://github.com/matrix-org/synapse/issues/3732))

Deprecations and Removals
-------------------------

- The Shared-Secret registration method of the legacy v1/register REST endpoint has been removed. For a replacement, please see [the admin/register API documentation](https://github.com/matrix-org/synapse/blob/master/docs/admin_api/register_api.rst). ([\#3703](https://github.com/matrix-org/synapse/issues/3703))

Internal Changes
----------------

- The test suite now can run under PostgreSQL. ([\#3423](https://github.com/matrix-org/synapse/issues/3423))
- Refactor HTTP replication endpoints to reduce code duplication ([\#3632](https://github.com/matrix-org/synapse/issues/3632))
- Tests now correctly execute on Python 3. ([\#3647](https://github.com/matrix-org/synapse/issues/3647))
- Sytests can now be run inside a Docker container. ([\#3660](https://github.com/matrix-org/synapse/issues/3660))
- Port over enough to Python 3 to allow the sytests to start. ([\#3668](https://github.com/matrix-org/synapse/issues/3668))
- Update docker base image from alpine 3.7 to 3.8. ([\#3669](https://github.com/matrix-org/synapse/issues/3669))
- Rename synapse.util.async to synapse.util.async_helpers to mitigate async becoming a keyword on Python 3.7. ([\#3678](https://github.com/matrix-org/synapse/issues/3678))
- Synapse's tests are now formatted with the black autoformatter. ([\#3679](https://github.com/matrix-org/synapse/issues/3679))
- Implemented a new testing base class to reduce test boilerplate. ([\#3684](https://github.com/matrix-org/synapse/issues/3684))
- Rename MAU prometheus metrics ([\#3690](https://github.com/matrix-org/synapse/issues/3690))
- add new error type ResourceLimit ([\#3707](https://github.com/matrix-org/synapse/issues/3707))
- Logcontexts for replication command handlers ([\#3709](https://github.com/matrix-org/synapse/issues/3709))
- Update admin register API documentation to reference a real user ID. ([\#3712](https://github.com/matrix-org/synapse/issues/3712))

1 files changed, 37 insertions, 98 deletions

diff --git a/tests/events/test_utils.py b/tests/events/test_utils.py
index f51d99419e..ff217ca8b9 100644
--- a/tests/events/test_utils.py
+++ b/tests/events/test_utils.py
@@ -31,25 +31,20 @@ def MockEvent(**kwargs):
 class PruneEventTestCase(unittest.TestCase):
     """ Asserts that a new event constructed with `evdict` will look like
     `matchdict` when it is redacted. """
+
     def run_test(self, evdict, matchdict):
-        self.assertEquals(
-            prune_event(FrozenEvent(evdict)).get_dict(),
-            matchdict
-        )
+        self.assertEquals(prune_event(FrozenEvent(evdict)).get_dict(), matchdict)
 
     def test_minimal(self):
         self.run_test(
-            {
-                'type': 'A',
-                'event_id': '$test:domain',
-            },
+            {'type': 'A', 'event_id': '$test:domain'},
             {
                 'type': 'A',
                 'event_id': '$test:domain',
                 'content': {},
                 'signatures': {},
                 'unsigned': {},
-            }
+            },
         )
 
     def test_basic_keys(self):
@@ -70,23 +65,19 @@ class PruneEventTestCase(unittest.TestCase):
                 'content': {},
                 'signatures': {},
                 'unsigned': {},
-            }
+            },
         )
 
     def test_unsigned_age_ts(self):
         self.run_test(
-            {
-                'type': 'B',
-                'event_id': '$test:domain',
-                'unsigned': {'age_ts': 20},
-            },
+            {'type': 'B', 'event_id': '$test:domain', 'unsigned': {'age_ts': 20}},
             {
                 'type': 'B',
                 'event_id': '$test:domain',
                 'content': {},
                 'signatures': {},
                 'unsigned': {'age_ts': 20},
-            }
+            },
         )
 
         self.run_test(
@@ -101,23 +92,19 @@ class PruneEventTestCase(unittest.TestCase):
                 'content': {},
                 'signatures': {},
                 'unsigned': {},
-            }
+            },
         )
 
     def test_content(self):
         self.run_test(
-            {
-                'type': 'C',
-                'event_id': '$test:domain',
-                'content': {'things': 'here'},
-            },
+            {'type': 'C', 'event_id': '$test:domain', 'content': {'things': 'here'}},
             {
                 'type': 'C',
                 'event_id': '$test:domain',
                 'content': {},
                 'signatures': {},
                 'unsigned': {},
-            }
+            },
         )
 
         self.run_test(
@@ -132,27 +119,20 @@ class PruneEventTestCase(unittest.TestCase):
                 'content': {'creator': '@2:domain'},
                 'signatures': {},
                 'unsigned': {},
-            }
+            },
         )
 
 
 class SerializeEventTestCase(unittest.TestCase):
-
     def serialize(self, ev, fields):
         return serialize_event(ev, 1479807801915, only_event_fields=fields)
 
     def test_event_fields_works_with_keys(self):
         self.assertEquals(
             self.serialize(
-                MockEvent(
-                    sender="@alice:localhost",
-                    room_id="!foo:bar"
-                ),
-                ["room_id"]
+                MockEvent(sender="@alice:localhost", room_id="!foo:bar"), ["room_id"]
             ),
-            {
-                "room_id": "!foo:bar",
-            }
+            {"room_id": "!foo:bar"},
         )
 
     def test_event_fields_works_with_nested_keys(self):
@@ -161,17 +141,11 @@ class SerializeEventTestCase(unittest.TestCase):
                 MockEvent(
                     sender="@alice:localhost",
                     room_id="!foo:bar",
-                    content={
-                        "body": "A message",
-                    },
+                    content={"body": "A message"},
                 ),
-                ["content.body"]
+                ["content.body"],
             ),
-            {
-                "content": {
-                    "body": "A message",
-                }
-            }
+            {"content": {"body": "A message"}},
         )
 
     def test_event_fields_works_with_dot_keys(self):
@@ -180,17 +154,11 @@ class SerializeEventTestCase(unittest.TestCase):
                 MockEvent(
                     sender="@alice:localhost",
                     room_id="!foo:bar",
-                    content={
-                        "key.with.dots": {},
-                    },
+                    content={"key.with.dots": {}},
                 ),
-                ["content.key\.with\.dots"]
+                ["content.key\.with\.dots"],
             ),
-            {
-                "content": {
-                    "key.with.dots": {},
-                }
-            }
+            {"content": {"key.with.dots": {}}},
         )
 
     def test_event_fields_works_with_nested_dot_keys(self):
@@ -201,21 +169,12 @@ class SerializeEventTestCase(unittest.TestCase):
                     room_id="!foo:bar",
                     content={
                         "not_me": 1,
-                        "nested.dot.key": {
-                            "leaf.key": 42,
-                            "not_me_either": 1,
-                        },
+                        "nested.dot.key": {"leaf.key": 42, "not_me_either": 1},
                     },
                 ),
-                ["content.nested\.dot\.key.leaf\.key"]
+                ["content.nested\.dot\.key.leaf\.key"],
             ),
-            {
-                "content": {
-                    "nested.dot.key": {
-                        "leaf.key": 42,
-                    },
-                }
-            }
+            {"content": {"nested.dot.key": {"leaf.key": 42}}},
         )
 
     def test_event_fields_nops_with_unknown_keys(self):
@@ -224,17 +183,11 @@ class SerializeEventTestCase(unittest.TestCase):
                 MockEvent(
                     sender="@alice:localhost",
                     room_id="!foo:bar",
-                    content={
-                        "foo": "bar",
-                    },
+                    content={"foo": "bar"},
                 ),
-                ["content.foo", "content.notexists"]
+                ["content.foo", "content.notexists"],
             ),
-            {
-                "content": {
-                    "foo": "bar",
-                }
-            }
+            {"content": {"foo": "bar"}},
         )
 
     def test_event_fields_nops_with_non_dict_keys(self):
@@ -243,13 +196,11 @@ class SerializeEventTestCase(unittest.TestCase):
                 MockEvent(
                     sender="@alice:localhost",
                     room_id="!foo:bar",
-                    content={
-                        "foo": ["I", "am", "an", "array"],
-                    },
+                    content={"foo": ["I", "am", "an", "array"]},
                 ),
-                ["content.foo.am"]
+                ["content.foo.am"],
             ),
-            {}
+            {},
         )
 
     def test_event_fields_nops_with_array_keys(self):
@@ -258,13 +209,11 @@ class SerializeEventTestCase(unittest.TestCase):
                 MockEvent(
                     sender="@alice:localhost",
                     room_id="!foo:bar",
-                    content={
-                        "foo": ["I", "am", "an", "array"],
-                    },
+                    content={"foo": ["I", "am", "an", "array"]},
                 ),
-                ["content.foo.1"]
+                ["content.foo.1"],
             ),
-            {}
+            {},
         )
 
     def test_event_fields_all_fields_if_empty(self):
@@ -274,31 +223,21 @@ class SerializeEventTestCase(unittest.TestCase):
                     type="foo",
                     event_id="test",
                     room_id="!foo:bar",
-                    content={
-                        "foo": "bar",
-                    },
+                    content={"foo": "bar"},
                 ),
-                []
+                [],
             ),
             {
                 "type": "foo",
                 "event_id": "test",
                 "room_id": "!foo:bar",
-                "content": {
-                    "foo": "bar",
-                },
-                "unsigned": {}
-            }
+                "content": {"foo": "bar"},
+                "unsigned": {},
+            },
         )
 
     def test_event_fields_fail_if_fields_not_str(self):
         with self.assertRaises(TypeError):
             self.serialize(
-                MockEvent(
-                    room_id="!foo:bar",
-                    content={
-                        "foo": "bar",
-                    },
-                ),
-                ["room_id", 4]
+                MockEvent(room_id="!foo:bar", content={"foo": "bar"}), ["room_id", 4]
             )