Merge tag 'v0.33.3'

Features
--------

- Add support for the SNI extension to federation TLS connections. Thanks to @vojeroen! ([\#3439](https://github.com/matrix-org/synapse/issues/3439))
- Add /_media/r0/config ([\#3184](https://github.com/matrix-org/synapse/issues/3184))
- speed up /members API and add `at` and `membership` params as per MSC1227 ([\#3568](https://github.com/matrix-org/synapse/issues/3568))
- implement `summary` block in /sync response as per MSC688 ([\#3574](https://github.com/matrix-org/synapse/issues/3574))
- Add lazy-loading support to /messages as per MSC1227 ([\#3589](https://github.com/matrix-org/synapse/issues/3589))
- Add ability to limit number of monthly active users on the server ([\#3633](https://github.com/matrix-org/synapse/issues/3633))
- Support more federation endpoints on workers ([\#3653](https://github.com/matrix-org/synapse/issues/3653))
- Basic support for room versioning ([\#3654](https://github.com/matrix-org/synapse/issues/3654))
- Ability to disable client/server Synapse via conf toggle ([\#3655](https://github.com/matrix-org/synapse/issues/3655))
- Ability to whitelist specific threepids against monthly active user limiting ([\#3662](https://github.com/matrix-org/synapse/issues/3662))
- Add some metrics for the appservice and federation event sending loops ([\#3664](https://github.com/matrix-org/synapse/issues/3664))
- Where server is disabled, block ability for locked out users to read new messages ([\#3670](https://github.com/matrix-org/synapse/issues/3670))
- set admin uri via config, to be used in error messages where the user should contact the administrator ([\#3687](https://github.com/matrix-org/synapse/issues/3687))
- Synapse's presence functionality can now be disabled with the "use_presence" configuration option. ([\#3694](https://github.com/matrix-org/synapse/issues/3694))
- For resource limit blocked users, prevent writing into rooms ([\#3708](https://github.com/matrix-org/synapse/issues/3708))

Bugfixes
--------

- Fix occasional glitches in the synapse_event_persisted_position metric ([\#3658](https://github.com/matrix-org/synapse/issues/3658))
- Fix bug on deleting 3pid when using identity servers that don't support unbind API ([\#3661](https://github.com/matrix-org/synapse/issues/3661))
- Make the tests pass on Twisted < 18.7.0 ([\#3676](https://github.com/matrix-org/synapse/issues/3676))
- Don’t ship recaptcha_ajax.js, use it directly from Google ([\#3677](https://github.com/matrix-org/synapse/issues/3677))
- Fixes test_reap_monthly_active_users so it passes under postgres ([\#3681](https://github.com/matrix-org/synapse/issues/3681))
- Fix mau blocking calulation bug on login ([\#3689](https://github.com/matrix-org/synapse/issues/3689))
- Fix missing yield in synapse.storage.monthly_active_users.initialise_reserved_users ([\#3692](https://github.com/matrix-org/synapse/issues/3692))
- Improve HTTP request logging to include all requests ([\#3700](https://github.com/matrix-org/synapse/issues/3700))
- Avoid timing out requests while we are streaming back the response ([\#3701](https://github.com/matrix-org/synapse/issues/3701))
- Support more federation endpoints on workers ([\#3705](https://github.com/matrix-org/synapse/issues/3705), [\#3713](https://github.com/matrix-org/synapse/issues/3713))
- Fix "Starting db txn 'get_all_updated_receipts' from sentinel context" warning ([\#3710](https://github.com/matrix-org/synapse/issues/3710))
- Fix bug where `state_cache` cache factor ignored environment variables ([\#3719](https://github.com/matrix-org/synapse/issues/3719))
- Fix bug in v0.33.3rc1 which caused infinite loops and OOMs ([\#3723](https://github.com/matrix-org/synapse/issues/3723))
- Fix bug introduced in v0.33.3rc1 which made the ToS give a 500 error ([\#3732](https://github.com/matrix-org/synapse/issues/3732))

Deprecations and Removals
-------------------------

- The Shared-Secret registration method of the legacy v1/register REST endpoint has been removed. For a replacement, please see [the admin/register API documentation](https://github.com/matrix-org/synapse/blob/master/docs/admin_api/register_api.rst). ([\#3703](https://github.com/matrix-org/synapse/issues/3703))

Internal Changes
----------------

- The test suite now can run under PostgreSQL. ([\#3423](https://github.com/matrix-org/synapse/issues/3423))
- Refactor HTTP replication endpoints to reduce code duplication ([\#3632](https://github.com/matrix-org/synapse/issues/3632))
- Tests now correctly execute on Python 3. ([\#3647](https://github.com/matrix-org/synapse/issues/3647))
- Sytests can now be run inside a Docker container. ([\#3660](https://github.com/matrix-org/synapse/issues/3660))
- Port over enough to Python 3 to allow the sytests to start. ([\#3668](https://github.com/matrix-org/synapse/issues/3668))
- Update docker base image from alpine 3.7 to 3.8. ([\#3669](https://github.com/matrix-org/synapse/issues/3669))
- Rename synapse.util.async to synapse.util.async_helpers to mitigate async becoming a keyword on Python 3.7. ([\#3678](https://github.com/matrix-org/synapse/issues/3678))
- Synapse's tests are now formatted with the black autoformatter. ([\#3679](https://github.com/matrix-org/synapse/issues/3679))
- Implemented a new testing base class to reduce test boilerplate. ([\#3684](https://github.com/matrix-org/synapse/issues/3684))
- Rename MAU prometheus metrics ([\#3690](https://github.com/matrix-org/synapse/issues/3690))
- add new error type ResourceLimit ([\#3707](https://github.com/matrix-org/synapse/issues/3707))
- Logcontexts for replication command handlers ([\#3709](https://github.com/matrix-org/synapse/issues/3709))
- Update admin register API documentation to reference a real user ID. ([\#3712](https://github.com/matrix-org/synapse/issues/3712))

1 files changed, 95 insertions, 121 deletions

diff --git a/tests/handlers/test_presence.py b/tests/handlers/test_presence.py

index 121ce78634..fc2b646ba2 100644
--- a/tests/handlers/test_presence.py
+++ b/tests/handlers/test_presence.py
@@ -39,8 +39,7 @@ class PresenceUpdateTestCase(unittest.TestCase):
 
         prev_state = UserPresenceState.default(user_id)
         new_state = prev_state.copy_and_replace(
-            state=PresenceState.ONLINE,
-            last_active_ts=now,
+            state=PresenceState.ONLINE, last_active_ts=now
         )
 
         state, persist_and_notify, federation_ping = handle_update(
@@ -54,23 +53,22 @@ class PresenceUpdateTestCase(unittest.TestCase):
         self.assertEquals(state.last_federation_update_ts, now)
 
         self.assertEquals(wheel_timer.insert.call_count, 3)
-        wheel_timer.insert.assert_has_calls([
-            call(
-                now=now,
-                obj=user_id,
-                then=new_state.last_active_ts + IDLE_TIMER
-            ),
-            call(
-                now=now,
-                obj=user_id,
-                then=new_state.last_user_sync_ts + SYNC_ONLINE_TIMEOUT
-            ),
-            call(
-                now=now,
-                obj=user_id,
-                then=new_state.last_active_ts + LAST_ACTIVE_GRANULARITY
-            ),
-        ], any_order=True)
+        wheel_timer.insert.assert_has_calls(
+            [
+                call(now=now, obj=user_id, then=new_state.last_active_ts + IDLE_TIMER),
+                call(
+                    now=now,
+                    obj=user_id,
+                    then=new_state.last_user_sync_ts + SYNC_ONLINE_TIMEOUT,
+                ),
+                call(
+                    now=now,
+                    obj=user_id,
+                    then=new_state.last_active_ts + LAST_ACTIVE_GRANULARITY,
+                ),
+            ],
+            any_order=True,
+        )
 
     def test_online_to_online(self):
         wheel_timer = Mock()
@@ -79,14 +77,11 @@ class PresenceUpdateTestCase(unittest.TestCase):
 
         prev_state = UserPresenceState.default(user_id)
         prev_state = prev_state.copy_and_replace(
-            state=PresenceState.ONLINE,
-            last_active_ts=now,
-            currently_active=True,
+            state=PresenceState.ONLINE, last_active_ts=now, currently_active=True
         )
 
         new_state = prev_state.copy_and_replace(
-            state=PresenceState.ONLINE,
-            last_active_ts=now,
+            state=PresenceState.ONLINE, last_active_ts=now
         )
 
         state, persist_and_notify, federation_ping = handle_update(
@@ -101,23 +96,22 @@ class PresenceUpdateTestCase(unittest.TestCase):
         self.assertEquals(state.last_federation_update_ts, now)
 
         self.assertEquals(wheel_timer.insert.call_count, 3)
-        wheel_timer.insert.assert_has_calls([
-            call(
-                now=now,
-                obj=user_id,
-                then=new_state.last_active_ts + IDLE_TIMER
-            ),
-            call(
-                now=now,
-                obj=user_id,
-                then=new_state.last_user_sync_ts + SYNC_ONLINE_TIMEOUT
-            ),
-            call(
-                now=now,
-                obj=user_id,
-                then=new_state.last_active_ts + LAST_ACTIVE_GRANULARITY
-            ),
-        ], any_order=True)
+        wheel_timer.insert.assert_has_calls(
+            [
+                call(now=now, obj=user_id, then=new_state.last_active_ts + IDLE_TIMER),
+                call(
+                    now=now,
+                    obj=user_id,
+                    then=new_state.last_user_sync_ts + SYNC_ONLINE_TIMEOUT,
+                ),
+                call(
+                    now=now,
+                    obj=user_id,
+                    then=new_state.last_active_ts + LAST_ACTIVE_GRANULARITY,
+                ),
+            ],
+            any_order=True,
+        )
 
     def test_online_to_online_last_active_noop(self):
         wheel_timer = Mock()
@@ -132,8 +126,7 @@ class PresenceUpdateTestCase(unittest.TestCase):
         )
 
         new_state = prev_state.copy_and_replace(
-            state=PresenceState.ONLINE,
-            last_active_ts=now,
+            state=PresenceState.ONLINE, last_active_ts=now
         )
 
         state, persist_and_notify, federation_ping = handle_update(
@@ -148,23 +141,22 @@ class PresenceUpdateTestCase(unittest.TestCase):
         self.assertEquals(state.last_federation_update_ts, now)
 
         self.assertEquals(wheel_timer.insert.call_count, 3)
-        wheel_timer.insert.assert_has_calls([
-            call(
-                now=now,
-                obj=user_id,
-                then=new_state.last_active_ts + IDLE_TIMER
-            ),
-            call(
-                now=now,
-                obj=user_id,
-                then=new_state.last_user_sync_ts + SYNC_ONLINE_TIMEOUT
-            ),
-            call(
-                now=now,
-                obj=user_id,
-                then=new_state.last_active_ts + LAST_ACTIVE_GRANULARITY
-            ),
-        ], any_order=True)
+        wheel_timer.insert.assert_has_calls(
+            [
+                call(now=now, obj=user_id, then=new_state.last_active_ts + IDLE_TIMER),
+                call(
+                    now=now,
+                    obj=user_id,
+                    then=new_state.last_user_sync_ts + SYNC_ONLINE_TIMEOUT,
+                ),
+                call(
+                    now=now,
+                    obj=user_id,
+                    then=new_state.last_active_ts + LAST_ACTIVE_GRANULARITY,
+                ),
+            ],
+            any_order=True,
+        )
 
     def test_online_to_online_last_active(self):
         wheel_timer = Mock()
@@ -178,9 +170,7 @@ class PresenceUpdateTestCase(unittest.TestCase):
             currently_active=True,
         )
 
-        new_state = prev_state.copy_and_replace(
-            state=PresenceState.ONLINE,
-        )
+        new_state = prev_state.copy_and_replace(state=PresenceState.ONLINE)
 
         state, persist_and_notify, federation_ping = handle_update(
             prev_state, new_state, is_mine=True, wheel_timer=wheel_timer, now=now
@@ -193,18 +183,17 @@ class PresenceUpdateTestCase(unittest.TestCase):
         self.assertEquals(state.last_federation_update_ts, now)
 
         self.assertEquals(wheel_timer.insert.call_count, 2)
-        wheel_timer.insert.assert_has_calls([
-            call(
-                now=now,
-                obj=user_id,
-                then=new_state.last_active_ts + IDLE_TIMER
-            ),
-            call(
-                now=now,
-                obj=user_id,
-                then=new_state.last_user_sync_ts + SYNC_ONLINE_TIMEOUT
-            )
-        ], any_order=True)
+        wheel_timer.insert.assert_has_calls(
+            [
+                call(now=now, obj=user_id, then=new_state.last_active_ts + IDLE_TIMER),
+                call(
+                    now=now,
+                    obj=user_id,
+                    then=new_state.last_user_sync_ts + SYNC_ONLINE_TIMEOUT,
+                ),
+            ],
+            any_order=True,
+        )
 
     def test_remote_ping_timer(self):
         wheel_timer = Mock()
@@ -213,13 +202,10 @@ class PresenceUpdateTestCase(unittest.TestCase):
 
         prev_state = UserPresenceState.default(user_id)
         prev_state = prev_state.copy_and_replace(
-            state=PresenceState.ONLINE,
-            last_active_ts=now,
+            state=PresenceState.ONLINE, last_active_ts=now
         )
 
-        new_state = prev_state.copy_and_replace(
-            state=PresenceState.ONLINE,
-        )
+        new_state = prev_state.copy_and_replace(state=PresenceState.ONLINE)
 
         state, persist_and_notify, federation_ping = handle_update(
             prev_state, new_state, is_mine=False, wheel_timer=wheel_timer, now=now
@@ -232,13 +218,16 @@ class PresenceUpdateTestCase(unittest.TestCase):
         self.assertEquals(new_state.status_msg, state.status_msg)
 
         self.assertEquals(wheel_timer.insert.call_count, 1)
-        wheel_timer.insert.assert_has_calls([
-            call(
-                now=now,
-                obj=user_id,
-                then=new_state.last_federation_update_ts + FEDERATION_TIMEOUT
-            ),
-        ], any_order=True)
+        wheel_timer.insert.assert_has_calls(
+            [
+                call(
+                    now=now,
+                    obj=user_id,
+                    then=new_state.last_federation_update_ts + FEDERATION_TIMEOUT,
+                )
+            ],
+            any_order=True,
+        )
 
     def test_online_to_offline(self):
         wheel_timer = Mock()
@@ -247,14 +236,10 @@ class PresenceUpdateTestCase(unittest.TestCase):
 
         prev_state = UserPresenceState.default(user_id)
         prev_state = prev_state.copy_and_replace(
-            state=PresenceState.ONLINE,
-            last_active_ts=now,
-            currently_active=True,
+            state=PresenceState.ONLINE, last_active_ts=now, currently_active=True
         )
 
-        new_state = prev_state.copy_and_replace(
-            state=PresenceState.OFFLINE,
-        )
+        new_state = prev_state.copy_and_replace(state=PresenceState.OFFLINE)
 
         state, persist_and_notify, federation_ping = handle_update(
             prev_state, new_state, is_mine=True, wheel_timer=wheel_timer, now=now
@@ -273,14 +258,10 @@ class PresenceUpdateTestCase(unittest.TestCase):
 
         prev_state = UserPresenceState.default(user_id)
         prev_state = prev_state.copy_and_replace(
-            state=PresenceState.ONLINE,
-            last_active_ts=now,
-            currently_active=True,
+            state=PresenceState.ONLINE, last_active_ts=now, currently_active=True
         )
 
-        new_state = prev_state.copy_and_replace(
-            state=PresenceState.UNAVAILABLE,
-        )
+        new_state = prev_state.copy_and_replace(state=PresenceState.UNAVAILABLE)
 
         state, persist_and_notify, federation_ping = handle_update(
             prev_state, new_state, is_mine=True, wheel_timer=wheel_timer, now=now
@@ -293,13 +274,16 @@ class PresenceUpdateTestCase(unittest.TestCase):
         self.assertEquals(new_state.status_msg, state.status_msg)
 
         self.assertEquals(wheel_timer.insert.call_count, 1)
-        wheel_timer.insert.assert_has_calls([
-            call(
-                now=now,
-                obj=user_id,
-                then=new_state.last_user_sync_ts + SYNC_ONLINE_TIMEOUT
-            )
-        ], any_order=True)
+        wheel_timer.insert.assert_has_calls(
+            [
+                call(
+                    now=now,
+                    obj=user_id,
+                    then=new_state.last_user_sync_ts + SYNC_ONLINE_TIMEOUT,
+                )
+            ],
+            any_order=True,
+        )
 
 
 class PresenceTimeoutTestCase(unittest.TestCase):
@@ -314,9 +298,7 @@ class PresenceTimeoutTestCase(unittest.TestCase):
             last_user_sync_ts=now,
         )
 
-        new_state = handle_timeout(
-            state, is_mine=True, syncing_user_ids=set(), now=now
-        )
+        new_state = handle_timeout(state, is_mine=True, syncing_user_ids=set(), now=now)
 
         self.assertIsNotNone(new_state)
         self.assertEquals(new_state.state, PresenceState.UNAVAILABLE)
@@ -332,9 +314,7 @@ class PresenceTimeoutTestCase(unittest.TestCase):
             last_user_sync_ts=now - SYNC_ONLINE_TIMEOUT - 1,
         )
 
-        new_state = handle_timeout(
-            state, is_mine=True, syncing_user_ids=set(), now=now
-        )
+        new_state = handle_timeout(state, is_mine=True, syncing_user_ids=set(), now=now)
 
         self.assertIsNotNone(new_state)
         self.assertEquals(new_state.state, PresenceState.OFFLINE)
@@ -369,9 +349,7 @@ class PresenceTimeoutTestCase(unittest.TestCase):
             last_federation_update_ts=now - FEDERATION_PING_INTERVAL - 1,
         )
 
-        new_state = handle_timeout(
-            state, is_mine=True, syncing_user_ids=set(), now=now
-        )
+        new_state = handle_timeout(state, is_mine=True, syncing_user_ids=set(), now=now)
 
         self.assertIsNotNone(new_state)
         self.assertEquals(new_state, new_state)
@@ -388,9 +366,7 @@ class PresenceTimeoutTestCase(unittest.TestCase):
             last_federation_update_ts=now,
         )
 
-        new_state = handle_timeout(
-            state, is_mine=True, syncing_user_ids=set(), now=now
-        )
+        new_state = handle_timeout(state, is_mine=True, syncing_user_ids=set(), now=now)
 
         self.assertIsNone(new_state)
 
@@ -425,9 +401,7 @@ class PresenceTimeoutTestCase(unittest.TestCase):
             last_federation_update_ts=now,
         )
 
-        new_state = handle_timeout(
-            state, is_mine=True, syncing_user_ids=set(), now=now
-        )
+        new_state = handle_timeout(state, is_mine=True, syncing_user_ids=set(), now=now)
 
         self.assertIsNotNone(new_state)
         self.assertEquals(state, new_state)