Merge tag 'v1.38.0' into babolivier/dinsic_1.41.0

Synapse 1.38.0 (2021-07-13)
===========================

This release includes a database schema update which could result in elevated disk usage. See the [upgrade notes](https://matrix-org.github.io/synapse/develop/upgrade#upgrading-to-v1380) for more information.

No significant changes since 1.38.0rc3.

Synapse 1.38.0rc3 (2021-07-13)
==============================

Internal Changes
----------------

- Build the Debian packages in CI. ([\#10247](https://github.com/matrix-org/synapse/issues/10247), [\#10379](https://github.com/matrix-org/synapse/issues/10379))

Synapse 1.38.0rc2 (2021-07-09)
==============================

Bugfixes
--------

- Fix bug where inbound federation in a room could be delayed due to not correctly dropping a lock. Introduced in v1.37.1. ([\#10336](https://github.com/matrix-org/synapse/issues/10336))

Improved Documentation
----------------------

- Update links to documentation in the sample config. Contributed by @dklimpel. ([\#10287](https://github.com/matrix-org/synapse/issues/10287))
- Fix broken links in [INSTALL.md](INSTALL.md). Contributed by @dklimpel. ([\#10331](https://github.com/matrix-org/synapse/issues/10331))

Synapse 1.38.0rc1 (2021-07-06)
==============================

Features
--------

- Implement refresh tokens as specified by [MSC2918](https://github.com/matrix-org/matrix-doc/pull/2918). ([\#9450](https://github.com/matrix-org/synapse/issues/9450))
- Add support for evicting cache entries based on last access time. ([\#10205](https://github.com/matrix-org/synapse/issues/10205))
- Omit empty fields from the `/sync` response. Contributed by @deepbluev7. ([\#10214](https://github.com/matrix-org/synapse/issues/10214))
- Improve validation on federation `send_{join,leave,knock}` endpoints. ([\#10225](https://github.com/matrix-org/synapse/issues/10225), [\#10243](https://github.com/matrix-org/synapse/issues/10243))
- Add SSO `external_ids` to the Query User Account admin API. ([\#10261](https://github.com/matrix-org/synapse/issues/10261))
- Mark events received over federation which fail a spam check as "soft-failed". ([\#10263](https://github.com/matrix-org/synapse/issues/10263))
- Add metrics for new inbound federation staging area. ([\#10284](https://github.com/matrix-org/synapse/issues/10284))
- Add script to print information about recently registered users. ([\#10290](https://github.com/matrix-org/synapse/issues/10290))

Bugfixes
--------

- Fix a long-standing bug which meant that invite rejections and knocks were not sent out over federation in a timely manner. ([\#10223](https://github.com/matrix-org/synapse/issues/10223))
- Fix a bug introduced in v1.26.0 where only users who have set profile information could be deactivated with erasure enabled. ([\#10252](https://github.com/matrix-org/synapse/issues/10252))
- Fix a long-standing bug where Synapse would return errors after 2<sup>31</sup> events were handled by the server. ([\#10264](https://github.com/matrix-org/synapse/issues/10264), [\#10267](https://github.com/matrix-org/synapse/issues/10267), [\#10282](https://github.com/matrix-org/synapse/issues/10282), [\#10286](https://github.com/matrix-org/synapse/issues/10286), [\#10291](https://github.com/matrix-org/synapse/issues/10291), [\#10314](https://github.com/matrix-org/synapse/issues/10314), [\#10326](https://github.com/matrix-org/synapse/issues/10326))
- Fix the prometheus `synapse_federation_server_pdu_process_time` metric. Broke in v1.37.1. ([\#10279](https://github.com/matrix-org/synapse/issues/10279))
- Ensure that inbound events from federation that were being processed when Synapse was restarted get promptly processed on start up. ([\#10303](https://github.com/matrix-org/synapse/issues/10303))

Improved Documentation
----------------------

- Move the upgrade notes to [docs/upgrade.md](https://github.com/matrix-org/synapse/blob/develop/docs/upgrade.md) and convert them to markdown. ([\#10166](https://github.com/matrix-org/synapse/issues/10166))
- Choose Welcome & Overview as the default page for synapse documentation website. ([\#10242](https://github.com/matrix-org/synapse/issues/10242))
- Adjust the URL in the README.rst file to point to irc.libera.chat. ([\#10258](https://github.com/matrix-org/synapse/issues/10258))
- Fix homeserver config option name in presence router documentation. ([\#10288](https://github.com/matrix-org/synapse/issues/10288))
- Fix link pointing at the wrong section in the modules documentation page. ([\#10302](https://github.com/matrix-org/synapse/issues/10302))

Internal Changes
----------------

- Drop `Origin` and `Accept` from the value of the `Access-Control-Allow-Headers` response header. ([\#10114](https://github.com/matrix-org/synapse/issues/10114))
- Add type hints to the federation servlets. ([\#10213](https://github.com/matrix-org/synapse/issues/10213))
- Improve the reliability of auto-joining remote rooms. ([\#10237](https://github.com/matrix-org/synapse/issues/10237))
- Update the release script to use the semver terminology and determine the release branch based on the next version. ([\#10239](https://github.com/matrix-org/synapse/issues/10239))
- Fix type hints for computing auth events. ([\#10253](https://github.com/matrix-org/synapse/issues/10253))
- Improve the performance of the spaces summary endpoint by only recursing into spaces (and not rooms in general). ([\#10256](https://github.com/matrix-org/synapse/issues/10256))
- Move event authentication methods from `Auth` to `EventAuthHandler`. ([\#10268](https://github.com/matrix-org/synapse/issues/10268))
- Re-enable a SyTest after it has been fixed. ([\#10292](https://github.com/matrix-org/synapse/issues/10292))

3 files changed, 222 insertions, 31 deletions

diff --git a/tests/rest/client/v1/utils.py b/tests/rest/client/v1/utils.py

index ed55a640af..69798e95c3 100644
--- a/tests/rest/client/v1/utils.py
+++ b/tests/rest/client/v1/utils.py
@@ -52,6 +52,7 @@ class RestHelper:
         room_version: str = None,
         tok: str = None,
         expect_code: int = 200,
+        extra_content: Optional[Dict] = None,
     ) -> str:
         """
         Create a room.
@@ -72,7 +73,7 @@ class RestHelper:
         temp_id = self.auth_user_id
         self.auth_user_id = room_creator
         path = "/_matrix/client/r0/createRoom"
-        content = {}
+        content = extra_content or {}
         if not is_public:
             content["visibility"] = "private"
         if room_version:
diff --git a/tests/rest/client/v2_alpha/test_auth.py b/tests/rest/client/v2_alpha/test_auth.py

index 485e3650c3..6b90f838b6 100644
--- a/tests/rest/client/v2_alpha/test_auth.py
+++ b/tests/rest/client/v2_alpha/test_auth.py
@@ -20,7 +20,7 @@ import synapse.rest.admin
 from synapse.api.constants import LoginType
 from synapse.handlers.ui_auth.checkers import UserInteractiveAuthChecker
 from synapse.rest.client.v1 import login
-from synapse.rest.client.v2_alpha import auth, devices, register
+from synapse.rest.client.v2_alpha import account, auth, devices, register
 from synapse.rest.synapse.client import build_synapse_client_resource_tree
 from synapse.types import JsonDict, UserID
 
@@ -498,3 +498,221 @@ class UIAuthTests(unittest.HomeserverTestCase):
         self.delete_device(
             self.user_tok, self.device_id, 403, body={"auth": {"session": session_id}}
         )
+
+
+class RefreshAuthTests(unittest.HomeserverTestCase):
+    servlets = [
+        auth.register_servlets,
+        account.register_servlets,
+        login.register_servlets,
+        synapse.rest.admin.register_servlets_for_client_rest_resource,
+        register.register_servlets,
+    ]
+    hijack_auth = False
+
+    def prepare(self, reactor, clock, hs):
+        self.user_pass = "pass"
+        self.user = self.register_user("test", self.user_pass)
+
+    def test_login_issue_refresh_token(self):
+        """
+        A login response should include a refresh_token only if asked.
+        """
+        # Test login
+        body = {"type": "m.login.password", "user": "test", "password": self.user_pass}
+
+        login_without_refresh = self.make_request(
+            "POST", "/_matrix/client/r0/login", body
+        )
+        self.assertEqual(login_without_refresh.code, 200, login_without_refresh.result)
+        self.assertNotIn("refresh_token", login_without_refresh.json_body)
+
+        login_with_refresh = self.make_request(
+            "POST",
+            "/_matrix/client/r0/login?org.matrix.msc2918.refresh_token=true",
+            body,
+        )
+        self.assertEqual(login_with_refresh.code, 200, login_with_refresh.result)
+        self.assertIn("refresh_token", login_with_refresh.json_body)
+        self.assertIn("expires_in_ms", login_with_refresh.json_body)
+
+    def test_register_issue_refresh_token(self):
+        """
+        A register response should include a refresh_token only if asked.
+        """
+        register_without_refresh = self.make_request(
+            "POST",
+            "/_matrix/client/r0/register",
+            {
+                "username": "test2",
+                "password": self.user_pass,
+                "auth": {"type": LoginType.DUMMY},
+            },
+        )
+        self.assertEqual(
+            register_without_refresh.code, 200, register_without_refresh.result
+        )
+        self.assertNotIn("refresh_token", register_without_refresh.json_body)
+
+        register_with_refresh = self.make_request(
+            "POST",
+            "/_matrix/client/r0/register?org.matrix.msc2918.refresh_token=true",
+            {
+                "username": "test3",
+                "password": self.user_pass,
+                "auth": {"type": LoginType.DUMMY},
+            },
+        )
+        self.assertEqual(register_with_refresh.code, 200, register_with_refresh.result)
+        self.assertIn("refresh_token", register_with_refresh.json_body)
+        self.assertIn("expires_in_ms", register_with_refresh.json_body)
+
+    def test_token_refresh(self):
+        """
+        A refresh token can be used to issue a new access token.
+        """
+        body = {"type": "m.login.password", "user": "test", "password": self.user_pass}
+        login_response = self.make_request(
+            "POST",
+            "/_matrix/client/r0/login?org.matrix.msc2918.refresh_token=true",
+            body,
+        )
+        self.assertEqual(login_response.code, 200, login_response.result)
+
+        refresh_response = self.make_request(
+            "POST",
+            "/_matrix/client/unstable/org.matrix.msc2918.refresh_token/refresh",
+            {"refresh_token": login_response.json_body["refresh_token"]},
+        )
+        self.assertEqual(refresh_response.code, 200, refresh_response.result)
+        self.assertIn("access_token", refresh_response.json_body)
+        self.assertIn("refresh_token", refresh_response.json_body)
+        self.assertIn("expires_in_ms", refresh_response.json_body)
+
+        # The access and refresh tokens should be different from the original ones after refresh
+        self.assertNotEqual(
+            login_response.json_body["access_token"],
+            refresh_response.json_body["access_token"],
+        )
+        self.assertNotEqual(
+            login_response.json_body["refresh_token"],
+            refresh_response.json_body["refresh_token"],
+        )
+
+    @override_config({"access_token_lifetime": "1m"})
+    def test_refresh_token_expiration(self):
+        """
+        The access token should have some time as specified in the config.
+        """
+        body = {"type": "m.login.password", "user": "test", "password": self.user_pass}
+        login_response = self.make_request(
+            "POST",
+            "/_matrix/client/r0/login?org.matrix.msc2918.refresh_token=true",
+            body,
+        )
+        self.assertEqual(login_response.code, 200, login_response.result)
+        self.assertApproximates(
+            login_response.json_body["expires_in_ms"], 60 * 1000, 100
+        )
+
+        refresh_response = self.make_request(
+            "POST",
+            "/_matrix/client/unstable/org.matrix.msc2918.refresh_token/refresh",
+            {"refresh_token": login_response.json_body["refresh_token"]},
+        )
+        self.assertEqual(refresh_response.code, 200, refresh_response.result)
+        self.assertApproximates(
+            refresh_response.json_body["expires_in_ms"], 60 * 1000, 100
+        )
+
+    def test_refresh_token_invalidation(self):
+        """Refresh tokens are invalidated after first use of the next token.
+
+        A refresh token is considered invalid if:
+            - it was already used at least once
+            - and either
+                - the next access token was used
+                - the next refresh token was used
+
+        The chain of tokens goes like this:
+
+            login -|-> first_refresh -> third_refresh (fails)
+                   |-> second_refresh -> fifth_refresh
+                   |-> fourth_refresh (fails)
+        """
+
+        body = {"type": "m.login.password", "user": "test", "password": self.user_pass}
+        login_response = self.make_request(
+            "POST",
+            "/_matrix/client/r0/login?org.matrix.msc2918.refresh_token=true",
+            body,
+        )
+        self.assertEqual(login_response.code, 200, login_response.result)
+
+        # This first refresh should work properly
+        first_refresh_response = self.make_request(
+            "POST",
+            "/_matrix/client/unstable/org.matrix.msc2918.refresh_token/refresh",
+            {"refresh_token": login_response.json_body["refresh_token"]},
+        )
+        self.assertEqual(
+            first_refresh_response.code, 200, first_refresh_response.result
+        )
+
+        # This one as well, since the token in the first one was never used
+        second_refresh_response = self.make_request(
+            "POST",
+            "/_matrix/client/unstable/org.matrix.msc2918.refresh_token/refresh",
+            {"refresh_token": login_response.json_body["refresh_token"]},
+        )
+        self.assertEqual(
+            second_refresh_response.code, 200, second_refresh_response.result
+        )
+
+        # This one should not, since the token from the first refresh is not valid anymore
+        third_refresh_response = self.make_request(
+            "POST",
+            "/_matrix/client/unstable/org.matrix.msc2918.refresh_token/refresh",
+            {"refresh_token": first_refresh_response.json_body["refresh_token"]},
+        )
+        self.assertEqual(
+            third_refresh_response.code, 401, third_refresh_response.result
+        )
+
+        # The associated access token should also be invalid
+        whoami_response = self.make_request(
+            "GET",
+            "/_matrix/client/r0/account/whoami",
+            access_token=first_refresh_response.json_body["access_token"],
+        )
+        self.assertEqual(whoami_response.code, 401, whoami_response.result)
+
+        # But all other tokens should work (they will expire after some time)
+        for access_token in [
+            second_refresh_response.json_body["access_token"],
+            login_response.json_body["access_token"],
+        ]:
+            whoami_response = self.make_request(
+                "GET", "/_matrix/client/r0/account/whoami", access_token=access_token
+            )
+            self.assertEqual(whoami_response.code, 200, whoami_response.result)
+
+        # Now that the access token from the last valid refresh was used once, refreshing with the N-1 token should fail
+        fourth_refresh_response = self.make_request(
+            "POST",
+            "/_matrix/client/unstable/org.matrix.msc2918.refresh_token/refresh",
+            {"refresh_token": login_response.json_body["refresh_token"]},
+        )
+        self.assertEqual(
+            fourth_refresh_response.code, 403, fourth_refresh_response.result
+        )
+
+        # But refreshing from the last valid refresh token still works
+        fifth_refresh_response = self.make_request(
+            "POST",
+            "/_matrix/client/unstable/org.matrix.msc2918.refresh_token/refresh",
+            {"refresh_token": second_refresh_response.json_body["refresh_token"]},
+        )
+        self.assertEqual(
+            fifth_refresh_response.code, 200, fifth_refresh_response.result
+        )
diff --git a/tests/rest/client/v2_alpha/test_sync.py b/tests/rest/client/v2_alpha/test_sync.py

index 726a22f90c..1ac23e1769 100644
--- a/tests/rest/client/v2_alpha/test_sync.py
+++ b/tests/rest/client/v2_alpha/test_sync.py
@@ -47,35 +47,7 @@ class FilterTestCase(unittest.HomeserverTestCase):
         channel = self.make_request("GET", "/sync")
 
         self.assertEqual(channel.code, 200)
-        self.assertTrue(
-            {
-                "next_batch",
-                "rooms",
-                "presence",
-                "account_data",
-                "to_device",
-                "device_lists",
-            }.issubset(set(channel.json_body.keys()))
-        )
-
-    def test_sync_presence_disabled(self):
-        """
-        When presence is disabled, the key does not appear in /sync.
-        """
-        self.hs.config.use_presence = False
-
-        channel = self.make_request("GET", "/sync")
-
-        self.assertEqual(channel.code, 200)
-        self.assertTrue(
-            {
-                "next_batch",
-                "rooms",
-                "account_data",
-                "to_device",
-                "device_lists",
-            }.issubset(set(channel.json_body.keys()))
-        )
+        self.assertIn("next_batch", channel.json_body)
 
 
 class SyncFilterTestCase(unittest.HomeserverTestCase):