Merge branch 'release-v1.33.0' of github.com:matrix-org/synapse into matrix-org-hotfixes

40 files changed, 102 insertions, 50 deletions

diff --git a/CHANGES.md b/CHANGES.md
index 532b30e232..9a41607679 100644
--- a/CHANGES.md
+++ b/CHANGES.md
@@ -1,3 +1,56 @@
+Synapse 1.33.0rc1 (2021-04-28)
+==============================
+
+Features
+--------
+
+- Update experimental support for [MSC3083](https://github.com/matrix-org/matrix-doc/pull/3083): restricting room access via group membership. ([\#9800](https://github.com/matrix-org/synapse/issues/9800), [\#9814](https://github.com/matrix-org/synapse/issues/9814))
+- Add experimental support for handling presence on a worker. ([\#9819](https://github.com/matrix-org/synapse/issues/9819), [\#9820](https://github.com/matrix-org/synapse/issues/9820), [\#9828](https://github.com/matrix-org/synapse/issues/9828), [\#9850](https://github.com/matrix-org/synapse/issues/9850))
+- Return a new template when an user attempts to renew their account multiple times with the same token, stating that their account is set to expire. This replaces the invalid token template that would previously be shown in this case. This change concerns the optional account validity feature. ([\#9832](https://github.com/matrix-org/synapse/issues/9832))
+
+
+Bugfixes
+--------
+
+- Fixes the OIDC SSO flow when using a `public_baseurl` value including a non-root URL path. ([\#9726](https://github.com/matrix-org/synapse/issues/9726))
+- Fix thumbnail generation for some sites with non-standard content types. Contributed by @rkfg. ([\#9788](https://github.com/matrix-org/synapse/issues/9788))
+- Add some sanity checks to identity server passed to 3PID bind/unbind endpoints. ([\#9802](https://github.com/matrix-org/synapse/issues/9802))
+- Limit the size of HTTP responses read over federation. ([\#9833](https://github.com/matrix-org/synapse/issues/9833))
+- Fix a bug which could cause Synapse to get stuck in a loop of resyncing device lists. ([\#9867](https://github.com/matrix-org/synapse/issues/9867))
+- Fix a long-standing bug where errors from federation did not propagate to the client. ([\#9868](https://github.com/matrix-org/synapse/issues/9868))
+
+
+Improved Documentation
+----------------------
+
+- Add a note to the docker docs mentioning that we mirror upstream's supported Docker platforms. ([\#9801](https://github.com/matrix-org/synapse/issues/9801))
+
+
+Internal Changes
+----------------
+
+- Add a dockerfile for running Synapse in worker-mode under Complement. ([\#9162](https://github.com/matrix-org/synapse/issues/9162))
+- Apply `pyupgrade` across the codebase. ([\#9786](https://github.com/matrix-org/synapse/issues/9786))
+- Move some replication processing out of `generic_worker`. ([\#9796](https://github.com/matrix-org/synapse/issues/9796))
+- Replace `HomeServer.get_config()` with inline references. ([\#9815](https://github.com/matrix-org/synapse/issues/9815))
+- Rename some handlers and config modules to not duplicate the top-level module. ([\#9816](https://github.com/matrix-org/synapse/issues/9816))
+- Fix a long-standing bug which caused `max_upload_size` to not be correctly enforced. ([\#9817](https://github.com/matrix-org/synapse/issues/9817))
+- Reduce CPU usage of the user directory by reusing existing calculated room membership. ([\#9821](https://github.com/matrix-org/synapse/issues/9821))
+- Small speed up for joining large remote rooms. ([\#9825](https://github.com/matrix-org/synapse/issues/9825))
+- Introduce flake8-bugbear to the test suite and fix some of its lint violations. ([\#9838](https://github.com/matrix-org/synapse/issues/9838))
+- Only store the raw data in the in-memory caches, rather than objects that include references to e.g. the data stores. ([\#9845](https://github.com/matrix-org/synapse/issues/9845))
+- Limit length of accepted email addresses. ([\#9855](https://github.com/matrix-org/synapse/issues/9855))
+- Remove redundant `synapse.types.Collection` type definition. ([\#9856](https://github.com/matrix-org/synapse/issues/9856))
+- Handle recently added rate limits correctly when using `--no-rate-limit` with the demo scripts. ([\#9858](https://github.com/matrix-org/synapse/issues/9858))
+- Disable invite rate-limiting by default when running the unit tests. ([\#9871](https://github.com/matrix-org/synapse/issues/9871))
+- Pass a reactor into `SynapseSite` to make testing easier. ([\#9874](https://github.com/matrix-org/synapse/issues/9874))
+- Make `DomainSpecificString` an `attrs` class. ([\#9875](https://github.com/matrix-org/synapse/issues/9875))
+- Add type hints to `synapse.api.auth` and `synapse.api.auth_blocking` modules. ([\#9876](https://github.com/matrix-org/synapse/issues/9876))
+- Remove redundant `_PushHTTPChannel` test class. ([\#9878](https://github.com/matrix-org/synapse/issues/9878))
+- Remove backwards-compatibility code for Python versions < 3.6. ([\#9879](https://github.com/matrix-org/synapse/issues/9879))
+- Small performance improvement around handling new local presence updates. ([\#9887](https://github.com/matrix-org/synapse/issues/9887))
+
+
 Synapse 1.32.2 (2021-04-22)
 ===========================
 
diff --git a/changelog.d/9162.misc b/changelog.d/9162.misc
deleted file mode 100644
index 1083da8a7a..0000000000
--- a/changelog.d/9162.misc
+++ /dev/null
@@ -1 +0,0 @@
-Add a dockerfile for running Synapse in worker-mode under Complement.
\ No newline at end of file
diff --git a/changelog.d/9726.bugfix b/changelog.d/9726.bugfix
deleted file mode 100644
index 4ba0b24327..0000000000
--- a/changelog.d/9726.bugfix
+++ /dev/null
@@ -1 +0,0 @@
-Fixes the OIDC SSO flow when using a `public_baseurl` value including a non-root URL path.
\ No newline at end of file
diff --git a/changelog.d/9786.misc b/changelog.d/9786.misc
deleted file mode 100644
index cf265db749..0000000000
--- a/changelog.d/9786.misc
+++ /dev/null
@@ -1 +0,0 @@
-Apply `pyupgrade` across the codebase.
\ No newline at end of file
diff --git a/changelog.d/9788.bugfix b/changelog.d/9788.bugfix
deleted file mode 100644
index edb58fbd5b..0000000000
--- a/changelog.d/9788.bugfix
+++ /dev/null
@@ -1 +0,0 @@
-Fix thumbnail generation for some sites with non-standard content types. Contributed by @rkfg.
diff --git a/changelog.d/9796.misc b/changelog.d/9796.misc
deleted file mode 100644
index 59bb1813c3..0000000000
--- a/changelog.d/9796.misc
+++ /dev/null
@@ -1 +0,0 @@
-Move some replication processing out of `generic_worker`.
diff --git a/changelog.d/9800.feature b/changelog.d/9800.feature
deleted file mode 100644
index 9404ad2fc0..0000000000
--- a/changelog.d/9800.feature
+++ /dev/null
@@ -1 +0,0 @@
-Update experimental support for [MSC3083](https://github.com/matrix-org/matrix-doc/pull/3083): restricting room access via group membership.
diff --git a/changelog.d/9801.doc b/changelog.d/9801.doc
deleted file mode 100644
index 8b8b9d01d4..0000000000
--- a/changelog.d/9801.doc
+++ /dev/null
@@ -1 +0,0 @@
-Add a note to the docker docs mentioning that we mirror upstream's supported Docker platforms.
diff --git a/changelog.d/9802.bugfix b/changelog.d/9802.bugfix
deleted file mode 100644
index 0c72f7be47..0000000000
--- a/changelog.d/9802.bugfix
+++ /dev/null
@@ -1 +0,0 @@
-Add some sanity checks to identity server passed to 3PID bind/unbind endpoints.
diff --git a/changelog.d/9814.feature b/changelog.d/9814.feature
deleted file mode 100644
index 9404ad2fc0..0000000000
--- a/changelog.d/9814.feature
+++ /dev/null
@@ -1 +0,0 @@
-Update experimental support for [MSC3083](https://github.com/matrix-org/matrix-doc/pull/3083): restricting room access via group membership.
diff --git a/changelog.d/9815.misc b/changelog.d/9815.misc
deleted file mode 100644
index e33d012d3d..0000000000
--- a/changelog.d/9815.misc
+++ /dev/null
@@ -1 +0,0 @@
-Replace `HomeServer.get_config()` with inline references.
diff --git a/changelog.d/9816.misc b/changelog.d/9816.misc
deleted file mode 100644
index d098122500..0000000000
--- a/changelog.d/9816.misc
+++ /dev/null
@@ -1 +0,0 @@
-Rename some handlers and config modules to not duplicate the top-level module.
diff --git a/changelog.d/9817.misc b/changelog.d/9817.misc
deleted file mode 100644
index 8aa8895f05..0000000000
--- a/changelog.d/9817.misc
+++ /dev/null
@@ -1 +0,0 @@
-Fix a long-standing bug which caused `max_upload_size` to not be correctly enforced.
diff --git a/changelog.d/9819.feature b/changelog.d/9819.feature
deleted file mode 100644
index f56b0bb3bd..0000000000
--- a/changelog.d/9819.feature
+++ /dev/null
@@ -1 +0,0 @@
-Add experimental support for handling presence on a worker.
diff --git a/changelog.d/9820.feature b/changelog.d/9820.feature
deleted file mode 100644
index f56b0bb3bd..0000000000
--- a/changelog.d/9820.feature
+++ /dev/null
@@ -1 +0,0 @@
-Add experimental support for handling presence on a worker.
diff --git a/changelog.d/9821.misc b/changelog.d/9821.misc
deleted file mode 100644
index 03b2d2ed4d..0000000000
--- a/changelog.d/9821.misc
+++ /dev/null
@@ -1 +0,0 @@
-Reduce CPU usage of the user directory by reusing existing calculated room membership.
\ No newline at end of file
diff --git a/changelog.d/9825.misc b/changelog.d/9825.misc
deleted file mode 100644
index 42f3f15619..0000000000
--- a/changelog.d/9825.misc
+++ /dev/null
@@ -1 +0,0 @@
-Small speed up for joining large remote rooms.
diff --git a/changelog.d/9828.feature b/changelog.d/9828.feature
deleted file mode 100644
index f56b0bb3bd..0000000000
--- a/changelog.d/9828.feature
+++ /dev/null
@@ -1 +0,0 @@
-Add experimental support for handling presence on a worker.
diff --git a/changelog.d/9832.feature b/changelog.d/9832.feature
deleted file mode 100644
index e76395fbe8..0000000000
--- a/changelog.d/9832.feature
+++ /dev/null
@@ -1 +0,0 @@
-Don't return an error when a user attempts to renew their account multiple times with the same token. Instead, state when their account is set to expire. This change concerns the optional account validity feature.
\ No newline at end of file
diff --git a/changelog.d/9833.bugfix b/changelog.d/9833.bugfix
deleted file mode 100644
index 56f9c9626b..0000000000
--- a/changelog.d/9833.bugfix
+++ /dev/null
@@ -1 +0,0 @@
-Limit the size of HTTP responses read over federation.
diff --git a/changelog.d/9838.misc b/changelog.d/9838.misc
deleted file mode 100644
index b98ce56309..0000000000
--- a/changelog.d/9838.misc
+++ /dev/null
@@ -1 +0,0 @@
-Introduce flake8-bugbear to the test suite and fix some of its lint violations.
\ No newline at end of file
diff --git a/changelog.d/9845.misc b/changelog.d/9845.misc
deleted file mode 100644
index 875dd6d131..0000000000
--- a/changelog.d/9845.misc
+++ /dev/null
@@ -1 +0,0 @@
-Only store the raw data in the in-memory caches, rather than objects that include references to e.g. the data stores.
diff --git a/changelog.d/9850.feature b/changelog.d/9850.feature
deleted file mode 100644
index f56b0bb3bd..0000000000
--- a/changelog.d/9850.feature
+++ /dev/null
@@ -1 +0,0 @@
-Add experimental support for handling presence on a worker.
diff --git a/changelog.d/9855.misc b/changelog.d/9855.misc
deleted file mode 100644
index 6a3d700fde..0000000000
--- a/changelog.d/9855.misc
+++ /dev/null
@@ -1 +0,0 @@
-Limit length of accepted email addresses.
diff --git a/changelog.d/9856.misc b/changelog.d/9856.misc
deleted file mode 100644
index d67e8c386a..0000000000
--- a/changelog.d/9856.misc
+++ /dev/null
@@ -1 +0,0 @@
-Remove redundant `synapse.types.Collection` type definition.
diff --git a/changelog.d/9858.misc b/changelog.d/9858.misc
deleted file mode 100644
index f7e286fa69..0000000000
--- a/changelog.d/9858.misc
+++ /dev/null
@@ -1 +0,0 @@
-Handle recently added rate limits correctly when using `--no-rate-limit` with the demo scripts.
diff --git a/changelog.d/9867.bugfix b/changelog.d/9867.bugfix
deleted file mode 100644
index f236de247d..0000000000
--- a/changelog.d/9867.bugfix
+++ /dev/null
@@ -1 +0,0 @@
-Fix a bug which could cause Synapse to get stuck in a loop of resyncing device lists.
diff --git a/changelog.d/9868.bugfix b/changelog.d/9868.bugfix
deleted file mode 100644
index e2b4f97ad5..0000000000
--- a/changelog.d/9868.bugfix
+++ /dev/null
@@ -1 +0,0 @@
-Fix a long-standing bug where errors from federation did not propagate to the client.
diff --git a/changelog.d/9871.misc b/changelog.d/9871.misc
deleted file mode 100644
index b19acfab62..0000000000
--- a/changelog.d/9871.misc
+++ /dev/null
@@ -1 +0,0 @@
-Disable invite rate-limiting by default when running the unit tests.
\ No newline at end of file
diff --git a/changelog.d/9874.misc b/changelog.d/9874.misc
deleted file mode 100644
index ba1097e65e..0000000000
--- a/changelog.d/9874.misc
+++ /dev/null
@@ -1 +0,0 @@
-Pass a reactor into `SynapseSite` to make testing easier.
diff --git a/changelog.d/9875.misc b/changelog.d/9875.misc
deleted file mode 100644
index 9345c0bf45..0000000000
--- a/changelog.d/9875.misc
+++ /dev/null
@@ -1 +0,0 @@
-Make `DomainSpecificString` an `attrs` class.
diff --git a/changelog.d/9876.misc b/changelog.d/9876.misc
deleted file mode 100644
index 28390e32e6..0000000000
--- a/changelog.d/9876.misc
+++ /dev/null
@@ -1 +0,0 @@
-Add type hints to `synapse.api.auth` and `synapse.api.auth_blocking` modules.
diff --git a/changelog.d/9878.misc b/changelog.d/9878.misc
deleted file mode 100644
index 927876852d..0000000000
--- a/changelog.d/9878.misc
+++ /dev/null
@@ -1 +0,0 @@
-Remove redundant `_PushHTTPChannel` test class.
diff --git a/changelog.d/9879.misc b/changelog.d/9879.misc
deleted file mode 100644
index c9ca37cf48..0000000000
--- a/changelog.d/9879.misc
+++ /dev/null
@@ -1 +0,0 @@
-Remove backwards-compatibility code for Python versions < 3.6.
\ No newline at end of file
diff --git a/changelog.d/9887.misc b/changelog.d/9887.misc
deleted file mode 100644
index 650ebf85e6..0000000000
--- a/changelog.d/9887.misc
+++ /dev/null
@@ -1 +0,0 @@
-Small performance improvement around handling new local presence updates.
diff --git a/changelog.d/9900.bugfix b/changelog.d/9900.bugfix
new file mode 100644
index 0000000000..a8470fca3f
--- /dev/null
+++ b/changelog.d/9900.bugfix
@@ -0,0 +1 @@
+Fix tight loop handling presence replication when using workers. Introduced in v1.33.0rc1.
diff --git a/synapse/__init__.py b/synapse/__init__.py
index fbd49a93e1..5bbaa62de2 100644
--- a/synapse/__init__.py
+++ b/synapse/__init__.py
@@ -47,7 +47,7 @@ try:
 except ImportError:
     pass
 
-__version__ = "1.32.2"
+__version__ = "1.33.0rc1"
 
 if bool(os.environ.get("SYNAPSE_TEST_PATCH_LOG_CONTEXTS", False)):
     # We import here so that we don't have to install a bunch of deps when
diff --git a/synapse/federation/sender/__init__.py b/synapse/federation/sender/__init__.py
index 088260c2e9..deb40f4610 100644
--- a/synapse/federation/sender/__init__.py
+++ b/synapse/federation/sender/__init__.py
@@ -14,16 +14,7 @@
 
 import abc
 import logging
-from typing import (
-    TYPE_CHECKING,
-    Dict,
-    Hashable,
-    Iterable,
-    List,
-    Optional,
-    Set,
-    Tuple,
-)
+from typing import TYPE_CHECKING, Dict, Hashable, Iterable, List, Optional, Set, Tuple
 
 from prometheus_client import Counter
 
@@ -35,10 +26,7 @@ from synapse.events import EventBase
 from synapse.federation.sender.per_destination_queue import PerDestinationQueue
 from synapse.federation.sender.transaction_manager import TransactionManager
 from synapse.federation.units import Edu
-from synapse.logging.context import (
-    make_deferred_yieldable,
-    run_in_background,
-)
+from synapse.logging.context import make_deferred_yieldable, run_in_background
 from synapse.metrics import (
     LaterGauge,
     event_processing_loop_counter,
diff --git a/synapse/handlers/presence.py b/synapse/handlers/presence.py
index 969c73c1e7..12df35f26e 100644
--- a/synapse/handlers/presence.py
+++ b/synapse/handlers/presence.py
@@ -2026,18 +2026,40 @@ class PresenceFederationQueue:
             )
             return result["updates"], result["upto_token"], result["limited"]
 
+        # If the from_token is the current token then there's nothing to return
+        # and we can trivially no-op.
+        if from_token == self._next_id - 1:
+            return [], upto_token, False
+
         # We can find the correct position in the queue by noting that there is
         # exactly one entry per stream ID, and that the last entry has an ID of
         # `self._next_id - 1`, so we can count backwards from the end.
         #
+        # Since we are returning all states in the range `from_token < stream_id
+        # <= upto_token` we look for the index with a `stream_id` of `from_token
+        # + 1`.
+        #
         # Since the start of the queue is periodically truncated we need to
         # handle the case where `from_token` stream ID has already been dropped.
-        start_idx = max(from_token - self._next_id, -len(self._queue))
+        start_idx = max(from_token + 1 - self._next_id, -len(self._queue))
 
         to_send = []  # type: List[Tuple[int, Tuple[str, str]]]
         limited = False
         new_id = upto_token
         for _, stream_id, destinations, user_ids in self._queue[start_idx:]:
+            if stream_id <= from_token:
+                # Paranoia check that we are actually only sending states that
+                # are have stream_id strictly greater than from_token. We should
+                # never hit this.
+                logger.warning(
+                    "Tried returning presence federation stream ID: %d less than from_token: %d (next_id: %d, len: %d)",
+                    stream_id,
+                    from_token,
+                    self._next_id,
+                    len(self._queue),
+                )
+                continue
+
             if stream_id > upto_token:
                 break
 
diff --git a/tests/handlers/test_presence.py b/tests/handlers/test_presence.py
index 61271cd084..ce330e79cc 100644
--- a/tests/handlers/test_presence.py
+++ b/tests/handlers/test_presence.py
@@ -509,6 +509,14 @@ class PresenceFederationQueueTestCase(unittest.HomeserverTestCase):
 
         self.assertCountEqual(rows, expected_rows)
 
+        now_token = self.queue.get_current_token(self.instance_name)
+        rows, upto_token, limited = self.get_success(
+            self.queue.get_replication_rows("master", upto_token, now_token, 10)
+        )
+        self.assertEqual(upto_token, now_token)
+        self.assertFalse(limited)
+        self.assertCountEqual(rows, [])
+
     def test_send_and_get_split(self):
         state1 = UserPresenceState.default("@user1:test")
         state2 = UserPresenceState.default("@user2:test")
@@ -538,6 +546,20 @@ class PresenceFederationQueueTestCase(unittest.HomeserverTestCase):
 
         self.assertCountEqual(rows, expected_rows)
 
+        now_token = self.queue.get_current_token(self.instance_name)
+        rows, upto_token, limited = self.get_success(
+            self.queue.get_replication_rows("master", upto_token, now_token, 10)
+        )
+
+        self.assertEqual(upto_token, now_token)
+        self.assertFalse(limited)
+
+        expected_rows = [
+            (2, ("dest3", "@user3:test")),
+        ]
+
+        self.assertCountEqual(rows, expected_rows)
+
     def test_clear_queue_all(self):
         state1 = UserPresenceState.default("@user1:test")
         state2 = UserPresenceState.default("@user2:test")