Merge remote-tracking branch 'origin/develop' into erikj/destination_retry_cache

51 files changed, 128 insertions, 62 deletions

diff --git a/CHANGES.md b/CHANGES.md
index ee864c3c63..45d3cdb131 100644
--- a/CHANGES.md
+++ b/CHANGES.md
@@ -1,3 +1,77 @@
+Synapse 0.33.5.1 (2018-09-25)
+=============================
+
+Internal Changes
+----------------
+
+- Fix incompatibility with older Twisted version in tests. Thanks 
+  @OlegGirko! ([\#3940](https://github.com/matrix-org/synapse/issues/3940))
+
+
+Synapse 0.33.5 (2018-09-24)
+===========================
+
+No significant changes.
+
+
+Synapse 0.33.5rc1 (2018-09-17)
+==============================
+
+Features
+--------
+
+- Python 3.5 and 3.6 support is now in beta. ([\#3576](https://github.com/matrix-org/synapse/issues/3576))
+- Implement `event_format` filter param in `/sync` ([\#3790](https://github.com/matrix-org/synapse/issues/3790))
+- Add synapse_admin_mau:registered_reserved_users metric to expose number of real reaserved users ([\#3846](https://github.com/matrix-org/synapse/issues/3846))
+
+
+Bugfixes
+--------
+
+- Remove connection ID for replication prometheus metrics, as it creates a large number of new series. ([\#3788](https://github.com/matrix-org/synapse/issues/3788))
+- guest users should not be part of mau total ([\#3800](https://github.com/matrix-org/synapse/issues/3800))
+- Bump dependency on pyopenssl 16.x, to avoid incompatibility with recent Twisted. ([\#3804](https://github.com/matrix-org/synapse/issues/3804))
+- Fix existing room tags not coming down sync when joining a room ([\#3810](https://github.com/matrix-org/synapse/issues/3810))
+- Fix jwt import check ([\#3824](https://github.com/matrix-org/synapse/issues/3824))
+- fix VOIP crashes under Python 3 (#3821) ([\#3835](https://github.com/matrix-org/synapse/issues/3835))
+- Fix manhole so that it works with latest openssh clients ([\#3841](https://github.com/matrix-org/synapse/issues/3841))
+- Fix outbound requests occasionally wedging, which can result in federation breaking between servers. ([\#3845](https://github.com/matrix-org/synapse/issues/3845))
+- Show heroes if room name/canonical alias has been deleted ([\#3851](https://github.com/matrix-org/synapse/issues/3851))
+- Fix handling of redacted events from federation ([\#3859](https://github.com/matrix-org/synapse/issues/3859))
+-  ([\#3874](https://github.com/matrix-org/synapse/issues/3874))
+- Mitigate outbound federation randomly becoming wedged ([\#3875](https://github.com/matrix-org/synapse/issues/3875))
+
+
+Internal Changes
+----------------
+
+- CircleCI tests now run on the potential merge of a PR. ([\#3704](https://github.com/matrix-org/synapse/issues/3704))
+- http/ is now ported to Python 3. ([\#3771](https://github.com/matrix-org/synapse/issues/3771))
+- Improve human readable error messages for threepid registration/account update ([\#3789](https://github.com/matrix-org/synapse/issues/3789))
+- Make /sync slightly faster by avoiding needless copies ([\#3795](https://github.com/matrix-org/synapse/issues/3795))
+- handlers/ is now ported to Python 3. ([\#3803](https://github.com/matrix-org/synapse/issues/3803))
+- Limit the number of PDUs/EDUs per federation transaction ([\#3805](https://github.com/matrix-org/synapse/issues/3805))
+- Only start postgres instance for postgres tests on Travis CI ([\#3806](https://github.com/matrix-org/synapse/issues/3806))
+- tests/ is now ported to Python 3. ([\#3808](https://github.com/matrix-org/synapse/issues/3808))
+- crypto/ is now ported to Python 3. ([\#3822](https://github.com/matrix-org/synapse/issues/3822))
+- rest/ is now ported to Python 3. ([\#3823](https://github.com/matrix-org/synapse/issues/3823))
+- add some logging for the keyring queue ([\#3826](https://github.com/matrix-org/synapse/issues/3826))
+- speed up lazy loading by 2-3x ([\#3827](https://github.com/matrix-org/synapse/issues/3827))
+- Improved Dockerfile to remove build requirements after building reducing the image size. ([\#3834](https://github.com/matrix-org/synapse/issues/3834))
+- Disable lazy loading for incremental syncs for now ([\#3840](https://github.com/matrix-org/synapse/issues/3840))
+- federation/ is now ported to Python 3. ([\#3847](https://github.com/matrix-org/synapse/issues/3847))
+- Log when we retry outbound requests ([\#3853](https://github.com/matrix-org/synapse/issues/3853))
+- Removed some excess logging messages. ([\#3855](https://github.com/matrix-org/synapse/issues/3855))
+- Speed up purge history for rooms that have been previously purged ([\#3856](https://github.com/matrix-org/synapse/issues/3856))
+- Refactor some HTTP timeout code. ([\#3857](https://github.com/matrix-org/synapse/issues/3857))
+- Fix running merged builds on CircleCI ([\#3858](https://github.com/matrix-org/synapse/issues/3858))
+- Fix typo in replication stream exception. ([\#3860](https://github.com/matrix-org/synapse/issues/3860))
+- Add in flight real time metrics for Measure blocks ([\#3871](https://github.com/matrix-org/synapse/issues/3871))
+- Disable buffering and automatic retrying in treq requests to prevent timeouts. ([\#3872](https://github.com/matrix-org/synapse/issues/3872))
+- mention jemalloc in the README ([\#3877](https://github.com/matrix-org/synapse/issues/3877))
+- Remove unmaintained "nuke-room-from-db.sh" script ([\#3888](https://github.com/matrix-org/synapse/issues/3888))
+
+
 Synapse 0.33.4 (2018-09-07)
 ===========================
 
diff --git a/changelog.d/3576.feature b/changelog.d/3576.feature
deleted file mode 100644
index 02a10e370d..0000000000
--- a/changelog.d/3576.feature
+++ /dev/null
@@ -1 +0,0 @@
-Python 3.5+ is now supported.
diff --git a/changelog.d/3578.bugfix b/changelog.d/3578.bugfix
new file mode 100644
index 0000000000..9c52b6fa7e
--- /dev/null
+++ b/changelog.d/3578.bugfix
@@ -0,0 +1 @@
+Fix problem when playing media from Chrome using direct URL (thanks @remjey!)
diff --git a/changelog.d/3704.misc b/changelog.d/3704.misc
deleted file mode 100644
index aaae0fbd63..0000000000
--- a/changelog.d/3704.misc
+++ /dev/null
@@ -1 +0,0 @@
-CircleCI tests now run on the potential merge of a PR.
diff --git a/changelog.d/3771.misc b/changelog.d/3771.misc
deleted file mode 100644
index 47aa34bc04..0000000000
--- a/changelog.d/3771.misc
+++ /dev/null
@@ -1 +0,0 @@
-http/ is now ported to Python 3.
diff --git a/changelog.d/3788.bugfix b/changelog.d/3788.bugfix
deleted file mode 100644
index 72316fb881..0000000000
--- a/changelog.d/3788.bugfix
+++ /dev/null
@@ -1 +0,0 @@
-Remove connection ID for replication prometheus metrics, as it creates a large number of new series.
diff --git a/changelog.d/3789.misc b/changelog.d/3789.misc
deleted file mode 100644
index d2d5d91091..0000000000
--- a/changelog.d/3789.misc
+++ /dev/null
@@ -1 +0,0 @@
-Improve human readable error messages for threepid registration/account update
diff --git a/changelog.d/3790.feature b/changelog.d/3790.feature
deleted file mode 100644
index 2c4ac62fb5..0000000000
--- a/changelog.d/3790.feature
+++ /dev/null
@@ -1 +0,0 @@
-Implement `event_format` filter param in `/sync`
diff --git a/changelog.d/3795.misc b/changelog.d/3795.misc
deleted file mode 100644
index 9f64ee5e2b..0000000000
--- a/changelog.d/3795.misc
+++ /dev/null
@@ -1 +0,0 @@
-Make /sync slightly faster by avoiding needless copies
diff --git a/changelog.d/3800.bugfix b/changelog.d/3800.bugfix
deleted file mode 100644
index 6b2e18b4a6..0000000000
--- a/changelog.d/3800.bugfix
+++ /dev/null
@@ -1 +0,0 @@
-guest users should not be part of mau total
diff --git a/changelog.d/3803.misc b/changelog.d/3803.misc
deleted file mode 100644
index 2b60653c29..0000000000
--- a/changelog.d/3803.misc
+++ /dev/null
@@ -1 +0,0 @@
-handlers/ is now ported to Python 3.
diff --git a/changelog.d/3804.bugfix b/changelog.d/3804.bugfix
deleted file mode 100644
index a0cef20e3f..0000000000
--- a/changelog.d/3804.bugfix
+++ /dev/null
@@ -1 +0,0 @@
-Bump dependency on pyopenssl 16.x, to avoid incompatibility with recent Twisted.
diff --git a/changelog.d/3805.misc b/changelog.d/3805.misc
deleted file mode 100644
index 257feeb071..0000000000
--- a/changelog.d/3805.misc
+++ /dev/null
@@ -1 +0,0 @@
-Limit the number of PDUs/EDUs per federation transaction
diff --git a/changelog.d/3806.misc b/changelog.d/3806.misc
deleted file mode 100644
index 3c722eef2d..0000000000
--- a/changelog.d/3806.misc
+++ /dev/null
@@ -1 +0,0 @@
-Only start postgres instance for postgres tests on Travis CI
diff --git a/changelog.d/3808.misc b/changelog.d/3808.misc
deleted file mode 100644
index e5e1cd9e0e..0000000000
--- a/changelog.d/3808.misc
+++ /dev/null
@@ -1 +0,0 @@
-tests/ is now ported to Python 3.
diff --git a/changelog.d/3810.bugfix b/changelog.d/3810.bugfix
deleted file mode 100644
index 2b938a81ae..0000000000
--- a/changelog.d/3810.bugfix
+++ /dev/null
@@ -1 +0,0 @@
-Fix existing room tags not coming down sync when joining a room
diff --git a/changelog.d/3822.misc b/changelog.d/3822.misc
deleted file mode 100644
index 5250f31896..0000000000
--- a/changelog.d/3822.misc
+++ /dev/null
@@ -1 +0,0 @@
-crypto/ is now ported to Python 3.
diff --git a/changelog.d/3823.misc b/changelog.d/3823.misc
deleted file mode 100644
index 0da491ddaa..0000000000
--- a/changelog.d/3823.misc
+++ /dev/null
@@ -1 +0,0 @@
-rest/ is now ported to Python 3.
diff --git a/changelog.d/3824.bugfix b/changelog.d/3824.bugfix
deleted file mode 100644
index 99f199dcc6..0000000000
--- a/changelog.d/3824.bugfix
+++ /dev/null
@@ -1 +0,0 @@
-Fix jwt import check
\ No newline at end of file
diff --git a/changelog.d/3826.misc b/changelog.d/3826.misc
deleted file mode 100644
index a4d9a012f9..0000000000
--- a/changelog.d/3826.misc
+++ /dev/null
@@ -1 +0,0 @@
-add some logging for the keyring queue
diff --git a/changelog.d/3827.misc b/changelog.d/3827.misc
deleted file mode 100644
index bc294706cf..0000000000
--- a/changelog.d/3827.misc
+++ /dev/null
@@ -1 +0,0 @@
-speed up lazy loading by 2-3x
diff --git a/changelog.d/3834.misc b/changelog.d/3834.misc
deleted file mode 100644
index 8902f8fba7..0000000000
--- a/changelog.d/3834.misc
+++ /dev/null
@@ -1 +0,0 @@
-Improved Dockerfile to remove build requirements after building reducing the image size.
diff --git a/changelog.d/3835.bugfix b/changelog.d/3835.bugfix
deleted file mode 100644
index 00dbcbc8dc..0000000000
--- a/changelog.d/3835.bugfix
+++ /dev/null
@@ -1 +0,0 @@
-fix VOIP crashes under Python 3 (#3821)
diff --git a/changelog.d/3840.misc b/changelog.d/3840.misc
deleted file mode 100644
index b9585ae9be..0000000000
--- a/changelog.d/3840.misc
+++ /dev/null
@@ -1 +0,0 @@
-Disable lazy loading for incremental syncs for now
diff --git a/changelog.d/3841.bugfix b/changelog.d/3841.bugfix
deleted file mode 100644
index 2a48a7dd66..0000000000
--- a/changelog.d/3841.bugfix
+++ /dev/null
@@ -1 +0,0 @@
-Fix manhole so that it works with latest openssh clients
diff --git a/changelog.d/3845.bugfix b/changelog.d/3845.bugfix
deleted file mode 100644
index 5b7e8f1934..0000000000
--- a/changelog.d/3845.bugfix
+++ /dev/null
@@ -1 +0,0 @@
-Fix outbound requests occasionally wedging, which can result in federation breaking between servers.
diff --git a/changelog.d/3846.feature b/changelog.d/3846.feature
deleted file mode 100644
index 453c11d3f8..0000000000
--- a/changelog.d/3846.feature
+++ /dev/null
@@ -1 +0,0 @@
-Add synapse_admin_mau:registered_reserved_users metric to expose number of real reaserved users 
diff --git a/changelog.d/3847.misc b/changelog.d/3847.misc
deleted file mode 100644
index bf8b5afea4..0000000000
--- a/changelog.d/3847.misc
+++ /dev/null
@@ -1 +0,0 @@
-federation/ is now ported to Python 3.
\ No newline at end of file
diff --git a/changelog.d/3851.bugfix b/changelog.d/3851.bugfix
deleted file mode 100644
index b53a9efe7b..0000000000
--- a/changelog.d/3851.bugfix
+++ /dev/null
@@ -1 +0,0 @@
-Show heroes if room name/canonical alias has been deleted
diff --git a/changelog.d/3853.misc b/changelog.d/3853.misc
deleted file mode 100644
index db45d4983d..0000000000
--- a/changelog.d/3853.misc
+++ /dev/null
@@ -1 +0,0 @@
-Log when we retry outbound requests
diff --git a/changelog.d/3855.misc b/changelog.d/3855.misc
deleted file mode 100644
index a25bb020ba..0000000000
--- a/changelog.d/3855.misc
+++ /dev/null
@@ -1 +0,0 @@
-Removed some excess logging messages.
\ No newline at end of file
diff --git a/changelog.d/3856.misc b/changelog.d/3856.misc
deleted file mode 100644
index 36c311eb3d..0000000000
--- a/changelog.d/3856.misc
+++ /dev/null
@@ -1 +0,0 @@
-Speed up purge history for rooms that have been previously purged
diff --git a/changelog.d/3857.misc b/changelog.d/3857.misc
deleted file mode 100644
index e128d193d9..0000000000
--- a/changelog.d/3857.misc
+++ /dev/null
@@ -1 +0,0 @@
-Refactor some HTTP timeout code.
\ No newline at end of file
diff --git a/changelog.d/3858.misc b/changelog.d/3858.misc
deleted file mode 100644
index 4644db5330..0000000000
--- a/changelog.d/3858.misc
+++ /dev/null
@@ -1 +0,0 @@
-Fix running merged builds on CircleCI
\ No newline at end of file
diff --git a/changelog.d/3859.bugfix b/changelog.d/3859.bugfix
deleted file mode 100644
index ec5b172464..0000000000
--- a/changelog.d/3859.bugfix
+++ /dev/null
@@ -1 +0,0 @@
-Fix handling of redacted events from federation
diff --git a/changelog.d/3860.misc b/changelog.d/3860.misc
deleted file mode 100644
index 364239d3e3..0000000000
--- a/changelog.d/3860.misc
+++ /dev/null
@@ -1 +0,0 @@
-Fix typo in replication stream exception.
diff --git a/changelog.d/3871.misc b/changelog.d/3871.misc
deleted file mode 100644
index dd9510ceb6..0000000000
--- a/changelog.d/3871.misc
+++ /dev/null
@@ -1 +0,0 @@
-Add in flight real time metrics for Measure blocks
diff --git a/changelog.d/3872.misc b/changelog.d/3872.misc
deleted file mode 100644
index b450c506d8..0000000000
--- a/changelog.d/3872.misc
+++ /dev/null
@@ -1 +0,0 @@
-Disable buffering and automatic retrying in treq requests to prevent timeouts.
\ No newline at end of file
diff --git a/changelog.d/3874.bugfix b/changelog.d/3874.bugfix
deleted file mode 100644
index e69de29bb2..0000000000
--- a/changelog.d/3874.bugfix
+++ /dev/null
diff --git a/changelog.d/3875.bugfix b/changelog.d/3875.bugfix
deleted file mode 100644
index 2d2147dd4b..0000000000
--- a/changelog.d/3875.bugfix
+++ /dev/null
@@ -1 +0,0 @@
-Mitigate outbound federation randomly becoming wedged
diff --git a/changelog.d/3877.misc b/changelog.d/3877.misc
deleted file mode 100644
index a80fec4bd8..0000000000
--- a/changelog.d/3877.misc
+++ /dev/null
@@ -1 +0,0 @@
-mention jemalloc in the README
diff --git a/changelog.d/3888.misc b/changelog.d/3888.misc
deleted file mode 100644
index a10ede547e..0000000000
--- a/changelog.d/3888.misc
+++ /dev/null
@@ -1 +0,0 @@
-Remove unmaintained "nuke-room-from-db.sh" script
diff --git a/changelog.d/3925.misc b/changelog.d/3925.misc
new file mode 100644
index 0000000000..3e41f78ff5
--- /dev/null
+++ b/changelog.d/3925.misc
@@ -0,0 +1 @@
+Fix spurious exceptions when remote http client closes conncetion
diff --git a/changelog.d/3936.bugfix b/changelog.d/3936.bugfix
new file mode 100644
index 0000000000..49b02b9e27
--- /dev/null
+++ b/changelog.d/3936.bugfix
@@ -0,0 +1 @@
+Fix out-of-bounds error when LLing yourself
diff --git a/changelog.d/3947.misc b/changelog.d/3947.misc
new file mode 100644
index 0000000000..5a9a22bed9
--- /dev/null
+++ b/changelog.d/3947.misc
@@ -0,0 +1 @@
+Require attrs 16.0.0 or later
diff --git a/synapse/__init__.py b/synapse/__init__.py
index 9dbe0b9f10..b1f7a89fba 100644
--- a/synapse/__init__.py
+++ b/synapse/__init__.py
@@ -27,4 +27,4 @@ try:
 except ImportError:
     pass
 
-__version__ = "0.33.4"
+__version__ = "0.33.5.1"
diff --git a/synapse/handlers/sync.py b/synapse/handlers/sync.py
index b598916b21..c7d69d9d80 100644
--- a/synapse/handlers/sync.py
+++ b/synapse/handlers/sync.py
@@ -713,10 +713,6 @@ class SyncHandler(object):
                     )
                 ]
 
-                # always make sure we LL ourselves so we know we're in the room
-                # (if we are), to fix https://github.com/vector-im/riot-web/issues/7209
-                types.append((EventTypes.Member, sync_config.user.to_string()))
-
                 # only apply the filtering to room members
                 filtered_types = [EventTypes.Member]
 
@@ -726,6 +722,13 @@ class SyncHandler(object):
             }
 
             if full_state:
+                if lazy_load_members:
+                    # always make sure we LL ourselves so we know we're in the room
+                    # (if we are) to fix https://github.com/vector-im/riot-web/issues/7209
+                    # We only need apply this on full state syncs given we disabled
+                    # LL for incr syncs in #3840.
+                    types.append((EventTypes.Member, sync_config.user.to_string()))
+
                 if batch:
                     current_state_ids = yield self.store.get_state_ids_for_event(
                         batch.events[-1].event_id, types=types,
@@ -794,7 +797,7 @@ class SyncHandler(object):
             else:
                 state_ids = {}
                 if lazy_load_members:
-                    if types:
+                    if types and batch.events:
                         # We're returning an incremental sync, with no
                         # "gap" since the previous sync, so normally there would be
                         # no state to return.
diff --git a/synapse/http/server.py b/synapse/http/server.py
index 2d5c23e673..b4b25cab19 100644
--- a/synapse/http/server.py
+++ b/synapse/http/server.py
@@ -84,10 +84,21 @@ def wrap_json_request_handler(h):
             logger.info(
                 "%s SynapseError: %s - %s", request, code, e.msg
             )
-            respond_with_json(
-                request, code, e.error_dict(), send_cors=True,
-                pretty_print=_request_user_agent_is_curl(request),
-            )
+
+            # Only respond with an error response if we haven't already started
+            # writing, otherwise lets just kill the connection
+            if request.startedWriting:
+                if request.transport:
+                    try:
+                        request.transport.abortConnection()
+                    except Exception:
+                        # abortConnection throws if the connection is already closed
+                        pass
+            else:
+                respond_with_json(
+                    request, code, e.error_dict(), send_cors=True,
+                    pretty_print=_request_user_agent_is_curl(request),
+                )
 
         except Exception:
             # failure.Failure() fishes the original Failure out
@@ -100,16 +111,26 @@ def wrap_json_request_handler(h):
                 request,
                 f.getTraceback().rstrip(),
             )
-            respond_with_json(
-                request,
-                500,
-                {
-                    "error": "Internal server error",
-                    "errcode": Codes.UNKNOWN,
-                },
-                send_cors=True,
-                pretty_print=_request_user_agent_is_curl(request),
-            )
+            # Only respond with an error response if we haven't already started
+            # writing, otherwise lets just kill the connection
+            if request.startedWriting:
+                if request.transport:
+                    try:
+                        request.transport.abortConnection()
+                    except Exception:
+                        # abortConnection throws if the connection is already closed
+                        pass
+            else:
+                respond_with_json(
+                    request,
+                    500,
+                    {
+                        "error": "Internal server error",
+                        "errcode": Codes.UNKNOWN,
+                    },
+                    send_cors=True,
+                    pretty_print=_request_user_agent_is_curl(request),
+                )
 
     return wrap_async_request_handler(wrapped_request_handler)
 
diff --git a/synapse/python_dependencies.py b/synapse/python_dependencies.py
index 0d8de600cf..c779f69fa0 100644
--- a/synapse/python_dependencies.py
+++ b/synapse/python_dependencies.py
@@ -58,7 +58,9 @@ REQUIREMENTS = {
     "phonenumbers>=8.2.0": ["phonenumbers"],
     "six": ["six"],
     "prometheus_client": ["prometheus_client"],
-    "attrs": ["attr"],
+
+    # we use attr.s(slots), which arrived in 16.0.0
+    "attrs>=16.0.0": ["attr>=16.0.0"],
     "netaddr>=0.7.18": ["netaddr"],
 }
 
diff --git a/synapse/rest/media/v1/download_resource.py b/synapse/rest/media/v1/download_resource.py
index ca90964d1d..f911b120b1 100644
--- a/synapse/rest/media/v1/download_resource.py
+++ b/synapse/rest/media/v1/download_resource.py
@@ -52,6 +52,7 @@ class DownloadResource(Resource):
             b" script-src 'none';"
             b" plugin-types application/pdf;"
             b" style-src 'unsafe-inline';"
+            b" media-src 'self';"
             b" object-src 'self';"
         )
         server_name, media_id, name = parse_media_id(request)
diff --git a/tests/http/test_fedclient.py b/tests/http/test_fedclient.py
index 66c09f63b6..f3cb1423f0 100644
--- a/tests/http/test_fedclient.py
+++ b/tests/http/test_fedclient.py
@@ -54,7 +54,7 @@ class FederationClientTests(HomeserverTestCase):
     def test_client_never_connect(self):
         """
         If the HTTP request is not connected and is timed out, it'll give a
-        ConnectingCancelledError.
+        ConnectingCancelledError or TimeoutError.
         """
         d = self.cl.get_json("testserv:8008", "foo/bar", timeout=10000)
 
@@ -76,7 +76,7 @@ class FederationClientTests(HomeserverTestCase):
         self.reactor.advance(10.5)
         f = self.failureResultOf(d)
 
-        self.assertIsInstance(f.value, ConnectingCancelledError)
+        self.assertIsInstance(f.value, (ConnectingCancelledError, TimeoutError))
 
     def test_client_connect_no_response(self):
         """