Merge branch 'develop' of github.com:matrix-org/synapse into erikj/faster_typing

80 files changed, 53 insertions, 27 deletions

diff --git a/changelog.d/3378.misc b/changelog.d/3378.misc
deleted file mode 100644
index 8f88f88e69..0000000000
--- a/changelog.d/3378.misc
+++ /dev/null
@@ -1 +0,0 @@
-Removed the link to the unmaintained matrix-synapse-auto-deploy project from the readme.
diff --git a/changelog.d/3576.feature b/changelog.d/3576.feature
new file mode 100644
index 0000000000..02a10e370d
--- /dev/null
+++ b/changelog.d/3576.feature
@@ -0,0 +1 @@
+Python 3.5+ is now supported.
diff --git a/changelog.d/3659.feature b/changelog.d/3659.feature
deleted file mode 100644
index a5b4821c09..0000000000
--- a/changelog.d/3659.feature
+++ /dev/null
@@ -1 +0,0 @@
-Support profile API endpoints on workers
diff --git a/changelog.d/3673.misc b/changelog.d/3673.misc
deleted file mode 100644
index d672111fb9..0000000000
--- a/changelog.d/3673.misc
+++ /dev/null
@@ -1 +0,0 @@
-Refactor state module to support multiple room versions
diff --git a/changelog.d/3680.feature b/changelog.d/3680.feature
deleted file mode 100644
index 4edaaf76a8..0000000000
--- a/changelog.d/3680.feature
+++ /dev/null
@@ -1 +0,0 @@
-Server notices for resource limit blocking
diff --git a/changelog.d/3704.misc b/changelog.d/3704.misc
new file mode 100644
index 0000000000..aaae0fbd63
--- /dev/null
+++ b/changelog.d/3704.misc
@@ -0,0 +1 @@
+CircleCI tests now run on the potential merge of a PR.
diff --git a/changelog.d/3722.bugfix b/changelog.d/3722.bugfix
deleted file mode 100644
index 16cbaf76cb..0000000000
--- a/changelog.d/3722.bugfix
+++ /dev/null
@@ -1 +0,0 @@
-Fix error collecting prometheus metrics when run on dedicated thread due to threading concurrency issues
diff --git a/changelog.d/3724.feature b/changelog.d/3724.feature
deleted file mode 100644
index 1b374ccf47..0000000000
--- a/changelog.d/3724.feature
+++ /dev/null
@@ -1 +0,0 @@
-Allow guests to use /rooms/:roomId/event/:eventId
diff --git a/changelog.d/3725.misc b/changelog.d/3725.misc
deleted file mode 100644
index 91ab9d7137..0000000000
--- a/changelog.d/3725.misc
+++ /dev/null
@@ -1 +0,0 @@
-The synapse.storage module has been ported to Python 3.
diff --git a/changelog.d/3726.misc b/changelog.d/3726.misc
deleted file mode 100644
index c4f66ec998..0000000000
--- a/changelog.d/3726.misc
+++ /dev/null
@@ -1 +0,0 @@
-Split the state_group_cache into member and non-member state events (and so speed up LL /sync)
diff --git a/changelog.d/3727.misc b/changelog.d/3727.misc
deleted file mode 100644
index 0b83220d90..0000000000
--- a/changelog.d/3727.misc
+++ /dev/null
@@ -1 +0,0 @@
-Log failure to authenticate remote servers as warnings (without stack traces)
diff --git a/changelog.d/3730.misc b/changelog.d/3730.misc
deleted file mode 100644
index b1ea84f732..0000000000
--- a/changelog.d/3730.misc
+++ /dev/null
@@ -1 +0,0 @@
-The CONTRIBUTING guidelines have been updated to mention our use of Markdown and that .misc files have content.
diff --git a/changelog.d/3734.misc b/changelog.d/3734.misc
deleted file mode 100644
index 4f6e4b3848..0000000000
--- a/changelog.d/3734.misc
+++ /dev/null
@@ -1 +0,0 @@
-Reference the need for an HTTP replication port when using the federation_reader worker
diff --git a/changelog.d/3735.misc b/changelog.d/3735.misc
deleted file mode 100644
index f17004be71..0000000000
--- a/changelog.d/3735.misc
+++ /dev/null
@@ -1 +0,0 @@
-Fix minor spelling error in federation client documentation.
diff --git a/changelog.d/3737.misc b/changelog.d/3737.misc
deleted file mode 100644
index 0361da4ebc..0000000000
--- a/changelog.d/3737.misc
+++ /dev/null
@@ -1 +0,0 @@
-Remove redundant state resolution function
diff --git a/changelog.d/3740.misc b/changelog.d/3740.misc
deleted file mode 100644
index 4dcb7fb5de..0000000000
--- a/changelog.d/3740.misc
+++ /dev/null
@@ -1 +0,0 @@
-The test suite now passes on PostgreSQL.
diff --git a/changelog.d/3746.misc b/changelog.d/3746.misc
deleted file mode 100644
index fc00ee773a..0000000000
--- a/changelog.d/3746.misc
+++ /dev/null
@@ -1 +0,0 @@
-Fix MAU cache invalidation due to missing yield
diff --git a/changelog.d/3747.bugfix b/changelog.d/3747.bugfix
deleted file mode 100644
index c41e2a1213..0000000000
--- a/changelog.d/3747.bugfix
+++ /dev/null
@@ -1 +0,0 @@
-Fix bug where we resent "limit exceeded" server notices repeatedly
diff --git a/changelog.d/3749.feature b/changelog.d/3749.feature
deleted file mode 100644
index 9f8837b106..0000000000
--- a/changelog.d/3749.feature
+++ /dev/null
@@ -1 +0,0 @@
-Add mau_trial_days config param, so that users only get counted as MAU after N days.
diff --git a/changelog.d/3751.feature b/changelog.d/3751.feature
deleted file mode 100644
index dc9742b15b..0000000000
--- a/changelog.d/3751.feature
+++ /dev/null
@@ -1 +0,0 @@
-Require twisted 17.1 or later (fixes [#3741](https://github.com/matrix-org/synapse/issues/3741)).
diff --git a/changelog.d/3753.bugfix b/changelog.d/3753.bugfix
deleted file mode 100644
index b4301267df..0000000000
--- a/changelog.d/3753.bugfix
+++ /dev/null
@@ -1 +0,0 @@
-Fix bug where we broke sync when using limit_usage_by_mau but hadn't configured server notices
diff --git a/changelog.d/3754.bugfix b/changelog.d/3754.bugfix
deleted file mode 100644
index 6e3ec80194..0000000000
--- a/changelog.d/3754.bugfix
+++ /dev/null
@@ -1 +0,0 @@
-Fix 'federation_domain_whitelist' such that an empty list correctly blocks all outbound federation traffic
diff --git a/changelog.d/3755.bugfix b/changelog.d/3755.bugfix
deleted file mode 100644
index 6a1f83f0ce..0000000000
--- a/changelog.d/3755.bugfix
+++ /dev/null
@@ -1 +0,0 @@
-Fix tagging of server notice rooms
diff --git a/changelog.d/3756.bugfix b/changelog.d/3756.bugfix
deleted file mode 100644
index 6a1f83f0ce..0000000000
--- a/changelog.d/3756.bugfix
+++ /dev/null
@@ -1 +0,0 @@
-Fix tagging of server notice rooms
diff --git a/changelog.d/3758.bugfix b/changelog.d/3758.bugfix
deleted file mode 100644
index 862739bfe8..0000000000
--- a/changelog.d/3758.bugfix
+++ /dev/null
@@ -1 +0,0 @@
-Fix 'admin_uri' config variable and error parameter to be 'admin_contact' to match the spec.
diff --git a/changelog.d/3760.bugfix b/changelog.d/3760.bugfix
deleted file mode 100644
index ce61fb8a2b..0000000000
--- a/changelog.d/3760.bugfix
+++ /dev/null
@@ -1 +0,0 @@
-Don't return non-LL-member state in incremental sync state blocks
diff --git a/changelog.d/3764.misc b/changelog.d/3764.misc
deleted file mode 100644
index f3614f1982..0000000000
--- a/changelog.d/3764.misc
+++ /dev/null
@@ -1 +0,0 @@
-Make sure that we close db connections opened during init
\ No newline at end of file
diff --git a/changelog.d/3768.bugfix b/changelog.d/3768.bugfix
deleted file mode 100644
index a039a7fa68..0000000000
--- a/changelog.d/3768.bugfix
+++ /dev/null
@@ -1 +0,0 @@
-Fix bug in sending presence over federation
diff --git a/changelog.d/3771.misc b/changelog.d/3771.misc
new file mode 100644
index 0000000000..47aa34bc04
--- /dev/null
+++ b/changelog.d/3771.misc
@@ -0,0 +1 @@
+http/ is now ported to Python 3.
diff --git a/changelog.d/3777.bugfix b/changelog.d/3777.bugfix
deleted file mode 100644
index 46efc543a7..0000000000
--- a/changelog.d/3777.bugfix
+++ /dev/null
@@ -1 +0,0 @@
-Fix bug where preserved threepid user comes to sign up and server is mau blocked
diff --git a/changelog.d/3788.bugfix b/changelog.d/3788.bugfix
new file mode 100644
index 0000000000..72316fb881
--- /dev/null
+++ b/changelog.d/3788.bugfix
@@ -0,0 +1 @@
+Remove connection ID for replication prometheus metrics, as it creates a large number of new series.
diff --git a/changelog.d/3790.feature b/changelog.d/3790.feature
new file mode 100644
index 0000000000..2c4ac62fb5
--- /dev/null
+++ b/changelog.d/3790.feature
@@ -0,0 +1 @@
+Implement `event_format` filter param in `/sync`
diff --git a/changelog.d/3795.misc b/changelog.d/3795.misc
new file mode 100644
index 0000000000..9f64ee5e2b
--- /dev/null
+++ b/changelog.d/3795.misc
@@ -0,0 +1 @@
+Make /sync slightly faster by avoiding needless copies
diff --git a/changelog.d/3800.bugfix b/changelog.d/3800.bugfix
new file mode 100644
index 0000000000..6b2e18b4a6
--- /dev/null
+++ b/changelog.d/3800.bugfix
@@ -0,0 +1 @@
+guest users should not be part of mau total
diff --git a/changelog.d/3803.misc b/changelog.d/3803.misc
new file mode 100644
index 0000000000..2b60653c29
--- /dev/null
+++ b/changelog.d/3803.misc
@@ -0,0 +1 @@
+handlers/ is now ported to Python 3.
diff --git a/changelog.d/3804.bugfix b/changelog.d/3804.bugfix
new file mode 100644
index 0000000000..a0cef20e3f
--- /dev/null
+++ b/changelog.d/3804.bugfix
@@ -0,0 +1 @@
+Bump dependency on pyopenssl 16.x, to avoid incompatibility with recent Twisted.
diff --git a/changelog.d/3805.misc b/changelog.d/3805.misc
new file mode 100644
index 0000000000..257feeb071
--- /dev/null
+++ b/changelog.d/3805.misc
@@ -0,0 +1 @@
+Limit the number of PDUs/EDUs per federation transaction
diff --git a/changelog.d/3806.misc b/changelog.d/3806.misc
new file mode 100644
index 0000000000..3c722eef2d
--- /dev/null
+++ b/changelog.d/3806.misc
@@ -0,0 +1 @@
+Only start postgres instance for postgres tests on Travis CI
diff --git a/changelog.d/3808.misc b/changelog.d/3808.misc
new file mode 100644
index 0000000000..e5e1cd9e0e
--- /dev/null
+++ b/changelog.d/3808.misc
@@ -0,0 +1 @@
+tests/ is now ported to Python 3.
diff --git a/changelog.d/3810.bugfix b/changelog.d/3810.bugfix
new file mode 100644
index 0000000000..2b938a81ae
--- /dev/null
+++ b/changelog.d/3810.bugfix
@@ -0,0 +1 @@
+Fix existing room tags not coming down sync when joining a room
diff --git a/changelog.d/3822.misc b/changelog.d/3822.misc
new file mode 100644
index 0000000000..5250f31896
--- /dev/null
+++ b/changelog.d/3822.misc
@@ -0,0 +1 @@
+crypto/ is now ported to Python 3.
diff --git a/changelog.d/3823.misc b/changelog.d/3823.misc
new file mode 100644
index 0000000000..0da491ddaa
--- /dev/null
+++ b/changelog.d/3823.misc
@@ -0,0 +1 @@
+rest/ is now ported to Python 3.
diff --git a/changelog.d/3824.bugfix b/changelog.d/3824.bugfix
new file mode 100644
index 0000000000..99f199dcc6
--- /dev/null
+++ b/changelog.d/3824.bugfix
@@ -0,0 +1 @@
+Fix jwt import check
\ No newline at end of file
diff --git a/changelog.d/3826.misc b/changelog.d/3826.misc
new file mode 100644
index 0000000000..a4d9a012f9
--- /dev/null
+++ b/changelog.d/3826.misc
@@ -0,0 +1 @@
+add some logging for the keyring queue
diff --git a/changelog.d/3827.misc b/changelog.d/3827.misc
new file mode 100644
index 0000000000..bc294706cf
--- /dev/null
+++ b/changelog.d/3827.misc
@@ -0,0 +1 @@
+speed up lazy loading by 2-3x
diff --git a/changelog.d/3834.misc b/changelog.d/3834.misc
new file mode 100644
index 0000000000..8902f8fba7
--- /dev/null
+++ b/changelog.d/3834.misc
@@ -0,0 +1 @@
+Improved Dockerfile to remove build requirements after building reducing the image size.
diff --git a/changelog.d/3835.bugfix b/changelog.d/3835.bugfix
new file mode 100644
index 0000000000..00dbcbc8dc
--- /dev/null
+++ b/changelog.d/3835.bugfix
@@ -0,0 +1 @@
+fix VOIP crashes under Python 3 (#3821)
diff --git a/changelog.d/3840.misc b/changelog.d/3840.misc
new file mode 100644
index 0000000000..b9585ae9be
--- /dev/null
+++ b/changelog.d/3840.misc
@@ -0,0 +1 @@
+Disable lazy loading for incremental syncs for now
diff --git a/changelog.d/3841.bugfix b/changelog.d/3841.bugfix
new file mode 100644
index 0000000000..2a48a7dd66
--- /dev/null
+++ b/changelog.d/3841.bugfix
@@ -0,0 +1 @@
+Fix manhole so that it works with latest openssh clients
diff --git a/changelog.d/3845.bugfix b/changelog.d/3845.bugfix
new file mode 100644
index 0000000000..5b7e8f1934
--- /dev/null
+++ b/changelog.d/3845.bugfix
@@ -0,0 +1 @@
+Fix outbound requests occasionally wedging, which can result in federation breaking between servers.
diff --git a/changelog.d/3846.feature b/changelog.d/3846.feature
new file mode 100644
index 0000000000..453c11d3f8
--- /dev/null
+++ b/changelog.d/3846.feature
@@ -0,0 +1 @@
+Add synapse_admin_mau:registered_reserved_users metric to expose number of real reaserved users 
diff --git a/changelog.d/3847.misc b/changelog.d/3847.misc
new file mode 100644
index 0000000000..bf8b5afea4
--- /dev/null
+++ b/changelog.d/3847.misc
@@ -0,0 +1 @@
+federation/ is now ported to Python 3.
\ No newline at end of file
diff --git a/changelog.d/3851.bugfix b/changelog.d/3851.bugfix
new file mode 100644
index 0000000000..b53a9efe7b
--- /dev/null
+++ b/changelog.d/3851.bugfix
@@ -0,0 +1 @@
+Show heroes if room name/canonical alias has been deleted
diff --git a/changelog.d/3853.misc b/changelog.d/3853.misc
new file mode 100644
index 0000000000..db45d4983d
--- /dev/null
+++ b/changelog.d/3853.misc
@@ -0,0 +1 @@
+Log when we retry outbound requests
diff --git a/changelog.d/3855.misc b/changelog.d/3855.misc
new file mode 100644
index 0000000000..a25bb020ba
--- /dev/null
+++ b/changelog.d/3855.misc
@@ -0,0 +1 @@
+Removed some excess logging messages.
\ No newline at end of file
diff --git a/changelog.d/3856.misc b/changelog.d/3856.misc
new file mode 100644
index 0000000000..36c311eb3d
--- /dev/null
+++ b/changelog.d/3856.misc
@@ -0,0 +1 @@
+Speed up purge history for rooms that have been previously purged
diff --git a/changelog.d/3857.misc b/changelog.d/3857.misc
new file mode 100644
index 0000000000..e128d193d9
--- /dev/null
+++ b/changelog.d/3857.misc
@@ -0,0 +1 @@
+Refactor some HTTP timeout code.
\ No newline at end of file
diff --git a/changelog.d/3858.misc b/changelog.d/3858.misc
new file mode 100644
index 0000000000..4644db5330
--- /dev/null
+++ b/changelog.d/3858.misc
@@ -0,0 +1 @@
+Fix running merged builds on CircleCI
\ No newline at end of file
diff --git a/changelog.d/3859.bugfix b/changelog.d/3859.bugfix
new file mode 100644
index 0000000000..ec5b172464
--- /dev/null
+++ b/changelog.d/3859.bugfix
@@ -0,0 +1 @@
+Fix handling of redacted events from federation
diff --git a/changelog.d/3860.misc b/changelog.d/3860.misc
new file mode 100644
index 0000000000..364239d3e3
--- /dev/null
+++ b/changelog.d/3860.misc
@@ -0,0 +1 @@
+Fix typo in replication stream exception.
diff --git a/changelog.d/3871.misc b/changelog.d/3871.misc
new file mode 100644
index 0000000000..dd9510ceb6
--- /dev/null
+++ b/changelog.d/3871.misc
@@ -0,0 +1 @@
+Add in flight real time metrics for Measure blocks
diff --git a/changelog.d/3872.misc b/changelog.d/3872.misc
new file mode 100644
index 0000000000..b450c506d8
--- /dev/null
+++ b/changelog.d/3872.misc
@@ -0,0 +1 @@
+Disable buffering and automatic retrying in treq requests to prevent timeouts.
\ No newline at end of file
diff --git a/changelog.d/3873.misc b/changelog.d/3873.misc
new file mode 100644
index 0000000000..8104b5c085
--- /dev/null
+++ b/changelog.d/3873.misc
@@ -0,0 +1,2 @@
+Remove documentation regarding installation on Cygwin, the use of WSL is 
+recommended instead.
diff --git a/changelog.d/3874.bugfix b/changelog.d/3874.bugfix
new file mode 100644
index 0000000000..e69de29bb2
--- /dev/null
+++ b/changelog.d/3874.bugfix
diff --git a/changelog.d/3875.bugfix b/changelog.d/3875.bugfix
new file mode 100644
index 0000000000..2d2147dd4b
--- /dev/null
+++ b/changelog.d/3875.bugfix
@@ -0,0 +1 @@
+Mitigate outbound federation randomly becoming wedged
diff --git a/changelog.d/3877.misc b/changelog.d/3877.misc
new file mode 100644
index 0000000000..a80fec4bd8
--- /dev/null
+++ b/changelog.d/3877.misc
@@ -0,0 +1 @@
+mention jemalloc in the README
diff --git a/changelog.d/3879.bugfix b/changelog.d/3879.bugfix
new file mode 100644
index 0000000000..82cb2a8ebc
--- /dev/null
+++ b/changelog.d/3879.bugfix
@@ -0,0 +1 @@
+Don't ratelimit autojoins
diff --git a/changelog.d/3883.feature b/changelog.d/3883.feature
new file mode 100644
index 0000000000..c11e5c5309
--- /dev/null
+++ b/changelog.d/3883.feature
@@ -0,0 +1 @@
+Adding the ability to change MAX_UPLOAD_SIZE for the docker container variables.
\ No newline at end of file
diff --git a/changelog.d/3888.misc b/changelog.d/3888.misc
new file mode 100644
index 0000000000..a10ede547e
--- /dev/null
+++ b/changelog.d/3888.misc
@@ -0,0 +1 @@
+Remove unmaintained "nuke-room-from-db.sh" script
diff --git a/changelog.d/3889.bugfix b/changelog.d/3889.bugfix
new file mode 100644
index 0000000000..e976425987
--- /dev/null
+++ b/changelog.d/3889.bugfix
@@ -0,0 +1 @@
+Fix 500 error when deleting unknown room alias
diff --git a/changelog.d/3892.bugfix b/changelog.d/3892.bugfix
new file mode 100644
index 0000000000..8b30afab04
--- /dev/null
+++ b/changelog.d/3892.bugfix
@@ -0,0 +1 @@
+Fix some b'abcd' noise in logs and metrics
diff --git a/changelog.d/3894.feature b/changelog.d/3894.feature
new file mode 100644
index 0000000000..1ed0cccdb2
--- /dev/null
+++ b/changelog.d/3894.feature
@@ -0,0 +1 @@
+Report "python_version" in the phone home stats
diff --git a/changelog.d/3895.bugfix b/changelog.d/3895.bugfix
new file mode 100644
index 0000000000..8b30afab04
--- /dev/null
+++ b/changelog.d/3895.bugfix
@@ -0,0 +1 @@
+Fix some b'abcd' noise in logs and metrics
diff --git a/changelog.d/3897.misc b/changelog.d/3897.misc
new file mode 100644
index 0000000000..87e7ac796e
--- /dev/null
+++ b/changelog.d/3897.misc
@@ -0,0 +1 @@
+Fix typo in README, synaspse -> synapse
\ No newline at end of file
diff --git a/changelog.d/3899.bugfix b/changelog.d/3899.bugfix
new file mode 100644
index 0000000000..5120e3a823
--- /dev/null
+++ b/changelog.d/3899.bugfix
@@ -0,0 +1 @@
+When we join a room, always try the server we used for the alias lookup first, to avoid unresponsive and out-of-date servers.
diff --git a/changelog.d/3903.misc b/changelog.d/3903.misc
new file mode 100644
index 0000000000..49b64bf333
--- /dev/null
+++ b/changelog.d/3903.misc
@@ -0,0 +1 @@
+Increase the timeout when filling missing events in federation requests
\ No newline at end of file
diff --git a/changelog.d/3906.misc b/changelog.d/3906.misc
new file mode 100644
index 0000000000..11709186d3
--- /dev/null
+++ b/changelog.d/3906.misc
@@ -0,0 +1 @@
+Improve logging of outbound federation requests
\ No newline at end of file
diff --git a/changelog.d/3907.bugfix b/changelog.d/3907.bugfix
new file mode 100644
index 0000000000..45e010c052
--- /dev/null
+++ b/changelog.d/3907.bugfix
@@ -0,0 +1 @@
+Fix incorrect server-name indication for outgoing federation requests
\ No newline at end of file
diff --git a/changelog.d/3909.misc b/changelog.d/3909.misc
new file mode 100644
index 0000000000..11709186d3
--- /dev/null
+++ b/changelog.d/3909.misc
@@ -0,0 +1 @@
+Improve logging of outbound federation requests
\ No newline at end of file
diff --git a/changelog.d/3910.bugfix b/changelog.d/3910.bugfix
new file mode 100644
index 0000000000..22ec2adc33
--- /dev/null
+++ b/changelog.d/3910.bugfix
@@ -0,0 +1 @@
+Fix bug where things occaisonally were not being timed out correctly.