summary refs log tree commit diff
path: root/contrib/systemd (unfollow)
Commit message (Collapse)AuthorFilesLines
2021-05-27Pass the origin when calculating the spaces summary over GET. (#10079)Patrick Cloke2-1/+2
Fixes a bug due to conflicting PRs which were merged. (One added a new caller to a method, the other added a new parameter to the same method.)
2021-05-25 1.35.0rc1 v1.35.0rc1Erik Johnston39-38/+65
2021-05-24Run complement with Synapse workers manually. (#10039)Patrick Cloke3-7/+27
Adds an option to complement.sh to run Synapse in worker mode (instead of the default monolith mode).
2021-05-24Add missing type hints to synapse.util (#9982)Patrick Cloke9-25/+39
2021-05-24Fix docker image to not log at `/homeserver.log` (#10045)Sergio Miguéns2-1/+4
Fixes #9970 Signed-off-by: Sergio Miguéns Iglesias lonyelon@lony.xyz
2021-05-24Remove tls_fingerprints option (#9280)Jerin J Titus6-95/+1
Signed-off-by: Jerin J Titus <72017981+jerinjtitus@users.noreply.github.com>
2021-05-24Fix --no-daemonize for synctl with workers (#9995)Richard van der Hoff2-70/+33
2021-05-24Fix off-by-one-error in synapse_port_db (#9991)Richard van der Hoff5-9/+10
fixes #9979
2021-05-24Remove `keylen` from `LruCache`. (#9993)Richard van der Hoff11-55/+80
`keylen` seems to be a thing that is frequently incorrectly set, and we don't really need it. The only time it was used was to figure out if we had removed a subtree in `del_multi`, which we can do better by changing `TreeCache.pop` to return a different type (`TreeCacheNode`). Commits should be independently reviewable.
2021-05-24Enable experimental spaces by default. (#10011)Patrick Cloke4-2/+35
The previous spaces_enabled flag now defaults to true and is exposed in the sample config.
2021-05-24Add missing entry to the table of contents of room admin API (#10043)Dirk Klimpel2-0/+2
2021-05-24Fix `get_state_ids_for_event` return type typo to match what the function ↵Eric Eastwood2-1/+2
actually does (#10050) It looks like a typo copy/paste from `get_state_for_event` above.
2021-05-21Don't hammer the database for destination retry timings every ~5mins (#10036)Erik Johnston10-76/+62
2021-05-21Fix /upload 500'ing when presented a very large image (#10029)Michael Telatynski3-0/+12
* Fix /upload 500'ing when presented a very large image Catch DecompressionBombError and re-raise as ThumbnailErrors * Set PIL's MAX_IMAGE_PIXELS to match homeserver.yaml to get it to bomb out quicker, to load less into memory in the case of super large images * Add changelog entry for 10029
2021-05-21Add a test for update_presence (#10033)Andrew Morgan2-1/+47
https://github.com/matrix-org/synapse/issues/9962 uncovered that we accidentally removed all but one of the presence updates that we store in the database when persisting multiple updates. This could cause users' presence state to be stale. The bug was fixed in #10014, and this PR just adds a test that failed on the old code, and was used to initially verify the bug. The test attempts to insert some presence into the database in a batch using `PresenceStore.update_presence`, and then simply pulls it out again.
2021-05-21Remove unused properties from the SpaceSummaryHandler. (#10038)Patrick Cloke2-2/+1
2021-05-21Fixed removal of new presence stream states (#10014)Marek Matys2-9/+10
Fixes: https://github.com/matrix-org/synapse/issues/9962 This is a fix for above problem. I fixed it by swaping the order of insertion of new records and deletion of old ones. This ensures that we don't delete fresh database records as we do deletes before inserts. Signed-off-by: Marek Matys <themarcq@gmail.com>
2021-05-21Add a batching queue implementation. (#10017)Erik Johnston3-0/+323
2021-05-20Add `Keyring.verify_events_for_server` and reduce memory usage (#10018)Erik Johnston3-22/+94
Also add support for giving a callback to generate the JSON object to verify. This should reduce memory usage, as we no longer have the event in memory in dict form (which has a large memory footprint) for extend periods of time.
2021-05-20Use ijson to parse the response to `/send_join`, reducing memory usage. (#9958)Erik Johnston7-58/+227
Instead of parsing the full response to `/send_join` into Python objects (which can be huge for large rooms) and *then* parsing that into events, we instead use ijson to stream parse the response directly into `EventBase` objects.
2021-05-20Allow a user who could join a restricted room to see it in spaces summary. ↵Patrick Cloke4-54/+254
(#9922) This finishes up the experimental implementation of MSC3083 by showing the restricted rooms in the spaces summary (from MSC2946).
2021-05-20Update user_directory.md (#10016)Javier Junquera Sánchez2-1/+2
Signed-off-by: Javier Junquera Sánchez <javier@junquera.io>
2021-05-19Update the contrib grafana dashboard (#10001)Erik Johnston2-644/+3558
2021-05-19Hardened systemd unit files (#9803)Savyasachee Jha3-0/+102
Signed-off-by: Savyasachee Jha savya.jha@hawkradius.com
2021-05-18Refactor checking restricted join rules (#10007)Patrick Cloke4-51/+50
To be more consistent with similar code. The check now automatically raises an AuthError instead of passing back a boolean. It also absorbs some shared logic between callers.
2021-05-18Use a database table to hold the users that should have full presence sent ↵Andrew Morgan11-158/+479
to them, instead of something in-memory (#9823)
2021-05-17Fix the allowed range of valid ordering characters for spaces. (#10002)Patrick Cloke2-2/+3
\x7F was meant to be \0x7E (~) this was originally incorrect in MSC1772.
2021-05-17Clarify comments in the space summary handler. (#9974)Patrick Cloke2-5/+47
2021-05-17Changelog v1.34.0 release-v1.34.0Erik Johnston1-6/+6
2021-05-17Update UPGRADE.rstErik Johnston1-7/+4
2021-05-17 1.34.0Erik Johnston3-3/+16
2021-05-17Split multiplart email sending into a dedicated handler (#9977)Brendan Abolivier5-90/+122
Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
2021-05-14Update postgres docs (#9989)Richard van der Hoff3-103/+98
2021-05-14Run `black` on the scripts (#9981)Richard van der Hoff8-76/+141
Turns out these scripts weren't getting linted.
2021-05-14Minor `@cachedList` enhancements (#9975)Richard van der Hoff6-20/+31
- use a tuple rather than a list for the iterable that is passed into the wrapped function, for performance - test that we can pass an iterable and that keys are correctly deduped.
2021-05-14Remove unnecessary SystemRandom from SQLBaseStore (#9987)Dan Callahan3-3/+3
It's not obvious that instances of SQLBaseStore each need their own instances of random.SystemRandom(); let's just use random directly. Introduced by 52839886d664576831462e033b88e5aba4c019e3 Signed-off-by: Dan Callahan <danc@element.io>
2021-05-14Remove superfluous call to bool() (#9986)Dan Callahan2-1/+2
Our strtobool already returns a bool, so no need to re-cast here Signed-off-by: Dan Callahan <danc@element.io>
2021-05-14Simplify exception handling in is_ascii. (#9985)Dan Callahan2-3/+2
We can get away with just catching UnicodeError here. ⋮ +-- ValueError | +-- UnicodeError | +-- UnicodeDecodeError | +-- UnicodeEncodeError | +-- UnicodeTranslateError ⋮ https://docs.python.org/3/library/exceptions.html#exception-hierarchy Signed-off-by: Dan Callahan <danc@element.io>
2021-05-14Use Python's secrets module instead of random (#9984)Dan Callahan2-8/+12
Functionally identical, but more obviously cryptographically secure. ...Explicit is better than implicit? Avoids needing to know that SystemRandom() implies a CSPRNG, and complies with the big scary red box on the documentation for random: > Warning: > The pseudo-random generators of this module should not be used for > security purposes. For security or cryptographic uses, see the > secrets module. https://docs.python.org/3/library/random.html Signed-off-by: Dan Callahan <danc@element.io>
2021-05-14Support enabling opentracing by user (#9978)Richard van der Hoff5-17/+56
Add a config option which allows enabling opentracing by user id, eg for debugging requests made by a test user.
2021-05-14Update minimum supported version in postgres.md (#9988)Andrew Morgan2-1/+2
2021-05-13Update SSO mapping providers documentation about unique IDs. (#9980)Patrick Cloke2-7/+12
2021-05-12Incorporate changes from review v1.34.0rc1Brendan Abolivier1-2/+6
2021-05-12Refer and link to the upgrade notes rather than to the file nameBrendan Abolivier1-1/+1
2021-05-12 1.34.0rc1Brendan Abolivier36-35/+62
2021-05-12Correctly ratelimit invites when creating a room (#9968)Brendan Abolivier6-12/+157
* Correctly ratelimit invites when creating a room Also allow ratelimiting for more than one action at a time.
2021-05-12Change the format of access tokens away from macaroons (#5588)Richard van der Hoff9-103/+78
2021-05-12Run cache_joined_hosts_for_event in background (#9951)Erik Johnston2-5/+41
2021-05-11Tests for to-device messages (#9965)Richard van der Hoff2-0/+202
2021-05-11Sort child events according to MSC1772 for the spaces summary API. (#9954)Patrick Cloke3-2/+151
This should help ensure that equivalent results are achieved between homeservers querying for the summary of a space. This implements modified MSC1772 rules, according to MSC2946. The different is that the origin_server_ts of the m.room.create event is not used as a tie-breaker since this might not be known if the homeserver is not part of the room.
2021-05-11Support fetching the spaces summary via GET over federation. (#9947)Patrick Cloke4-0/+29
Per changes in MSC2946, the C-S and S-S APIs for spaces summary should use GET requests. Until this is stable, the POST endpoints still exist. This does not switch federation requests to use the GET version yet since it is newly added and already deployed servers might not support it. When switching to the stable endpoint we should switch to GET requests.
2021-05-11Send the `m.room.create` stripped event with invites (support MSC1772). (#9966)Patrick Cloke5-5/+34
MSC1772 specifies the m.room.create event should be sent as part of the invite_state. This was done optionally behind an experimental flag, but is now done by default due to MSC1772 being approved.
2021-05-11Use link to advisory rather than to the CVE repo v1.33.2 release-v1.33.2Brendan Abolivier1-1/+1
2021-05-11Add config option to hide device names over federation (#9945)Aaron Raimist4-1/+20
Now that cross signing exists there is much less of a need for other people to look at devices and verify them individually. This PR adds a config option to allow you to prevent device display names from being shared with other servers. Signed-off-by: Aaron Raimist <aaron@raim.ist>
2021-05-11 1.33.2Brendan Abolivier4-2/+23
2021-05-11Add debug logging for issue #9533 (#9959)Richard van der Hoff6-2/+42
Hopefully this will help us track down where to-device messages are getting lost/delayed.
2021-05-11Fix `m.room_key_request` to-device messages (#9961)Richard van der Hoff4-26/+32
fixes #9960
2021-05-11Merge pull request from GHSA-x345-32rc-8h85Richard van der Hoff6-68/+296
* tests for push rule pattern matching * tests for acl pattern matching * factor out common `re.escape` * Factor out common re.compile * Factor out common anchoring code * add word_boundary support to `glob_to_regex` * Use `glob_to_regex` in push rule evaluator NB that this drops support for character classes. I don't think anyone ever used them. * Improve efficiency of globs with multiple wildcards The idea here is that we compress multiple `*` globs into a single `.*`. We also need to consider `?`, since `*?*` is as hard to implement efficiently as `**`. * add assertion on regex pattern * Fix mypy * Simplify glob_to_regex * Inline the glob_to_regex helper function Signed-off-by: Dan Callahan <danc@element.io> * Moar comments Signed-off-by: Dan Callahan <danc@element.io> Co-authored-by: Dan Callahan <danc@element.io>
2021-05-10Improve performance of backfilling in large rooms. (#9935)Erik Johnston2-69/+55
We were pulling the full auth chain for the room out of the DB each time we backfilled, which can be *huge* for large rooms and is totally unnecessary.
2021-05-10Fix port_db on empty db (#9930)Richard van der Hoff6-54/+69
... and test it.
2021-05-07Always cache 'event_to_prev_state_group' (#9950)Erik Johnston2-6/+8
Fixes regression in send PDU times introduced in #9905.
2021-05-07Fix make_full_schema to create the db with the right options and user (#9931)Richard van der Hoff2-9/+11
2021-05-07Unpin attrs dep after new version has been released (#9946)Erik Johnston2-1/+2
c.f. #9936
2021-05-07Reorganise the database schema directories (#9932)Richard van der Hoff284-47/+81
The hope here is that by moving all the schema files into synapse/storage/schema, it gets a bit easier for newcomers to navigate. It certainly got easier for me to write a helpful README. There's more to do on that front, but I'll follow up with other PRs for that.
2021-05-06Export jemalloc stats to prometheus when used (#9882)Erik Johnston4-0/+200
2021-05-06Revert "Leave out optional keys from /sync (#9919)" (#9940)Erik Johnston4-51/+50
This reverts commit e9eb3549d32a6f93d07de8dbd5e1ebe54c8d8278.
2021-05-06 1.33.1 v1.33.1 release-v1.33.1Erik Johnston4-2/+16
2021-05-06Pin attrs to <21.1.0 (#9937)Erik Johnston2-1/+3
Fixes #9936
2021-05-05Follow-up to #9915 to correct the identifier for room types.Patrick Cloke1-1/+1
2021-05-05Include the time of the create event in Spaces Summary. (#9928)Patrick Cloke2-0/+2
This is an update based on changes to MSC2946. The origin_server_ts of the m.room.create event is copied into the creation_ts field for each room returned from the spaces summary.
2021-05-05Increase perf of handling presence when joining large rooms. (#9916)Erik Johnston3-82/+87
2021-05-05Support stable MSC1772 spaces identifiers. (#9915)Patrick Cloke3-2/+10
Support both the unstable and stable identifiers. A future release will disable the unstable identifiers.
2021-05-05Optionally track memory usage of each LruCache (#9881)Erik Johnston8-1/+97
This will double count slightly in the presence of interned strings. It's off by default as it can consume a lot of resources.
2021-05-05Limit how often GC happens by time. (#9902)Erik Johnston6-3/+63
Synapse can be quite memory intensive, and unless care is taken to tune the GC thresholds it can end up thrashing, causing noticable performance problems for large servers. We fix this by limiting how often we GC a given generation, regardless of current counts/thresholds. This does not help with the reverse problem where the thresholds are set too high, but that should only happen in situations where they've been manually configured. Adds a `gc_min_seconds_between` config option to override the defaults. Fixes #9890.
2021-05-05Don't set the external cache if its been done recently (#9905)Erik Johnston3-5/+34
2021-05-05Use get_current_users_in_room from store and not StateHandler (#9910)Erik Johnston12-17/+26
2021-05-05Docker healthcheck timings - add startup delay and changed interval (#9913)Christopher May-Townsend3-4/+16
* Add healthcheck startup delay by 5secs and reduced interval check to 15s to reduce waiting time for docker aware edge routers bringing an instance online
2021-05-05Leave out optional keys from /sync (#9919)DeepBlueV7.X4-50/+51
This leaves out all optional keys from /sync. This should be fine for all clients tested against conduit already, but it may break some clients, as such we should check, that at least most of them don't break horribly and maybe back out some of the individual changes. (We can probably always leave out groups for example, while the others may cause more issues.) Signed-off-by: Nicolas Werner <nicolas.werner@hotmail.de>
2021-05-05 1.33.0 v1.33.0 release-v1.33.0Brendan Abolivier4-2/+16
2021-05-04Time external cache response time (#9904)Erik Johnston2-10/+27
2021-05-04Add port parameter to the sample config for psycopg2 args (#9911)Andrew Morgan3-0/+3
Adds the `port` option with the default value to the sample config file.
2021-04-30Build Debian packages for Ubuntu 21.04 Hirsute (#9909)Dan Callahan2-3/+5
Signed-off-by: Dan Callahan <danc@element.io>
2021-04-29typo in changelog v1.33.0rc2Andrew Morgan1-1/+1
2021-04-29Add missing type hints to handlers and fix a Spam Checker type hint. (#9896)Patrick Cloke8-54/+82
The user_may_create_room_alias method on spam checkers declared the room_alias parameter as a str when in reality it is passed a RoomAlias object.
2021-04-291.33.0rc2Andrew Morgan3-2/+10
2021-04-29Delete room endpoint (#9889)ThibF5-64/+128
Support the delete of a room through DELETE request and mark previous request as deprecated through documentation. Signed-off-by: Thibault Ferrante <thibault.ferrante@pm.me>
2021-04-28Fix tight loop handling presence replication. (#9900)Erik Johnston3-1/+46
Only affects workers. Introduced in #9819. Fixes #9899.
2021-04-28Use the parent's logging context name for runWithConnection. (#9895)Patrick Cloke2-1/+4
This fixes a regression where the logging context for runWithConnection was reported as runWithConnection instead of the connection name, e.g. "POST-XYZ".
2021-04-28Reword account validity template change to sound less like a bugfix v1.33.0rc1Andrew Morgan1-1/+1
2021-04-28Reduce memory footprint of caches (#9886)Erik Johnston2-18/+60
2021-04-281.33.0rc1Andrew Morgan36-35/+54
2021-04-28Revert "Experimental Federation Speedup (#9702)"Andrew Morgan5-138/+93
This reverts commit 05e8c70c059f8ebb066e029bc3aa3e0cefef1019.
2021-04-28Add type hints to presence handler (#9885)Erik Johnston2-70/+90
2021-04-27Remove various bits of compatibility code for Python <3.6 (#9879)Andrew Morgan16-98/+29
I went through and removed a bunch of cruft that was lying around for compatibility with old Python versions. This PR also will now prevent Synapse from starting unless you're running Python 3.6+.
2021-04-27Pass errors back to the client when trying multiple federation destinations. ↵Patrick Cloke2-58/+61
(#9868) This ensures that something like an auth error (403) will be returned to the requester instead of attempting to try more servers, which will likely result in the same error, and then passing back a generic 400 error.
2021-04-27Use current state table for `presence.get_interested_remotes` (#9887)Erik Johnston2-7/+3
This should be a lot quicker than asking the state handler.
2021-04-23Improved validation for received requests (#9817)Richard van der Hoff15-31/+174
* Simplify `start_listening` callpath * Correctly check the size of uploaded files
2021-04-23Kill off `_PushHTTPChannel`. (#9878)Richard van der Hoff3-121/+20
First of all, a fixup to `FakeChannel` which is needed to make it work with the default HTTP channel implementation. Secondly, it looks like we no longer need `_PushHTTPChannel`, because as of #8013, the producer that gets attached to the `HTTPChannel` is now an `IPushProducer`. This is good, because it means we can remove a whole load of test-specific boilerplate which causes variation between tests and production.
2021-04-23Allow OIDC cookies to work on non-root public baseurls (#9726)Andrew Morgan3-9/+22
Applied a (slightly modified) patch from https://github.com/matrix-org/synapse/issues/9574. As far as I understand this would allow the cookie set during the OIDC flow to work on deployments using public baseurls that do not sit at the URL path root.
2021-04-23pass a reactor into SynapseSite (#9874)Richard van der Hoff7-24/+43
2021-04-23Add type hints to auth and auth_blocking. (#9876)Patrick Cloke4-44/+48
2021-04-23Make DomainSpecificString an attrs class (#9875)Erik Johnston4-8/+24
2021-04-23Remove room and user invite ratelimits in default unit test config (#9871)Andrew Morgan2-0/+5
2021-04-23Split presence out of master (#9820)Erik Johnston17-245/+245
2021-04-23Check for space membership during a remote join of a restricted room (#9814)Patrick Cloke6-68/+131
When receiving a /send_join request for a room with join rules set to 'restricted', check if the user is a member of the spaces defined in the 'allow' key of the join rules. This only applies to an experimental room version, as defined in MSC3083.
2021-04-23Only store data in caches, not "smart" objects (#9845)Erik Johnston3-141/+182
2021-04-23Limit the size of HTTP responses read over federation. (#9833)Richard van der Hoff4-8/+110
2021-04-22Handle all new rate limits in demo scripts (#9858)manuroe2-12/+43
2021-04-22Limit length of accepted email addresses (#9855)Erik Johnston6-7/+100
2021-04-22Clear the resync bit after resyncing device lists (#9867)Richard van der Hoff3-10/+17
Fixes #9866.
2021-04-22Remove `synapse.types.Collection` (#9856)Richard van der Hoff26-62/+77
This is no longer required, since we have dropped support for Python 3.5.
2021-04-22A regression can't be introduced twice v1.32.2 release-v1.32.2Andrew Morgan1-2/+2
2021-04-22Note regression was in 1.32.0 and 1.32.1Andrew Morgan1-2/+2
2021-04-21Update dates in changelogsAndrew Morgan2-2/+2
2021-04-211.32.2Andrew Morgan4-2/+18
2021-04-21Note LoggingContext signature change incompatibility in 1.32.0 (#9859) release-v1.32.1Andrew Morgan2-9/+16
1.32.0 also introduced an incompatibility with Synapse modules that make use of `synapse.logging.context.LoggingContext`, such as [synapse-s3-storage-provider](https://github.com/matrix-org/synapse-s3-storage-provider). This PR adds a note to the 1.32.0 changelog and upgrade notes about it.
2021-04-21Make LoggingContext's name optional (#9857)Richard van der Hoff2-3/+13
Fixes https://github.com/matrix-org/synapse-s3-storage-provider/issues/55
2021-04-21Clarify 1.32.0/1 changelog and upgrade notes v1.32.1Andrew Morgan2-8/+7
2021-04-21Add regression notes to CHANGES.md; fix link in 1.32.0 changelogAndrew Morgan1-2/+6
2021-04-21Add link to fixing prometheus to 1.32.0 upgrade notes; 1.32.1 has a fixAndrew Morgan2-2/+6
2021-04-21Fix typo in link to regression in 1.32.0 upgrade notesAndrew Morgan1-1/+1
2021-04-211.32.1Andrew Morgan4-2/+16
2021-04-21Mention Prometheus metrics regression in v1.32.0Andrew Morgan2-0/+15
2021-04-21Stop BackgroundProcessLoggingContext making new prometheus timeseries (#9854)Richard van der Hoff3-5/+18
This undoes part of b076bc276e881b262048307b6a226061d96c4a8d.
2021-04-20Rename handler and config modules which end in handler/config. (#9816)Patrick Cloke17-35/+43
2021-04-20Mention Prometheus metrics regression in v1.32.0 release-v1.32.0Andrew Morgan2-0/+15
2021-04-20Further tweaking on gpg signing key noticeAndrew Morgan1-2/+5
2021-04-20Add note about expired Debian gpg signing keys to CHANGES.mdAndrew Morgan1-0/+6
2021-04-20Update v1.32.0 changelog. It's m.login.application_service, not plural v1.32.0Andrew Morgan1-1/+1
2021-04-20 1.32.0Andrew Morgan4-6/+18
2021-04-20Add Application Service registration type requirement + py35, pg95 ↵Andrew Morgan1-0/+18
deprecation notices to v1.32.0 upgrade notes (#9849) Fixes https://github.com/matrix-org/synapse/issues/9846. Adds important removal information from the top of https://github.com/matrix-org/synapse/releases/tag/v1.32.0rc1 into UPGRADE.rst.
2021-04-20Always use the name as the log ID. (#9829)Patrick Cloke8-34/+26
As far as I can tell our logging contexts are meant to log the request ID, or sometimes the request ID followed by a suffix (this is generally stored in the name field of LoggingContext). There's also code to log the name@memory location, but I'm not sure this is ever used. This simplifies the code paths to require every logging context to have a name and use that in logging. For sub-contexts (created via nested_logging_contexts, defer_to_threadpool, Measure) we use the current context's str (which becomes their name or the string "sentinel") and then potentially modify that (e.g. add a suffix).
2021-04-20Add presence federation stream (#9819)Erik Johnston6-31/+426
2021-04-20Fix bug where we sent remote presence states to remote servers (#9850)Erik Johnston3-3/+13
2021-04-20Fix (final) Bugbear violations (#9838)Jonathan de Jong23-49/+46
2021-04-19Port "Allow users to click account renewal links multiple times without ↵Andrew Morgan18-263/+496
hitting an 'Invalid Token' page #74" from synapse-dinsic (#9832) This attempts to be a direct port of https://github.com/matrix-org/synapse-dinsic/pull/74 to mainline. There was some fiddling required to deal with the changes that have been made to mainline since (mainly dealing with the split of `RegistrationWorkerStore` from `RegistrationStore`, and the changes made to `self.make_request` in test code).
2021-04-19Sanity check identity server passed to bind/unbind. (#9802)Denis Kasak3-3/+59
Signed-off-by: Denis Kasak <dkasak@termina.org.uk>
2021-04-19Don't send normal presence updates over federation replication stream (#9828)Erik Johnston5-183/+75
2021-04-16User directory: use calculated room membership state instead (#9821)Andrew Morgan3-7/+36
Fixes: #9797. Should help reduce CPU usage on the user directory, especially when memberships change in rooms with lots of state history.
2021-04-16Small speed up joining large remote rooms (#9825)Erik Johnston2-21/+34
There are a couple of points in `persist_events` where we are doing a query per event in series, which we can replace.
2021-04-14remove `HomeServer.get_config` (#9815)Richard van der Hoff9-18/+16
Every single time I want to access the config object, I have to remember whether or not we use `get_config`. Let's just get rid of it.
2021-04-14Separate creating an event context from persisting it in the federation ↵Patrick Cloke3-67/+118
handler (#9800) This refactoring allows adding logic that uses the event context before persisting it.
2021-04-14Revert "Check for space membership during a remote join of a restricted ↵Patrick Cloke7-238/+131
room. (#9763)" This reverts commit cc51aaaa7adb0ec2235e027b5184ebda9b660ec4. The PR was prematurely merged and not yet approved.
2021-04-14Check for space membership during a remote join of a restricted room. (#9763)Patrick Cloke7-131/+238
When receiving a /send_join request for a room with join rules set to 'restricted', check if the user is a member of the spaces defined in the 'allow' key of the join rules. This only applies to an experimental room version, as defined in MSC3083.
2021-04-14Experimental Federation Speedup (#9702)Jonathan de Jong5-97/+129
This basically speeds up federation by "squeezing" each individual dual database call (to destinations and destination_rooms), which previously happened per every event, into one call for an entire batch (100 max). Signed-off-by: Jonathan de Jong <jonathan@automatia.nl>
2021-04-14Move some replication processing out of generic_worker (#9796)Erik Johnston6-483/+486
Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
2021-04-14More robust handling of the Content-Type header for thumbnail generation (#9788)rkfg3-0/+5
Signed-off-by: Sergey Shpikin <rkfg@rkfg.me>
2021-04-14Remove redundant "coding: utf-8" lines (#9786)Jonathan de Jong651-651/+1
Part of #9744 Removes all redundant `# -*- coding: utf-8 -*-` lines from files, as python 3 automatically reads source code as utf-8 now. `Signed-off-by: Jonathan de Jong <jonathan@automatia.nl>`
2021-04-14Add a dockerfile for running a set of Synapse worker processes (#9162)Andrew Morgan11-6/+867
This PR adds a Dockerfile and some supporting files to the `docker/` directory. The Dockerfile's intention is to spin up a container with: * A Synapse main process. * Any desired worker processes, defined by a `SYNAPSE_WORKERS` environment variable supplied at runtime. * A redis for worker communication. * A nginx for routing traffic. * A supervisord to start all worker processes and monitor them if any go down. Note that **this is not currently intended to be used in production**. If you'd like to use Synapse workers with Docker, instead make use of the official image, with one worker per container. The purpose of this dockerfile is currently to allow testing Synapse in worker mode with the [Complement](https://github.com/matrix-org/complement/) test suite. `configure_workers_and_start.py` is where most of the magic happens in this PR. It reads from environment variables (documented in the file) and creates all necessary config files for the processes. It is the entrypoint of the Dockerfile, and thus is run any time the docker container is spun up, recreating all config files in case you want to use a different set of workers. One can specify which workers they'd like to use by setting the `SYNAPSE_WORKERS` environment variable (as a comma-separated list of arbitrary worker names) or by setting it to `*` for all worker processes. We will be using the latter in CI. Huge thanks to @MatMaul for helping get this all working :tada: This PR is paired with its equivalent on the Complement side: https://github.com/matrix-org/complement/pull/62. Note, for the purpose of testing this PR before it's merged: You'll need to (re)build the base Synapse docker image for everything to work (`matrixdotorg/synapse:latest`). Then build the worker-based docker image on top (`matrixdotorg/synapse:workers`).
2021-04-14Add note to docker docs explaining platform support (#9801)Andrew Morgan2-3/+7
Context is in https://github.com/matrix-org/synapse/issues/9764#issuecomment-818615894. I struggled to find a more official link for this. The problem occurs when using WSL1 instead of WSL2, which some Windows platforms (at least Server 2019) still don't have. Docker have updated their documentation to paint a much happier picture now given WSL2's support. The last sentence here can probably be removed once WSL1 is no longer around... though that will likely not be for a very long time.
2021-04-13Update changelog for v1.32.0 v1.32.0rc1Andrew Morgan1-1/+1
2021-04-13 1.32.0rc1Andrew Morgan37-36/+73
2021-04-13Add release helper script (#9713)Erik Johnston3-0/+252
Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com> Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
2021-04-13Bump black configuration to target py36 (#9781)Dan Callahan11-15/+16
Signed-off-by: Dan Callahan <danc@element.io>
2021-04-13Add an admin API to manage ratelimit for a specific user (#9648)Dirk Klimpel6-6/+573
2021-04-12Drop Python 3.5 from Trove classifier metadata. (#9782)Dan Callahan2-1/+1
* Drop Python 3.5 from Trove classifier metadata. Signed-off-by: Dan Callahan <danc@element.io>
2021-04-12Add option to skip unit tests when building debs (#9793)Dan Callahan4-13/+34
Signed-off-by: Dan Callahan <danc@element.io>
2021-04-12Require AppserviceRegistrationType (#9548)Will Hunt5-23/+60
This change ensures that the appservice registration behaviour follows the spec. We decided to do this for Dendrite, so it made sense to also make a PR for synapse to correct the behaviour.
2021-04-09Use mock from the stdlib. (#9772)Patrick Cloke82-126/+86
2021-04-09Fix duplicate logging of exceptions in transaction processing (#9780)Richard van der Hoff2-7/+4
There's no point logging this twice.
2021-04-09Enable complement tests for MSC2946. (#9771)Patrick Cloke2-1/+2
By providing the additional build tag for `msc2946`.
2021-04-09Proof of concept for GitHub Actions (#9661)Dan Callahan2-0/+323
Signed-off-by: Dan Callahan <danc@element.io>
2021-04-09Remove old admin API `GET /_synapse/admin/v1/users/<user_id>` (#9401)Dirk Klimpel5-27/+16
Related: #8334 Deprecated in: #9429 - Synapse 1.28.0 (2021-02-25) `GET /_synapse/admin/v1/users/<user_id>` has no - unit tests - documentation API in v2 is available (#5925 - 12/2019, v1.7.0). API is misleading. It expects `user_id` and returns a list of all users. Signed-off-by: Dirk Klimpel dirk@klimpel.org
2021-04-08Bugbear: Add Mutable Parameter fixes (#9682)Jonathan de Jong38-113/+224
Part of #9366 Adds in fixes for B006 and B008, both relating to mutable parameter lint errors. Signed-off-by: Jonathan de Jong <jonathan@automatia.nl>
2021-04-08remove unused param on `make_tuple_comparison_clause`Richard van der Hoff5-9/+3
2021-04-08Fix sharded federation sender sometimes using 100% CPU.Erik Johnston2-2/+5
We pull all destinations requiring catchup from the DB in batches. However, if all those destinations get filtered out (due to the federation sender being sharded), then the `last_processed` destination doesn't get updated, and we keep requesting the same set repeatedly.
2021-04-08update test_old_deps scriptRichard van der Hoff1-1/+1
2021-04-08Update tox.ini to remove py35Richard van der Hoff1-8/+6
2021-04-08drop support for stretch and xenialRichard van der Hoff1-2/+0
2021-04-08Drop support for sqlite<3.22 as wellRichard van der Hoff6-92/+14
2021-04-08Require py36 and Postgres 9.6Richard van der Hoff3-3/+4
2021-04-08unpin olddeps build from py36Richard van der Hoff2-5/+5
2021-04-08Fix incompatibility with tox 2.5Richard van der Hoff2-6/+13
Apparently on tox 2.5, `usedevelop` overrides `skip_install`, so we end up trying to install the full dependencies even for the `-old` environment.
2021-04-08Put opencontainers labels to the final image (#9765)Johannes Wienke2-5/+6
They don't make any sense on the intermediate builder image. The final images needs them to be of use for anyone. Signed-off-by: Johannes Wienke <languitar@semipol.de>
2021-04-08MSC3083: Check for space membership during a local join of restricted rooms. ↵Patrick Cloke3-2/+76
(#9735) When joining a room with join rules set to 'restricted', check if the user is a member of the spaces defined in the 'allow' key of the join rules. This only applies to an experimental room version, as defined in MSC3083.
2021-04-08Record more information into structured logs. (#9654)Patrick Cloke10-88/+255
Records additional request information into the structured logs, e.g. the requester, IP address, etc.
2021-04-06Don't report anything from GaugeBucketCollector metrics until data is ↵Andrew Morgan2-3/+14
present (#8926) This PR modifies `GaugeBucketCollector` to only report data once it has been updated, rather than initially reporting a value of 0. Fixes zero values being reported for some metrics on startup until a background job to update the metric's value runs later.
2021-04-06Add a Synapse Module for configuring presence update routing (#9491)Andrew Morgan14-64/+1282
At the moment, if you'd like to share presence between local or remote users, those users must be sharing a room together. This isn't always the most convenient or useful situation though. This PR adds a module to Synapse that will allow deployments to set up extra logic on where presence updates should be routed. The module must implement two methods, `get_users_for_states` and `get_interested_users`. These methods are given presence updates or user IDs and must return information that Synapse will use to grant passing presence updates around. A method is additionally added to `ModuleApi` which allows triggering a set of users to receive the current, online presence information for all users they are considered interested in. This is the equivalent of that user receiving presence information during an initial sync. The goal of this module is to be fairly generic and useful for a variety of applications, with hard requirements being: * Sending state for a specific set or all known users to a defined set of local and remote users. * The ability to trigger an initial sync for specific users, so they receive all current state.
2021-04-06Add type hints to expiring cache. (#9730)Patrick Cloke8-54/+65
2021-04-06Fix reported bugbear: too broad exception assertion (#9753)Andrew Morgan2-2/+4
2021-04-06Remove outdated constraint on remote_media_cache_thumbnails (#9725)Richard van der Hoff3-3/+41
The `remote_media_cache_thumbnails_media_origin_media_id_thumbna_key` constraint is superceded by `remote_media_repository_thumbn_media_origin_id_width_height_met` (which adds `thumbnail_method` to the unique key). PR #7124 made an attempt to remove the old constraint, but got the name wrong, so it didn't work. Here we update the bg update and rerun it. Fixes #8649.
2021-04-06 1.31.0 v1.31.0 release-v1.31.0Erik Johnston6-7/+27
2021-04-06Add deprecation policy doc (#9723)Erik Johnston4-2/+46
Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com> Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
2021-04-06Fix version for bugbear (#9734)Jonathan de Jong2-1/+2
2021-04-06Add type hints to the federation handler and server. (#9743)Patrick Cloke4-95/+97
2021-04-06Convert storage test cases to HomeserverTestCase. (#9736)Patrick Cloke11-499/+265
2021-04-05Update mypy configuration: `no_implicit_optional = True` (#9742)Jonathan de Jong10-11/+21
2021-04-02Fix version for bugbear (#9734)Jonathan de Jong2-1/+2
2021-04-01Improve tracing for to device messages (#9686)Erik Johnston7-19/+102
2021-04-01Add `order_by` to list user admin API (#9691)Dirk Klimpel6-31/+248
2021-03-31Add an experimental room version to support restricted join rules. (#9717)Patrick Cloke6-11/+297
Per MSC3083.
2021-03-31Revert "Use 'dmypy run' in lint.sh instead of 'mypy' (#9701)" (#9720)Patrick Cloke2-1/+2
2021-03-31Make sample config allowed_local_3pids regex stricter. (#9719)Denis Kasak3-4/+5
The regex should be terminated so that subdomain matches of another domain are not accepted. Just ensuring that someone doesn't shoot themselves in the foot by copying our example. Signed-off-by: Denis Kasak <dkasak@termina.org.uk>
2021-03-31Deprecate imp (#9718)Cristina2-3/+9
Fixes #9642. Signed-off-by: Cristina Muñoz <hi@xmunoz.com>
2021-03-31Rewrite complement.sh (#9685)Andrew Morgan2-11/+39
This PR rewrites the original complement.sh script with a number of improvements: * We can now use a local checkout of Complement (configurable with `COMPLEMENT_DIR`), though the default behaviour still downloads the master branch. * You can now specify a regex of test names to run, or just run all tests. * We now use the Synapse test blacklist tag (so all tests will pass).
2021-03-30Include m.room.create in invite_room_state for Spaces (#9710)Richard van der Hoff2-0/+5
2021-03-30Replace `room_invite_state_types` with `room_prejoin_state` (#9700)Richard van der Hoff8-43/+144
`room_invite_state_types` was inconvenient as a configuration setting, because anyone that ever set it would not receive any new types that were added to the defaults. Here, we deprecate the old setting, and replace it with a couple of new settings under `room_prejoin_state`.
2021-03-30Make RateLimiter class check for ratelimit overrides (#9711)Erik Johnston16-154/+241
This should fix a class of bug where we forget to check if e.g. the appservice shouldn't be ratelimited. We also check the `ratelimit_override` table to check if the user has ratelimiting disabled. That table is really only meant to override the event sender ratelimiting, so we don't use any values from it (as they might not make sense for different rate limits), but we do infer that if ratelimiting is disabled for the user we should disabled all ratelimits. Fixes #9663
2021-03-30Update changelog v1.31.0rc1Erik Johnston1-3/+7
2021-03-30 1.31.0rc1Erik Johnston45-44/+65
2021-03-30Use 'dmypy run' in lint.sh instead of 'mypy' (#9701)Andrew Morgan2-1/+2
For it's obvious performance benefits. `dmypy` support landed in #9692.
2021-03-29Add type hints to DictionaryCache and TTLCache. (#9442)Patrick Cloke7-67/+96
2021-03-29Clarify that register_new_matrix_user is present also when installed via ↵blakehawkins2-8/+19
non-pip package (#9074) Signed-off-by: blakehawkins blake.hawkins.11@gmail.com
2021-03-29Add type hints for the federation sender. (#9681)Patrick Cloke7-59/+177
Includes an abstract base class which both the FederationSender and the FederationRemoteSendQueue must implement.
2021-03-29Update the OIDC sample config (#9695)Richard van der Hoff3-62/+7
I've reiterated the advice about using `oidc` to migrate, since I've seen a few people caught by this. I've also removed a couple of the examples as they are duplicating the OIDC documentation, and I think they might be leading people astray.
2021-03-29Fix CI by ignore type for None module import (#9709)Andrew Morgan2-1/+2
2021-03-29Fix `re.Pattern` mypy error on 3.6 (#9703)Jonathan de Jong2-2/+3
2021-03-29Fix the suggested pip incantation for cryptography (#9699)Richard van der Hoff2-2/+16
If you have the wrong version of `cryptography` installed, synapse suggests: ``` To install run: pip install --upgrade --force 'cryptography>=3.4.7;python_version>='3.6'' ``` However, the use of ' inside '...' doesn't work, so when you run this, you get an error.
2021-03-26Make pip install faster in Docker build for Complement testing (#9610)Eric Eastwood2-43/+42
Make pip install faster in Docker build for [Complement](https://github.com/matrix-org/complement) testing. If files have changed in a `COPY` command, Docker will invalidate all of the layers below. So I changed the order of operations to install all dependencies before we `COPY synapse /synapse/synapse/`. This allows Docker to use our cached layer of dependencies even when we change the source of Synapse and speed up builds dramatically! `53.5s` -> `3.7s` builds 🤘 As an alternative, I did try using BuildKit caches but this still took 30 seconds overall on that step. 15 seconds to gather the dependencies from the cache and another 15 seconds to `Installing collected packages`. Fix https://github.com/matrix-org/synapse/issues/9364
2021-03-26Suppress CryptographyDeprecationWarning (#9698)Richard van der Hoff4-14/+26
This warning is somewhat confusing to users, so let's suppress it
2021-03-26Make it possible to use dmypy (#9692)Erik Johnston16-17/+56
Running `dmypy run` will do a `mypy` check while spinning up a daemon that makes rerunning `dmypy run` a lot faster. `dmypy` doesn't support `follow_imports = silent` and has `local_partial_types` enabled, so this PR enables those options and fixes the issues that were newly raised. Note that `local_partial_types` will be enabled by default in upcoming mypy releases.
2021-03-26Update cahngelog v1.30.1 release-v1.30.1Erik Johnston1-3/+2
2021-03-26Update cahngelogErik Johnston1-2/+14
2021-03-26 1.30.1Erik Johnston5-3/+26
2021-03-26Explicitly upgrade openssl in docker file and enforce new version of ↵Erik Johnston4-20/+28
cryptography (#9697)
2021-03-26Preserve host in example apache config (#9696)Paul Tötterman2-5/+11
Fixes redirect loop Signed-off-by: Paul Tötterman <paul.totterman@iki.fi>
2021-03-25Use interpreter from $PATH instead of absolute paths in various scripts ↵Quentin Gliech17-17/+18
using /usr/bin/env (#9689) On NixOS, `bash` isn't under `/bin/bash` but rather in some directory in `$PATH`. Locally, I've been patching those scripts to make them work. `/usr/bin/env` seems to be the only [portable way](https://unix.stackexchange.com/questions/29608/why-is-it-better-to-use-usr-bin-env-name-instead-of-path-to-name-as-my) to use binaries from the PATH as interpreters. Signed-off-by: Quentin Gliech <quentingliech@gmail.com>
2021-03-25platform specific prerequisites in source install (#9667)Serban Constantin2-7/+9
Make it clearer in the source install step that the platform specific prerequisites must be installed first. Signed-off-by: Serban Constantin <serban.constantin@gmail.com>
2021-03-25Add a storage method for returning all current presence from all users (#9650)Andrew Morgan3-3/+69
Split off from https://github.com/matrix-org/synapse/pull/9491 Adds a storage method for getting the current presence of all local users, optionally excluding those that are offline. This will be used by the code in #9491 when a PresenceRouter module informs Synapse that a given user should have `"ALL"` user presence updates routed to them. Specifically, it is used here: https://github.com/matrix-org/synapse/blob/b588f16e391d664b11f43257eabf70663f0c6d59/synapse/handlers/presence.py#L1131-L1133 Note that there is a `get_all_presence_updates` function just above. That function is intended to walk up the table through stream IDs, and is primarily used by the presence replication stream. I could possibly make use of it in the PresenceRouter-related code, but it would be a bit of a bodge.
2021-03-24Fixed undefined variable error in catchup (#9664)Erik Johnston2-0/+3
Broke in #9640 Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
2021-03-24Fix typo in changelog.Patrick Cloke2-2/+2
2021-03-24Enable addtional flake8-bugbear linting checks. (#9659)Jonathan de Jong8-8/+9
2021-03-24Spaces summary: call out to other servers (#9653)Richard van der Hoff4-27/+324
When we hit an unknown room in the space tree, see if there are other servers that we might be able to poll to get the data. Fixes: #9447
2021-03-24docs: fallback/web endpoint does not appear to be mounted on workers (#9679)Ben Banfield-Zanin2-2/+2
2021-03-24Bump mypy-zope to 0.2.13. (#9678)Patrick Cloke2-1/+2
This fixes an error ("Cannot determine consistent method resolution order (MRO)") when running mypy with a cache.
2021-03-24Add type hints to misc. files. (#9676)Patrick Cloke6-54/+57
2021-03-24Add a type hints for service notices to the HomeServer object. (#9675)Patrick Cloke11-40/+52
2021-03-23Increase default join burst ratelimiting (#9674)Erik Johnston3-6/+7
It's legitimate behaviour to try and join a bunch of rooms at once.
2021-03-23Fix federation stall on concurrent access errors (#9639)Jonathan de Jong2-36/+10
2021-03-23Federation API for Space summary (#9652)Richard van der Hoff3-54/+197
Builds on the work done in #9643 to add a federation API for space summaries. There's a bit of refactoring of the existing client-server code first, to avoid too much duplication.
2021-03-23Import HomeServer from the proper module. (#9665)Patrick Cloke59-58/+59
2021-03-22Allow providing credentials to HTTPS_PROXY (#9657)Andrew Morgan4-34/+184
Addresses https://github.com/matrix-org/synapse-dinsic/issues/70 This PR causes `ProxyAgent` to attempt to extract credentials from an `HTTPS_PROXY` env var. If credentials are found, a `Proxy-Authorization` header ([details](https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Proxy-Authorization)) is sent to the proxy server to authenticate against it. The headers are *not* passed to the remote server. Also added some type hints.
2021-03-22Include opencontainers labels in Docker image (#9612)Johannes Wienke2-0/+6
Cf. https://github.com/opencontainers/image-spec/blob/master/annotations.md#pre-defined-annotation-keys Signed-off-by: Johannes Wienke <languitar@semipol.de>
2021-03-22Fixed code misc. quality issues (#9649)Ankit Dobhal4-3/+4
- Merge 'isinstance' calls. - Remove unnecessary dict call outside of comprehension. - Use 'sys.exit()' calls.
2021-03-22 1.30.0 v1.30.0 release-v1.30.0Erik Johnston3-3/+16
2021-03-19Incorporate reviewBrendan Abolivier2-2/+2
2021-03-19Fix lintBrendan Abolivier2-11/+10
2021-03-19fix mypyRichard van der Hoff1-4/+7
2021-03-18federation_client: handle inline signing_keys in hs.yaml (#9647)Richard van der Hoff2-54/+18
2021-03-18federation_client: stop adding URL prefix (#9645)Richard van der Hoff2-2/+3
2021-03-18Fix type-hints from bad merge.Patrick Cloke1-2/+2
2021-03-18Initial spaces summary API (#9643)Richard van der Hoff6-3/+277
This is very bare-bones for now: federation will come soon, while pagination is descoped for now but will come later.
2021-03-18Move support for MSC3026 behind an experimental flagBrendan Abolivier3-3/+18
2021-03-18Consistently check whether a password may be set for a user. (#9636)Dirk Klimpel5-57/+122
2021-03-18Make federation catchup send last event from any server. (#9640)Erik Johnston4-38/+141
Currently federation catchup will send the last *local* event that we failed to send to the remote. This can cause issues for large rooms where lots of servers have sent events while the remote server was down, as when it comes back up again it'll be flooded with events from various points in the DAG. Instead, let's make it so that all the servers send the most recent events, even if its not theirs. The remote should deduplicate the events, so there shouldn't be much overhead in doing this. Alternatively, the servers could only send local events if they were also extremities and hope that the other server will send the event over, but that is a bit risky.
2021-03-18Implement MSC3026: busy presence stateBrendan Abolivier6-1/+27
2021-03-17Ensure we use a copy of the event content dict before modifying it in ↵Andrew Morgan5-2/+147
serialize_event (#9585) This bug was discovered by DINUM. We were modifying `serialized_event["content"]`, which - if you've got `USE_FROZEN_DICTS` turned on or are [using a third party rules module](https://github.com/matrix-org/synapse/blob/17cd48fe5171d50da4cb59db647b993168e7dfab/synapse/events/third_party_rules.py#L73-L76) - will raise a 500 if you try to a edit a reply to a message. `serialized_event["content"]` could be set to the edit event's content, instead of a copy of it, which is bad as we attempt to modify it. Instead, we also end up modifying the original event's content. DINUM uses a third party rules module, which meant the event's content got frozen and thus an exception was raised. To be clear, the problem is not that the event's content was frozen. In fact doing so helped us uncover the fact we weren't copying event content correctly.
2021-03-17Fix up types for the typing handler. (#9638)Patrick Cloke4-14/+30
By splitting this to two separate methods the callers know what methods they can expect on the handler.
2021-03-17only save remote cross-signing keys if they're different from the current ↵Hubert Chathi2-4/+19
ones (#9634) Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
2021-03-17Fix bad naming of storage function (#9637)Erik Johnston4-4/+7
We had two functions named `get_forward_extremities_for_room` and `get_forward_extremeties_for_room` that took different paramters. We rename one of them to avoid confusion.