summary refs log tree commit diff
path: root/synapse (follow)
Commit message (Collapse)AuthorAgeFilesLines
* Add type hints to state handler. (#10482)Patrick Cloke2021-07-263-19/+28
|
* Update the MSC3083 support to verify if joins are from an authorized server. ↵Patrick Cloke2021-07-2613-79/+540
| | | | (#10254)
* Add type hints to synapse.federation.transport.client. (#10408)Patrick Cloke2021-07-261-201/+298
|
* Merge tag 'v1.39.0rc2' into developErik Johnston2021-07-231-1/+1
|\ | | | | | | | | | | | | | | | | | | | | | | | | | | | | Synapse 1.39.0rc2 (2021-07-22) ============================== Bugfixes -------- - Always include `device_one_time_keys_count` key in `/sync` response to work around a bug in Element Android that broke encryption for new devices. ([\#10457](https://github.com/matrix-org/synapse/issues/10457)) Internal Changes ---------------- - Move docker image build to Github Actions. ([\#10416](https://github.com/matrix-org/synapse/issues/10416))
| * 1.39.0rc2 v1.39.0rc2Erik Johnston2021-07-221-1/+1
| |
| * Merge commit '7da24b975dfb10c277cf963dfddb88f55b1ca598' into release-v1.39Erik Johnston2021-07-221-4/+7
| |\
* | \ Merge branch 'master' into developErik Johnston2021-07-231-4/+7
|\ \ \
| * | | 1.38.1 v1.38.1Erik Johnston2021-07-221-1/+1
| | |/ | |/|
| * | Always send device_one_time_keys_count (#10457)David Baker2021-07-221-4/+7
| | | | | | | | | | | | | | | | | | | | | As per comment Fixes https://github.com/matrix-org/synapse/issues/10456 See also https://github.com/vector-im/element-android/issues/3725
* | | Add `creation_ts` to list users admin API (#10448)Dirk Klimpel2021-07-223-12/+11
| | | | | | | | | Signed-off-by: Dirk Klimpel dirk@klimpel.org
* | | Replace `or_ignore` in `simple_insert` with `simple_upsert` (#10442)Erik Johnston2021-07-225-99/+43
| | | | | | | | | | | | | | | | | | | | | | | | Now that we have `simple_upsert` that should be used in preference to trying to insert and looking for an exception. The main benefit is that we ERROR message don't get written to postgres logs. We also have tidy up the return value on `simple_upsert`, rather than having a tri-state of inserted/not-inserted/unknown.
* | | Fix a handful of type annotations. (#10446)Richard van der Hoff2021-07-223-5/+5
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * switch from `types.CoroutineType` to `typing.Coroutine` these should be identical semantically, and since `defer.ensureDeferred` is defined to take a `typing.Coroutine`, will keep mypy happy * Fix some annotations on inlineCallbacks functions * changelog
* | | Add type hints to additional servlet functions (#10437)Patrick Cloke2021-07-214-64/+173
| | | | | | | | | | | | | | | | | | | | | | | | | | | Improves type hints for: * parse_{boolean,integer} * parse_{boolean,integer}_from_args * parse_json_{value,object}_from_request And fixes any incorrect calls that resulted from unknown types.
* | | Update the notification email subject when invited to a space. (#10426)Patrick Cloke2021-07-212-2/+20
| | |
* | | Add a return type to parse_string. (#10438)Patrick Cloke2021-07-2112-45/+85
| | | | | | | | | | | | And set the required attribute in a few places which will error if a parameter is not provided.
* | | Switch to `chunk` events so we can auth via power_levels (MSC2716) (#10432)Eric Eastwood2021-07-212-6/+17
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Previously, we were using `content.chunk_id` to connect one chunk to another. But these events can be from any `sender` and we can't tell who should be able to send historical events. We know we only want the application service to do it but these events have the sender of a real historical message, not the application service user ID as the sender. Other federated homeservers also have no indicator which senders are an application service on the originating homeserver. So we want to auth all of the MSC2716 events via power_levels and have them be sent by the application service with proper PL levels in the room.
* | | Merge tag 'v1.39.0rc1' into developAndrew Morgan2021-07-202-2/+6
|\ \ \ | | |/ | |/| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Synapse 1.39.0rc1 (2021-07-20) ============================== The Third-Party Event Rules module interface has been deprecated in favour of the generic module interface introduced in Synapse v1.37.0. Support for the old interface is planned to be removed in September 2021. See the [upgrade notes](https://matrix-org.github.io/synapse/latest/upgrade.html#upgrading-to-v1390) for more information. Features -------- - Add the ability to override the account validity feature with a module. ([\#9884](https://github.com/matrix-org/synapse/issues/9884)) - The spaces summary API now returns any joinable rooms, not only rooms which are world-readable. ([\#10298](https://github.com/matrix-org/synapse/issues/10298), [\#10305](https://github.com/matrix-org/synapse/issues/10305)) - Add a new version of the R30 phone-home metric, which removes a false impression of retention given by the old R30 metric. ([\#10332](https://github.com/matrix-org/synapse/issues/10332), [\#10427](https://github.com/matrix-org/synapse/issues/10427)) - Allow providing credentials to `http_proxy`. ([\#10360](https://github.com/matrix-org/synapse/issues/10360)) Bugfixes -------- - Fix error while dropping locks on shutdown. Introduced in v1.38.0. ([\#10433](https://github.com/matrix-org/synapse/issues/10433)) - Add base starting insertion event when no chunk ID is specified in the historical batch send API. ([\#10250](https://github.com/matrix-org/synapse/issues/10250)) - Fix historical batch send endpoint (MSC2716) rejecting batches with messages from multiple senders. ([\#10276](https://github.com/matrix-org/synapse/issues/10276)) - Fix purging rooms that other homeservers are still sending events for. Contributed by @ilmari. ([\#10317](https://github.com/matrix-org/synapse/issues/10317)) - Fix errors during backfill caused by previously purged redaction events. Contributed by Andreas Rammhold (@andir). ([\#10343](https://github.com/matrix-org/synapse/issues/10343)) - Fix the user directory becoming broken (and noisy errors being logged) when knocking and room statistics are in use. ([\#10344](https://github.com/matrix-org/synapse/issues/10344)) - Fix newly added `synapse_federation_server_oldest_inbound_pdu_in_staging` prometheus metric to measure age rather than timestamp. ([\#10355](https://github.com/matrix-org/synapse/issues/10355)) - Fix PostgreSQL sometimes using table scans for queries against `state_groups_state` table, taking a long time and a large amount of IO. ([\#10359](https://github.com/matrix-org/synapse/issues/10359)) - Fix `make_room_admin` failing for users that have left a private room. ([\#10367](https://github.com/matrix-org/synapse/issues/10367)) - Fix a number of logged errors caused by remote servers being down. ([\#10400](https://github.com/matrix-org/synapse/issues/10400), [\#10414](https://github.com/matrix-org/synapse/issues/10414)) - Responses from `/make_{join,leave,knock}` no longer include signatures, which will turn out to be invalid after events are returned to `/send_{join,leave,knock}`. ([\#10404](https://github.com/matrix-org/synapse/issues/10404)) Improved Documentation ---------------------- - Updated installation dependencies for newer macOS versions and ARM Macs. Contributed by Luke Walsh. ([\#9971](https://github.com/matrix-org/synapse/issues/9971)) - Simplify structure of room admin API. ([\#10313](https://github.com/matrix-org/synapse/issues/10313)) - Refresh the logcontext dev documentation. ([\#10353](https://github.com/matrix-org/synapse/issues/10353)), ([\#10337](https://github.com/matrix-org/synapse/issues/10337)) - Add delegation example for caddy in the reverse proxy documentation. Contributed by @moritzdietz. ([\#10368](https://github.com/matrix-org/synapse/issues/10368)) - Fix and clarify some links in `docs` and `contrib`. ([\#10370](https://github.com/matrix-org/synapse/issues/10370)), ([\#10322](https://github.com/matrix-org/synapse/issues/10322)), ([\#10399](https://github.com/matrix-org/synapse/issues/10399)) - Make deprecation notice of the spam checker doc more obvious. ([\#10395](https://github.com/matrix-org/synapse/issues/10395)) - Add instructions on installing Debian packages for release candidates. ([\#10396](https://github.com/matrix-org/synapse/issues/10396)) Deprecations and Removals ------------------------- - Remove functionality associated with the unused `room_stats_historical` and `user_stats_historical` tables. Contributed by @xmunoz. ([\#9721](https://github.com/matrix-org/synapse/issues/9721)) - The third-party event rules module interface is deprecated in favour of the generic module interface introduced in Synapse v1.37.0. See the [upgrade notes](https://matrix-org.github.io/synapse/latest/upgrade.html#upgrading-to-v1390) for more information. ([\#10386](https://github.com/matrix-org/synapse/issues/10386)) Internal Changes ---------------- - Convert `room_depth.min_depth` column to a `BIGINT`. ([\#10289](https://github.com/matrix-org/synapse/issues/10289)) - Add tests to characterise the current behaviour of R30 phone-home metrics. ([\#10315](https://github.com/matrix-org/synapse/issues/10315)) - Rebuild event context and auth when processing specific results from `ThirdPartyEventRules` modules. ([\#10316](https://github.com/matrix-org/synapse/issues/10316)) - Minor change to the code that populates `user_daily_visits`. ([\#10324](https://github.com/matrix-org/synapse/issues/10324)) - Re-enable Sytests that were disabled for the 1.37.1 release. ([\#10345](https://github.com/matrix-org/synapse/issues/10345), [\#10357](https://github.com/matrix-org/synapse/issues/10357)) - Run `pyupgrade` on the codebase. ([\#10347](https://github.com/matrix-org/synapse/issues/10347), [\#10348](https://github.com/matrix-org/synapse/issues/10348)) - Switch `application_services_txns.txn_id` database column to `BIGINT`. ([\#10349](https://github.com/matrix-org/synapse/issues/10349)) - Convert internal type variable syntax to reflect wider ecosystem use. ([\#10350](https://github.com/matrix-org/synapse/issues/10350), [\#10380](https://github.com/matrix-org/synapse/issues/10380), [\#10381](https://github.com/matrix-org/synapse/issues/10381), [\#10382](https://github.com/matrix-org/synapse/issues/10382), [\#10418](https://github.com/matrix-org/synapse/issues/10418)) - Make the Github Actions workflow configuration more efficient. ([\#10383](https://github.com/matrix-org/synapse/issues/10383)) - Add type hints to `get_{domain,localpart}_from_id`. ([\#10385](https://github.com/matrix-org/synapse/issues/10385)) - When building Debian packages for prerelease versions, set the Section accordingly. ([\#10391](https://github.com/matrix-org/synapse/issues/10391)) - Add type hints and comments to event auth code. ([\#10393](https://github.com/matrix-org/synapse/issues/10393)) - Stagger sending of presence update to remote servers, reducing CPU spikes caused by starting many connections to remote servers at once. ([\#10398](https://github.com/matrix-org/synapse/issues/10398)) - Remove unused `events_by_room` code (tech debt). ([\#10421](https://github.com/matrix-org/synapse/issues/10421)) - Add a github actions job which records success of other jobs. ([\#10430](https://github.com/matrix-org/synapse/issues/10430))
| * | Fix dropping locks on shut down (#10433)Erik Johnston2021-07-201-1/+5
| | |
| * | 1.39.0rc1Andrew Morgan2021-07-201-1/+1
| | |
* | | MSC3244 room capabilities implementation (#10283)Michael Telatynski2021-07-203-3/+46
| | |
* | | Revert "Fix dropping locks on shut down"Erik Johnston2021-07-201-5/+1
| | | | | | | | | | | | This reverts commit 83f1ccfcaba76785ab4bd91e3177724e2dbb85ed.
* | | Fix dropping locks on shut downErik Johnston2021-07-201-1/+5
|/ /
* | Port the ThirdPartyEventRules module interface to the new generic interface ↵Brendan Abolivier2021-07-207-70/+220
| | | | | | | | | | (#10386) Port the third-party event rules interface to the generic module interface introduced in v1.37.0
* | Fix exception when failing to get remote room list (#10414)Erik Johnston2021-07-201-1/+5
| |
* | Factorise `get_datastore` calls in phone_stats_home. (#10427)reivilibre2021-07-191-16/+18
| | | | | | Follow-up to #10332.
* | Add a new version of the R30 phone-home metric, which removes a false ↵reivilibre2021-07-192-0/+133
| | | | | | | | | | impression of retention given by the old R30 metric (#10332) Signed-off-by: Olivier Wilkinson (reivilibre) <olivier@librepush.net>
* | [pyupgrade] `synapse/` (#10348)Jonathan de Jong2021-07-1928-102/+85
| | | | | | | | | | | | | | | | | | This PR is tantamount to running ``` pyupgrade --py36-plus --keep-percent-format `find synapse/ -type f -name "*.py"` ``` Part of #9744
* | Remove unused `events_by_room` (#10421)Eric Eastwood2021-07-191-4/+0
| | | | | | | | | | It looks like it was first used and introduced in https://github.com/matrix-org/synapse/commit/5130d80d79fe1f95ce03b8f1cfd4fbf0a32f5ac8#diff-8a4a36a7728107b2ccaff2cb405dbab229a1100fe50653a63d1aa9ac10ae45e8R305 but the But the usage was removed in https://github.com/matrix-org/synapse/commit/4c6a31cd6efa25be4c9f1b357e8f92065fac63eb#diff-8a4a36a7728107b2ccaff2cb405dbab229a1100fe50653a63d1aa9ac10ae45e8
* | One last inline type hint (for the whole repo) (#10418)Jonathan de Jong2021-07-161-1/+1
| |
* | Use inline type hints in `handlers/` and `rest/`. (#10382)Jonathan de Jong2021-07-1642-215/+211
| |
* | Add a module type for account validity (#9884)Brendan Abolivier2021-07-169-137/+395
| | | | | | | | | | This adds an API for third-party plugin modules to implement account validity, so they can provide this feature instead of Synapse. The module implementing the current behaviour for this feature can be found at https://github.com/matrix-org/synapse-email-account-validity. To allow for a smooth transition between the current feature and the new module, hooks have been added to the existing account validity endpoints to allow their behaviours to be overridden by a module.
* | Do not include signatures/hashes in make_{join,leave,knock} responses. (#10404)Patrick Cloke2021-07-162-6/+17
| | | | | | | | These signatures would end up invalid since the joining/leaving/knocking server would modify the response before calling send_{join,leave,knock}.
* | Use inline type hints in `http/federation/`, `storage/` and `util/` (#10381)Jonathan de Jong2021-07-1537-162/+149
| |
* | Reduce likelihood of Postgres table scanning `state_groups_state`. (#10359)Erik Johnston2021-07-151-0/+34
| | | | | | | | | | | | | | | | The postgres statistics collector sometimes massively underestimates the number of distinct state groups are in the `state_groups_state`, which can cause postgres to use table scans for queries for multiple state groups. We fix this by manually setting `n_distinct` on the column.
* | Stagger send presence to remotes (#10398)Erik Johnston2021-07-152-5/+107
| | | | | | | | | | | | This is to help with performance, where trying to connect to thousands of hosts at once can consume a lot of CPU (due to TLS etc). Co-authored-by: Brendan Abolivier <babolivier@matrix.org>
* | Use inline type hints in various other places (in `synapse/`) (#10380)Jonathan de Jong2021-07-1578-336/+328
| |
* | Allow providing credentials to `http_proxy` (#10360)Dirk Klimpel2021-07-151-1/+11
| |
* | Fix a number of logged errors caused by remote servers being down. (#10400)Erik Johnston2021-07-154-19/+65
| |
* | Add type hints and comments to event auth code. (#10393)Patrick Cloke2021-07-141-0/+3
| |
* | Fix messages from multiple senders in historical chunk (MSC2716) (#10276)Eric Eastwood2021-07-132-11/+75
| | | | | | | | | | | | | | Fix messages from multiple senders in historical chunk. This also means that an app service does not need to define `?user_id` when using this endpoint. Follow-up to https://github.com/matrix-org/synapse/pull/9247 Part of MSC2716: https://github.com/matrix-org/matrix-doc/pull/2716
* | Add type hints to get_domain_from_id and get_localpart_from_id. (#10385)Patrick Cloke2021-07-132-26/+74
| |
* | Show all joinable rooms in the spaces summary. (#10298)Patrick Cloke2021-07-132-22/+59
| | | | | | | | | | | | | | | | | | | | Previously only world-readable rooms were shown. This means that rooms which are public, knockable, or invite-only with a pending invitation, are included in a space summary. It also applies the same logic to the experimental room version from MSC3083 -- if a user has access to the proper allowed rooms then it is shown in the spaces summary. This change is made per MSC3173 allowing stripped state of a room to be shown to any potential room joiner.
* | Merge branch 'master' into developRichard van der Hoff2021-07-131-1/+1
|\|
| * 1.38.0Richard van der Hoff2021-07-131-1/+1
| |
| * 1.38.0rc3 v1.38.0rc3Richard van der Hoff2021-07-131-1/+1
| |
* | Bugfix `make_room_admin` fails for users that have left a private room (#10367)Dirk Klimpel2021-07-131-1/+8
| | | | | | Fixes: #10338
* | Fix federation inbound age metric. (#10355)Erik Johnston2021-07-131-1/+3
| | | | | | We should be reporting the age rather than absolute timestamp.
* | Replace `room_depth.min_depth` with a BIGINT (#10289)Richard van der Hoff2021-07-123-7/+185
| | | | | | | | | | while I'm dealing with INTEGERs and BIGINTs, let's replace room_depth.min_depth with a BIGINT.
* | Fix the user directory becoming broken (and noisy errors being logged) when ↵reivilibre2021-07-091-2/+6
| | | | | | | | | | knocking and room statistics are in use. (#10344) Signed-off-by: Olivier Wilkinson (reivilibre) <olivier@librepush.net>
* | Switch `application_services_txns.txn_id` to BIGINT (#10349)Richard van der Hoff2021-07-091-0/+23
| |
* | Merge tag 'v1.38.0rc2' into developErik Johnston2021-07-093-3/+15
|\| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Synapse 1.38.0rc2 (2021-07-09) ============================== Bugfixes -------- - Fix bug where inbound federation in a room could be delayed due to not correctly dropping a lock. Introduced in v1.37.1. ([\#10336](https://github.com/matrix-org/synapse/issues/10336)) Improved Documentation ---------------------- - Update links to documentation in the sample config. Contributed by @dklimpel. ([\#10287](https://github.com/matrix-org/synapse/issues/10287)) - Fix broken links in [INSTALL.md](INSTALL.md). Contributed by @dklimpel. ([\#10331](https://github.com/matrix-org/synapse/issues/10331))
| * 1.38.0rc2Erik Johnston2021-07-091-1/+1
| |
| * Ensure we always drop the federation inbound lock (#10336)Erik Johnston2021-07-092-2/+14
| |
* | Upsert redactions in case they already exists (#10343)Andreas Rammhold2021-07-092-2/+3
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Upsert redactions in case they already exists Occasionally, in combination with retention, redactions aren't deleted from the database whenever they are due for deletion. The server will eventually try to backfill the deleted events and trip over the already existing redaction events. Switching to an UPSERT for those events allows us to recover from there situations. The retention code still needs fixing but that is outside of my current comfort zone on this code base. This is related to #8707 where the error was discussed already. Signed-off-by: Andreas Rammhold <andreas@rammhold.de> * Also purge redactions when purging events Previously redacints where left behind leading to backfilling issues when the server stumbled across the already existing yet to be backfilled redactions. This issues has been discussed in #8707. Signed-off-by: Andreas Rammhold <andreas@rammhold.de>
* | Add base starting insertion event when no chunk ID is provided (MSC2716) ↵Eric Eastwood2021-07-082-23/+97
| | | | | | | | | | | | | | | | | | (#10250) * Add base starting insertion point when no chunk ID is provided This is so we can have the marker event point to this initial insertion event and be able to traverse the events in the first chunk.
* | Remove functionality associated with unused historical stats tables (#9721)Cristina2021-07-085-328/+6
| | | | | | Fixes #9602
* | Minor changes to `user_daily_visits` (#10324)reivilibre2021-07-081-2/+2
| | | | | | | | | | | | | | * Use fake time in tests in _get_start_of_day. * Change the inequality of last_seen in user_daily_visits Co-authored-by: Erik Johnston <erik@matrix.org>
* | Rebuild event auth when rebuilding an event after a call to a ↵Brendan Abolivier2021-07-081-4/+6
| | | | | | | | | | `ThirdPartyEventRules` module (#10316) Because modules might send extra state events when processing an event (e.g. matrix-org/synapse-dinsic#100), and in some cases these extra events might get dropped if we don't recalculate the initial event's auth.
* | Merge remote-tracking branch 'origin/release-v1.38' into developRichard van der Hoff2021-07-0713-24/+26
|\| | | | | | | merge @dklimpel's fixes to the sample config
| * Update links to documentation in sample config (#10287)Dirk Klimpel2021-07-0713-24/+26
| | | | | | Signed-off-by: Dirk Klimpel dirk@klimpel.org
* | Merge branch 'release-v1.38' into developRichard van der Hoff2021-07-072-1/+11
|\|
| * ANALYZE new stream ordering column (#10326)Brendan Abolivier2021-07-071-0/+10
| | | | | | Fixes #10325
| * 1.38.0rc1Brendan Abolivier2021-07-061-1/+1
| |
* | Ignore EDUs for rooms we're not in (#10317)Dagfinn Ilmari Mannsåker2021-07-062-0/+29
|/
* Fix deactivate a user if he does not have a profile (#10252)Dirk Klimpel2021-07-061-4/+4
|
* Add script for getting info about recently registered users (#10290)Erik Johnston2021-07-062-1/+176
|
* Handle old staged inbound events (#10303)Erik Johnston2021-07-062-10/+66
| | | | | | | We might have events in the staging area if the service was restarted while there were unhandled events in the staging area. Fixes #10295
* Add support for evicting cache entries based on last access time. (#10205)Erik Johnston2021-07-055-66/+404
|
* Move methods involving event authentication to EventAuthHandler. (#10268)Patrick Cloke2021-07-019-104/+109
| | | Instead of mixing them with user authentication methods.
* fix ordering of bg update (#10291)Richard van der Hoff2021-07-011-1/+1
| | | | | this was a typo introduced in #10282. We don't want to end up doing the `replace_stream_ordering_column` update after anything that comes up in migration 60/03.
* Add SSO `external_ids` to Query User Account admin API (#10261)Dirk Klimpel2021-07-011-0/+7
| | | Related to #10251
* Add some metrics to staging area (#10284)Erik Johnston2021-07-011-0/+39
|
* Change more stream_ordering columns to BIGINT (#10286)Richard van der Hoff2021-06-301-0/+30
|
* Rebuild other indexes using `stream_ordering` (#10282)Richard van der Hoff2021-06-302-6/+55
| | | | We need to rebuild *all* of the indexes that use the current `stream_ordering` column.
* Merge branch 'master' into developErik Johnston2021-06-301-1/+1
|\
| * 1.37.1Erik Johnston2021-06-301-1/+1
| |
* | Correct type hints for synapse.event_auth. (#10253)Patrick Cloke2021-06-305-38/+50
| |
* | Fix the inbound PDU metric (#10279)Erik Johnston2021-06-305-27/+92
| | | | | | This broke in #10272
* | Merge branch 'release-v1.37' into developRichard van der Hoff2021-06-291-1/+1
|\|
| * 1.37.1rc1 v1.37.1rc1Richard van der Hoff2021-06-291-1/+1
| |
* | Merge branch 'release-v1.37' into developRichard van der Hoff2021-06-298-6/+610
|\|
| * v1.37.1a1Richard van der Hoff2021-06-291-1/+1
| |
| * Merge remote-tracking branch 'origin/erikj/async_federation_base_branch' ↵Richard van der Hoff2021-06-297-5/+609
| |\ | | | | | | | | | into release-v1.37
| | * Handle inbound events from federation asynchronously (#10272)Erik Johnston2021-06-293-5/+234
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Fixes #9490 This will break a couple of SyTest that are expecting failures to be added to the response of a federation /send, which obviously doesn't happen now that things are asynchronous. Two drawbacks: Currently there is no logic to handle any events left in the staging area after restart, and so they'll only be handled on the next incoming event in that room. That can be fixed separately. We now only process one event per room at a time. This can be fixed up further down the line.
| | * Add a distributed lock (#10269)Erik Johnston2021-06-294-0/+375
| | | | | | | | | This adds a simple best effort locking mechanism that works cross workers.
* | | Do not recurse into non-spaces in the spaces summary. (#10256)Patrick Cloke2021-06-292-2/+15
| | | | | | | | | | | | | | | Previously m.child.room events in non-space rooms would be treated as part of the room graph, but this is no longer supported.
* | | Fix `populate_stream_ordering2` background job (#10267)Richard van der Hoff2021-06-291-16/+12
| | | | | | | | | | | | It was possible for us not to find any rows in a batch, and hence conclude that we had finished. Let's not do that.
* | | Migrate stream_ordering to a bigint (#10264)Richard van der Hoff2021-06-293-16/+162
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Move background update names out to a separate class `EventsBackgroundUpdatesStore` gets inherited and we don't really want to further pollute the namespace. * Migrate stream_ordering to a bigint * changelog
* | | Soft-fail spammy events received over federation (#10263)Richard van der Hoff2021-06-291-6/+6
| | |
* | | Merge branch 'master' into developBrendan Abolivier2021-06-291-1/+1
|\| |
| * | 1.37.0Brendan Abolivier2021-06-291-1/+1
| |/
* | Add additional types to the federation transport server. (#10213)Patrick Cloke2021-06-282-118/+520
| |
* | Return errors from `send_join` etc if the event is rejected (#10243)Richard van der Hoff2021-06-241-7/+39
| | | | | | Rather than persisting rejected events via `send_join` and friends, raise a 403 if someone tries to pull a fast one.
* | Improve validation for `send_{join,leave,knock}` (#10225)Richard van der Hoff2021-06-243-181/+129
| | | | | | The idea here is to stop people sending things that aren't joins/leaves/knocks through these endpoints: previously you could send anything you liked through them. I wasn't able to find any security holes from doing so, but it doesn't sound like a good thing.
* | MSC2918 Refresh tokens implementation (#9450)Quentin Gliech2021-06-2410-58/+667
| | | | | | | | | | | | | | | | | | | | This implements refresh tokens, as defined by MSC2918 This MSC has been implemented client side in Hydrogen Web: vector-im/hydrogen-web#235 The basics of the MSC works: requesting refresh tokens on login, having the access tokens expire, and using the refresh token to get a new one. Signed-off-by: Quentin Gliech <quentingliech@gmail.com>
* | Merge tag 'v1.37.0rc1' into developBrendan Abolivier2021-06-242-7/+8
|\| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Synapse 1.37.0rc1 (2021-06-24) ============================== This release deprecates the current spam checker interface. See the [upgrade notes](https://matrix-org.github.io/synapse/develop/upgrade#deprecation-of-the-current-spam-checker-interface) for more information on how to update to the new generic module interface. This release also removes support for fetching and renewing TLS certificates using the ACME v1 protocol, which has been fully decommissioned by Let's Encrypt on June 1st 2021. Admins previously using this feature should use a [reverse proxy](https://matrix-org.github.io/synapse/develop/reverse_proxy.html) to handle TLS termination, or use an external ACME client (such as [certbot](https://certbot.eff.org/)) to retrieve a certificate and key and provide them to Synapse using the `tls_certificate_path` and `tls_private_key_path` configuration settings. Features -------- - Implement "room knocking" as per [MSC2403](https://github.com/matrix-org/matrix-doc/pull/2403). Contributed by @Sorunome and anoa. ([\#6739](https://github.com/matrix-org/synapse/issues/6739), [\#9359](https://github.com/matrix-org/synapse/issues/9359), [\#10167](https://github.com/matrix-org/synapse/issues/10167), [\#10212](https://github.com/matrix-org/synapse/issues/10212), [\#10227](https://github.com/matrix-org/synapse/issues/10227)) - Add experimental support for backfilling history into rooms ([MSC2716](https://github.com/matrix-org/matrix-doc/pull/2716)). ([\#9247](https://github.com/matrix-org/synapse/issues/9247)) - Implement a generic interface for third-party plugin modules. ([\#10062](https://github.com/matrix-org/synapse/issues/10062), [\#10206](https://github.com/matrix-org/synapse/issues/10206)) - Implement config option `sso.update_profile_information` to sync SSO users' profile information with the identity provider each time they login. Currently only displayname is supported. ([\#10108](https://github.com/matrix-org/synapse/issues/10108)) - Ensure that errors during startup are written to the logs and the console. ([\#10191](https://github.com/matrix-org/synapse/issues/10191)) Bugfixes -------- - Fix a bug introduced in Synapse v1.25.0 that prevented the `ip_range_whitelist` configuration option from working for federation and identity servers. Contributed by @mikure. ([\#10115](https://github.com/matrix-org/synapse/issues/10115)) - Remove a broken import line in Synapse's `admin_cmd` worker. Broke in Synapse v1.33.0. ([\#10154](https://github.com/matrix-org/synapse/issues/10154)) - Fix a bug introduced in Synapse v1.21.0 which could cause `/sync` to return immediately with an empty response. ([\#10157](https://github.com/matrix-org/synapse/issues/10157), [\#10158](https://github.com/matrix-org/synapse/issues/10158)) - Fix a minor bug in the response to `/_matrix/client/r0/user/{user}/openid/request_token` causing `expires_in` to be a float instead of an integer. Contributed by @lukaslihotzki. ([\#10175](https://github.com/matrix-org/synapse/issues/10175)) - Always require users to re-authenticate for dangerous operations: deactivating an account, modifying an account password, and adding 3PIDs. ([\#10184](https://github.com/matrix-org/synapse/issues/10184)) - Fix a bug introduced in Synpase v1.7.2 where remote server count metrics collection would be incorrectly delayed on startup. Found by @heftig. ([\#10195](https://github.com/matrix-org/synapse/issues/10195)) - Fix a bug introduced in Synapse v1.35.1 where an `allow` key of a `m.room.join_rules` event could be applied for incorrect room versions and configurations. ([\#10208](https://github.com/matrix-org/synapse/issues/10208)) - Fix performance regression in responding to user key requests over federation. Introduced in Synapse v1.34.0rc1. ([\#10221](https://github.com/matrix-org/synapse/issues/10221)) Improved Documentation ---------------------- - Add a new guide to decoding request logs. ([\#8436](https://github.com/matrix-org/synapse/issues/8436)) - Mention in the sample homeserver config that you may need to configure max upload size in your reverse proxy. Contributed by @aaronraimist. ([\#10122](https://github.com/matrix-org/synapse/issues/10122)) - Fix broken links in documentation. ([\#10180](https://github.com/matrix-org/synapse/issues/10180)) - Deploy a snapshot of the documentation website upon each new Synapse release. ([\#10198](https://github.com/matrix-org/synapse/issues/10198)) Deprecations and Removals ------------------------- - The current spam checker interface is deprecated in favour of a new generic modules system. See the [upgrade notes](https://matrix-org.github.io/synapse/develop/upgrade#deprecation-of-the-current-spam-checker-interface) for more information on how to update to the new system. ([\#10062](https://github.com/matrix-org/synapse/issues/10062), [\#10210](https://github.com/matrix-org/synapse/issues/10210), [\#10238](https://github.com/matrix-org/synapse/issues/10238)) - Stop supporting the unstable spaces prefixes from MSC1772. ([\#10161](https://github.com/matrix-org/synapse/issues/10161)) - Remove Synapse's support for automatically fetching and renewing certificates using the ACME v1 protocol. This protocol has been fully turned off by Let's Encrypt for existing installations on June 1st 2021. Admins previously using this feature should use a [reverse proxy](https://matrix-org.github.io/synapse/develop/reverse_proxy.html) to handle TLS termination, or use an external ACME client (such as [certbot](https://certbot.eff.org/)) to retrieve a certificate and key and provide them to Synapse using the `tls_certificate_path` and `tls_private_key_path` configuration settings. ([\#10194](https://github.com/matrix-org/synapse/issues/10194)) Internal Changes ---------------- - Update the database schema versioning to support gradual migration away from legacy tables. ([\#9933](https://github.com/matrix-org/synapse/issues/9933)) - Add type hints to the federation servlets. ([\#10080](https://github.com/matrix-org/synapse/issues/10080)) - Improve OpenTracing for event persistence. ([\#10134](https://github.com/matrix-org/synapse/issues/10134), [\#10193](https://github.com/matrix-org/synapse/issues/10193)) - Clean up the interface for injecting OpenTracing over HTTP. ([\#10143](https://github.com/matrix-org/synapse/issues/10143)) - Limit the number of in-flight `/keys/query` requests from a single device. ([\#10144](https://github.com/matrix-org/synapse/issues/10144)) - Refactor EventPersistenceQueue. ([\#10145](https://github.com/matrix-org/synapse/issues/10145)) - Document `SYNAPSE_TEST_LOG_LEVEL` to see the logger output when running tests. ([\#10148](https://github.com/matrix-org/synapse/issues/10148)) - Update the Complement build tags in GitHub Actions to test currently experimental features. ([\#10155](https://github.com/matrix-org/synapse/issues/10155)) - Add a `synapse_federation_soft_failed_events_total` metric to track how often events are soft failed. ([\#10156](https://github.com/matrix-org/synapse/issues/10156)) - Fetch the corresponding complement branch when performing CI. ([\#10160](https://github.com/matrix-org/synapse/issues/10160)) - Add some developer documentation about boolean columns in database schemas. ([\#10164](https://github.com/matrix-org/synapse/issues/10164)) - Add extra logging fields to better debug where events are being soft failed. ([\#10168](https://github.com/matrix-org/synapse/issues/10168)) - Add debug logging for when we enter and exit `Measure` blocks. ([\#10183](https://github.com/matrix-org/synapse/issues/10183)) - Improve comments in structured logging code. ([\#10188](https://github.com/matrix-org/synapse/issues/10188)) - Update [MSC3083](https://github.com/matrix-org/matrix-doc/pull/3083) support with modifications from the MSC. ([\#10189](https://github.com/matrix-org/synapse/issues/10189)) - Remove redundant DNS lookup limiter. ([\#10190](https://github.com/matrix-org/synapse/issues/10190)) - Upgrade `black` linting tool to 21.6b0. ([\#10197](https://github.com/matrix-org/synapse/issues/10197)) - Expose OpenTracing trace id in response headers. ([\#10199](https://github.com/matrix-org/synapse/issues/10199))
| * Fix wrapping of legacy check_registration_for_spam (#10238)Brendan Abolivier2021-06-231-6/+7
| | | | | | Fixes #10234
| * 1.37.0rc1Brendan Abolivier2021-06-231-1/+1
| |
* | Re-introduce "Leave out optional keys from /sync" change (#10214)Andrew Morgan2021-06-231-23/+46
| | | | | | Required some fixes due to merge conflicts with #6739, but nothing too hairy. The first commit is the same as the original (after merge conflict resolution) then two more for compatibility with the latest sync code.
* | Improve the reliability of auto-joining remote rooms (#10237)Brendan Abolivier2021-06-231-16/+47
| | | | | | | | | | | | If a room is remote and we don't have a user in it, always try to join it. It might fail if the room is invite-only, but we don't have a user to invite with, so at this point it's the best we can do. Fixes #10233 (at least to some extent)
* | Send out invite rejections and knocks over federation (#10223)Richard van der Hoff2021-06-231-0/+14
| | | | | | | | | | ensure that events sent via `send_leave` and `send_knock` are sent on to the rest of the federation.
* | Drop Origin & Accept from Access-Control-Allow-Headers value (#10114)Michael[tm] Smith2021-06-231-1/+1
|/ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Drop Origin & Accept from Access-Control-Allow-Headers value This change drops the Origin and Accept header names from the value of the Access-Control-Allow-Headers response header sent by Synapse. Per the CORS protocol, it’s not necessary or useful to include those header names. Details: Per-spec at https://fetch.spec.whatwg.org/#forbidden-header-name, Origin is a “forbidden header name” set by the browser and that frontend JavaScript code is never allowed to set. So the value of Access-Control-Allow-Headers isn’t relevant to Origin or in general to other headers set by the browser itself — the browser never ever consults the Access-Control-Allow-Headers value to confirm that it’s OK for the request to include an Origin header. And per-spec at https://fetch.spec.whatwg.org/#cors-safelisted-request-header, Accept is a “CORS-safelisted request-header”, which means that browsers allow requests to contain the Accept header regardless of whether the Access-Control-Allow-Headers value contains "Accept". So it’s unnecessary for the Access-Control-Allow-Headers to explicitly include Accept. Browsers will not perform a CORS preflight for requests containing an Accept request header. Related: https://github.com/matrix-org/matrix-doc/pull/3225 Signed-off-by: Michael[tm] Smith <mike@w3.org>
* Fix schema delta to not take as long on large servers (#10227)Erik Johnston2021-06-221-2/+5
| | | Introduced in #6739
* Warn users trying to use the deprecated spam checker interface (#10210)Brendan Abolivier2021-06-221-0/+15
| | | So admins aren't surprised if things break when we remove this code in a couple of months.
* Add endpoints for backfilling history (MSC2716) (#9247)Eric Eastwood2021-06-229-18/+565
| | | Work on https://github.com/matrix-org/matrix-doc/pull/2716
* Implement config option `sso.update_profile_information` (#10108)jkanefendt2021-06-212-1/+39
| | | | | | Implemented config option sso.update_profile_information to keep user's display name in sync with the SSO displayname. Signed-off-by: Johannes Kanefendt <johannes.kanefendt@krzn.de>
* Fix performance of responding to user key requests over federation (#10221)Erik Johnston2021-06-212-1/+10
| | | | | We were repeatedly looking up a config option in a loop (using the unclassed config style), which is expensive enough that it can cause large CPU usage.
* Check third party rules before persisting knocks over federation (#10212)Andrew Morgan2021-06-211-2/+2
| | | | | An accidental mis-ordering of operations during #6739 technically allowed an incoming knock event over federation in before checking it against any configured Third Party Access Rules modules. This PR corrects that by performing the TPAR check *before* persisting the event.
* Ensure that errors during startup are written to the logs and the console. ↵Richard van der Hoff2021-06-214-16/+54
| | | | | | | | (#10191) * Defer stdio redirection until we are about to start the reactor * Catch and handle exceptions during startup
* Fix a missing await when in the spaces summary. (#10208)Patrick Cloke2021-06-181-2/+1
| | | | | | | This could cause a minor data leak if someone defined a non-restricted join rule with an allow key or used a restricted join rule in an older room version, but this is unlikely. Additionally this starts adding unit tests to the spaces summary handler.
* Standardise the module interface (#10062)Brendan Abolivier2021-06-1813-126/+375
| | | This PR adds a common configuration section for all modules (see docs). These modules are then loaded at startup by the homeserver. Modules register their hooks and web resources using the new `register_[...]_callbacks` and `register_web_resource` methods of the module API.
* Expose opentracing trace id in response headers (#10199)Richard van der Hoff2021-06-182-0/+24
| | | Fixes: #9480
* Remove support for ACME v1 (#10194)Brendan Abolivier2021-06-179-455/+7
| | | | | Fixes #9778 ACME v1 has been fully decommissioned for existing installs on June 1st 2021(see https://community.letsencrypt.org/t/end-of-life-plan-for-acmev1/88430/27), so we can now safely remove it from Synapse.
* Update MSC3083 support per changes in the MSC. (#10189)Patrick Cloke2021-06-173-31/+46
| | | Adds a "type" field and generalize "space" to "room_id".
* Ensure that we do not cache empty sync responses after a timeout (#10158)Richard van der Hoff2021-06-173-15/+29
| | | Fixes #8518 by telling the ResponseCache not to cache the /sync response if the next_batch param is the same as the since token.
* Rip out the DNS lookup limiter (#10190)Richard van der Hoff2021-06-171-104/+0
| | | As I've written in various places in the past (#7113, #9865) I'm pretty sure this is doing nothing useful at all.
* update black to 21.6b0 (#10197)Marcus2021-06-174-5/+5
| | | | | Reformat all files with the new version. Signed-off-by: Marcus Hoffmann <bubu@bubu1.eu>
* Fix incorrect time magnitude on delayed call (#10195)Andrew Morgan2021-06-171-1/+1
| | | | | | | | | Fixes https://github.com/matrix-org/synapse/issues/10030. We were expecting milliseconds where we should have provided a value in seconds. The impact of this bug isn't too bad. The code is intended to count the number of remote servers that the homeserver can see and report that as a metric. This metric is supposed to run initially 1 second after server startup, and every 60s as well. Instead, it ran 1,000 seconds after server startup, and every 60s after startup. This fix allows for the correct metrics to be collected immediately, as well as preventing a random collection 1,000s in the future after startup.
* Add fields to better debug where events are being soft_failed (#10168)Eric Eastwood2021-06-171-3/+18
| | | Follow-up to https://github.com/matrix-org/synapse/pull/10156#discussion_r650292223
* Fix persist_events to stop leaking opentracing contexts (#10193)Richard van der Hoff2021-06-171-1/+1
|
* Improve comments in the structured logging code. (#10188)Patrick Cloke2021-06-161-4/+5
|
* Always require users to re-authenticate for dangerous operations. (#10184)Patrick Cloke2021-06-164-1/+19
| | | | | | | Dangerous actions means deactivating an account, modifying an account password, or adding a 3PID. Other actions (deleting devices, uploading keys) can re-use the same UI auth session if ui_auth.session_timeout is configured.
* Add debug logging when we enter/exit Measure block (#10183)Erik Johnston2021-06-161-0/+5
| | | It can be helpful to know when trying to track down slow requests.
* Fix requestOpenIdToken response: integer expires_in (#10175)Lukas Lihotzki2021-06-161-1/+1
| | | | | | | | | | `expires_in` must be an integer according to the OpenAPI spec: https://github.com/matrix-org/matrix-doc/blob/master/data/api/client-server/definitions/openid_token.yaml#L32 True division (`/`) returns a float instead (`"expires_in": 3600.0`). Floor division (`//`) returns an integer, so the response is spec compliant. Signed-off-by: Lukas Lihotzki <lukas@lihotzki.de>
* Make opentracing trace into event persistence (#10134)Richard van der Hoff2021-06-163-9/+98
| | | | | | | | | | | | | | * Trace event persistence When we persist a batch of events, set the parent opentracing span to the that from the request, so that we can trace all the way in. * changelog * When we force tracing, set a baggage item ... so that we can check again later. * Link in both directions between persist_events spans
* Merge branch 'master' into developErik Johnston2021-06-151-1/+1
|\
| * 1.36.0 v1.36.0 github/release-v1.36 release-v1.36Erik Johnston2021-06-151-1/+1
| |
* | Remove support for unstable MSC1772 prefixes. (#10161)Patrick Cloke2021-06-152-16/+3
| | | | | | | | The stable prefixes have been supported since v1.34.0. The unstable prefixes are not supported by any known clients.
* | Remove the experimental flag for knocking and use stable prefixes / ↵Patrick Cloke2021-06-1510-79/+19
| | | | | | | | | | | | | | endpoints. (#10167) * Room version 7 for knocking. * Stable prefixes and endpoints (both client and federation) for knocking. * Removes the experimental configuration flag.
* | Fix 'ip_range_whitelist' not working for federation servers (#10115)Michael Kutzner2021-06-152-13/+18
| | | | | | | | | | Add 'federation_ip_range_whitelist'. This allows backwards-compatibility, If 'federation_ip_range_blacklist' is set. Otherwise 'ip_range_whitelist' will be used for federation servers. Signed-off-by: Michael Kutzner 1mikure@gmail.com
* | Refactor `EventPersistenceQueue` (#10145)Richard van der Hoff2021-06-141-77/+88
| | | | | | some cleanup, pulled out of #10134.
* | Extend `ResponseCache` to pass a context object into the callback (#10157)Richard van der Hoff2021-06-143-30/+77
| | | | | | | | | | This is the first of two PRs which seek to address #8518. This first PR lays the groundwork by extending ResponseCache; a second PR (#10158) will update the SyncHandler to actually use it, and fix the bug. The idea here is that we allow the callback given to ResponseCache.wrap to decide whether its result should be cached or not. We do that by (optionally) passing a ResponseCacheContext into it, which it can modify.
* | Merge tag 'v1.36.0rc2' into developErik Johnston2021-06-111-1/+1
|\| | | | | | | | | | | | | | | | | | | | | | | Synapse 1.36.0rc2 (2021-06-11) ============================== Bugfixes -------- - Fix a bug which caused presence updates to stop working some time after a restart, when using a presence writer worker. Broke in v1.33.0. ([\#10149](https://github.com/matrix-org/synapse/issues/10149)) - Fix a bug when using federation sender worker where it would send out more presence updates than necessary, leading to high resource usage. Broke in v1.33.0. ([\#10163](https://github.com/matrix-org/synapse/issues/10163)) - Fix a bug where Synapse could send the same presence update to a remote twice. ([\#10165](https://github.com/matrix-org/synapse/issues/10165))
| * 1.36.0rc2Erik Johnston2021-06-111-1/+1
| |
| * Only send a presence state to a destination once (#10165)Erik Johnston2021-06-111-13/+12
| | | | | | | | It turns out that we were sending the same presence state to a remote potentially multiple times.
| * Fix sending presence over federation when using workers (#10163)Erik Johnston2021-06-111-6/+19
| | | | | | | | | | | | When using a federation sender we'd send out all local presence updates over federation even when they shouldn't be. Fixes #10153.
| * Fix bug when running presence off master (#10149)Erik Johnston2021-06-112-1/+16
| | | | | | Hopefully fixes #10027.
* | Only send a presence state to a destination once (#10165)Erik Johnston2021-06-111-13/+12
| | | | | | | | It turns out that we were sending the same presence state to a remote potentially multiple times.
* | Support for database schema version ranges (#9933)Richard van der Hoff2021-06-114-76/+108
| | | | | | This is essentially an implementation of the proposal made at https://hackmd.io/@richvdh/BJYXQMQHO, though the details have ended up looking slightly different.
* | Fix sending presence over federation when using workers (#10163)Erik Johnston2021-06-111-6/+19
| | | | | | | | | | | | When using a federation sender we'd send out all local presence updates over federation even when they shouldn't be. Fixes #10153.
* | Clean up a broken import in admin_cmd.py (#10154)Andrew Morgan2021-06-111-2/+0
| |
* | Fix bug when running presence off master (#10149)Erik Johnston2021-06-112-1/+16
| | | | | | Hopefully fixes #10027.
* | Add metrics to track how often events are `soft_failed` (#10156)Eric Eastwood2021-06-111-0/+7
| | | | | | | | | | | | | | | | | | | | | | Spawned from missing messages we were seeing on `matrix.org` from a federated Gtiter bridged room, https://gitlab.com/gitterHQ/webapp/-/issues/2770. The underlying issue in Synapse is tracked by https://github.com/matrix-org/synapse/issues/10066 where the message and join event race and the message is `soft_failed` before the `join` event reaches the remote federated server. Less soft_failed events = better and usually this should only trigger for events where people are doing bad things and trying to fuzz and fake everything.
* | Mention that you need to configure max upload size in reverse proxy as well ↵Aaron Raimist2021-06-101-0/+4
| | | | | | | | | | (#10122) Signed-off-by: Aaron Raimist <aaron@raim.ist>
* | Integrate knock rooms with the public rooms directory (#9359)Andrew Morgan2021-06-092-5/+10
| | | | | | | | | | | | | | | | | | | | This PR implements the ["Changes regarding the Public Rooms Directory"](https://github.com/Sorunome/matrix-doc/blob/soru/knock/proposals/2403-knock.md#changes-regarding-the-public-rooms-directory) section of knocking MSC2403. Specifically, it: * Allows rooms with `join_rule` "knock" to be returned by the query behind the public rooms directory * Adds the field `join_rule` to each room entry returned by a public rooms directory query, so clients can know whether to attempt a join or knock on a room Based on https://github.com/matrix-org/synapse/issues/6739. Complement tests for this change: https://github.com/matrix-org/complement/pull/72
* | Implement knock feature (#6739)Sorunome2021-06-0926-116/+1217
| | | | | | | | | | | | This PR aims to implement the knock feature as proposed in https://github.com/matrix-org/matrix-doc/pull/2403 Signed-off-by: Sorunome mail@sorunome.de Signed-off-by: Andrew Morgan andrewm@element.io
* | Limit the number of in-flight /keys/query requests from a single device. ↵Patrick Cloke2021-06-092-170/+185
| | | | | | | | (#10144)
* | Clean up the interface for injecting opentracing over HTTP (#10143)Richard van der Hoff2021-06-093-92/+25
| | | | | | | | | | | | | | * Remove unused helper functions * Clean up the interface for injecting opentracing over HTTP * changelog
* | Add type hints to the federation server transport. (#10080)Patrick Cloke2021-06-084-75/+193
|/
* 1.36.0rc1Andrew Morgan2021-06-081-1/+1
|
* Fix logging context when opening new DB connection (#10141)Erik Johnston2021-06-081-3/+9
| | | | Fixes #10140
* Correct type hints for parse_string(s)_from_args. (#10137)Patrick Cloke2021-06-086-83/+130
|
* When joining a remote room limit the number of events we concurrently check ↵Erik Johnston2021-06-084-256/+201
| | | | | signatures/hashes for (#10117) If we do hundreds of thousands at once the memory overhead can easily reach 500+ MB.
* Handle /backfill returning no events (#10133)Erik Johnston2021-06-081-13/+25
| | | Fixes #10123
* More database opentracing (#10136)Richard van der Hoff2021-06-071-2/+5
| | | Add a couple of extra logs/spans, to give a bit of a better idea.
* opentracing: use a consistent name for background processes (#10135)Richard van der Hoff2021-06-072-2/+4
| | | | ... otherwise we tend to get a namespace clash between the bg process and the functions that it calls.
* Always update AS last_pos, even on no events (#10107)14mRh4X0r2021-06-071-13/+12
| | | | | | | | | | Fixes #1834. `get_new_events_for_appservice` internally calls `get_events_as_list`, which will filter out any rejected events. If all returned events are filtered out, `_notify_interested_services` will return without updating the last handled stream position. If there are 100 consecutive such events, processing will halt altogether. Breaking the loop is now done by checking whether we're up-to-date with `current_max` in the loop condition, instead of relying on an empty `events` list. Signed-off-by: Willem Mulder <14mRh4X0r@gmail.com>
* Add missing type hints to the admin API servlets (#10105)Dirk Klimpel2021-06-075-40/+47
|
* Delete completes to-device messages earlier in /sync (#10124)Richard van der Hoff2021-06-071-10/+11
| | | | I hope this will improve https://github.com/matrix-org/synapse/issues/9564.
* Don't try and backfill the same room in parallel. (#10116)Erik Johnston2021-06-041-0/+8
| | | | | If backfilling is slow then the client may time out and retry, causing Synapse to start a new `/backfill` before the existing backfill has finished, duplicating work.
* Limit number of events in a replication request (#10118)Erik Johnston2021-06-041-2/+3
| | | Fixes #9956.
* Enable Prometheus metrics for the jaeger client library (#10112)Richard van der Hoff2021-06-041-0/+3
|
* Add OpenTracing for database activity. (#10113)Richard van der Hoff2021-06-032-33/+59
| | | | | | | | | | This adds quite a lot of OpenTracing decoration for database activity. Specifically it adds tracing at four different levels: * emit a span for each "interaction" - ie, the top level database function that we tend to call "transaction", but isn't really, because it can end up as multiple transactions. * emit a span while we hold a database connection open * emit a span for each database transaction - actual actual transaction. * emit a span for each database query. I'm aware this might be quite a lot of overhead, but even just running it on a local Synapse it looks really interesting, and I hope the overhead can be offset just by turning down the sampling frequency and finding other ways of tracing requests of interest (eg, the `force_tracing_for_users` setting).
* Improve opentracing annotations for Notifier (#10111)Richard van der Hoff2021-06-031-33/+33
| | | | | | | The existing tracing reports an error each time there is a timeout, which isn't really representative. Additionally, we log things about the way `wait_for_events` works (eg, the result of the callback) to the *parent* span, which is confusing.
* r0.6.1 support: /rooms/:roomId/aliases endpoint (#9224)Travis Ralston2021-06-031-1/+1
| | | | | | [MSC2432](https://github.com/matrix-org/matrix-doc/pull/2432) added this endpoint originally but it has since been included in the spec for nearly a year. This is progress towards https://github.com/matrix-org/synapse/issues/8334
* Merge branch 'master' into developPatrick Cloke2021-06-031-1/+1
|\
| * 1.35.1Patrick Cloke2021-06-031-1/+1
| |
| * Do not show invite-only rooms in spaces summary (unless joined/invited). ↵Patrick Cloke2021-06-031-10/+9
| | | | | | | | (#10109)
* | Add new admin APIs to remove media by media ID from quarantine. (#10044)Dirk Klimpel2021-06-022-10/+50
| | | | | | | | | | Related to: #6681, #5956, #10040 Signed-off-by: Dirk Klimpel dirk@klimpel.org
* | Rewrite the KeyRing (#10035)Erik Johnston2021-06-024-378/+281
| |
* | Do not show invite-only rooms in spaces summary (unless joined/invited). ↵Patrick Cloke2021-06-021-10/+9
| | | | | | | | (#10109)
* | Make /sync do less state res (#10102)Erik Johnston2021-06-021-2/+2
| |
* | Merge branch 'master' into developAndrew Morgan2021-06-013-2/+21
|\|
| * 1.35.0Andrew Morgan2021-06-011-1/+1
| |
| * 1.35.0rc3Erik Johnston2021-05-281-1/+1
| |
| * Merge remote-tracking branch 'origin/release-v1.35.0' into release-v1.35Erik Johnston2021-05-281-1/+1
| |\
| * | Limit the number of events sent over replication when persisting events. ↵Brendan Abolivier2021-05-281-7/+10
| | | | | | | | | | | | (#10082)
| * | Log method and path when dropping request due to size limit (#10091)Erik Johnston2021-05-281-1/+3
| | |
| * | Allow response of `/send_join` to be larger. (#10093)Erik Johnston2021-05-282-1/+20
| | | | | | | | | Fixes #10087.
* | | add a cache to have_seen_event (#9953)Richard van der Hoff2021-06-014-19/+81
| | | | | | | | | Empirically, this helped my server considerably when handling gaps in Matrix HQ. The problem was that we would repeatedly call have_seen_events for the same set of (50K or so) auth_events, each of which would take many minutes to complete, even though it's only an index scan.
* | | Fix opentracing inject to use the SpanContext, not the Span (#10074)Brad Murray2021-06-011-5/+5
| | | | | | | | | Signed-off-by: Brad Murray brad@beeper.com
* | | Set opentracing priority before setting other tags (#10092)Richard van der Hoff2021-05-284-11/+31
| | | | | | | | | ... because tags on spans which aren't being sampled get thrown away.
* | | Log method and path when dropping request due to size limit (#10091)Erik Johnston2021-05-281-1/+3
| | |
* | | Add `parse_strings_from_args` to get `prev_events` array (#10048)Eric Eastwood2021-05-281-43/+153
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Split out from https://github.com/matrix-org/synapse/pull/9247 Strings: - `parse_string` - `parse_string_from_args` - `parse_strings_from_args` For comparison with ints: - `parse_integer` - `parse_integer_from_args` Previous discussions: - https://github.com/matrix-org/synapse/pull/9247#discussion_r573195687 - https://github.com/matrix-org/synapse/pull/9247#discussion_r574214156 - https://github.com/matrix-org/synapse/pull/9247#discussion_r573264791 Signed-off-by: Eric Eastwood <erice@element.io>
* | | Move some more endpoints off master (#10084)Erik Johnston2021-05-272-6/+6
| | |
* | | Make reason and score optional for report_event (#10077)Callum Brown2021-05-272-10/+5
| | | | | | | | | | | | | | | | | | Implements MSC2414: https://github.com/matrix-org/matrix-doc/pull/2414 See #8551 Signed-off-by: Callum Brown <callum@calcuode.com>
* | | Limit the number of events sent over replication when persisting events. ↵Brendan Abolivier2021-05-271-7/+10
| | | | | | | | | | | | (#10082)
* | | Merge tag 'v1.35.0rc2' into developErik Johnston2021-05-272-2/+2
|\ \ \ | | |/ | |/| | | | | | | | | | | | | | | | | | | | | | Synapse 1.35.0rc2 (2021-05-27) ============================== Bugfixes -------- - Fix a bug introduced in v1.35.0rc1 when calling the spaces summary API via a GET request. ([\#10079](https://github.com/matrix-org/synapse/issues/10079))
| * | 1.35.0rc2 v1.35.0rc2 github/release-v1.35.0 release-v1.35.0Erik Johnston2021-05-271-1/+1
| |/
| * Pass the origin when calculating the spaces summary over GET. (#10079)Patrick Cloke2021-05-271-1/+1
| | | | | | | | Fixes a bug due to conflicting PRs which were merged. (One added a new caller to a method, the other added a new parameter to the same method.)
* | Fix up `BatchingQueue` (#10078)Erik Johnston2021-05-271-22/+48
| | | | | | Fixes #10068
* | Remove redundant code to reload tls cert (#10054)Richard van der Hoff2021-05-272-23/+4
| | | | | | | | | | we don't need to reload the tls cert if we don't have any tls listeners. Follow-up to #9280.
* | Combine `LruCache.invalidate` and `invalidate_many` (#9973)Richard van der Hoff2021-05-2710-49/+48
| | | | | | | | | | | | | | | | | | | | * Make `invalidate` and `invalidate_many` do the same thing ... so that we can do either over the invalidation replication stream, and also because they always confused me a bit. * Kill off `invalidate_many` * changelog
* | Remove the experimental spaces enabled flag. (#10063)Patrick Cloke2021-05-264-34/+8
| | | | | | | | In lieu of just always enabling the unstable spaces endpoint and unstable room version.
* | Add an admin API for unprotecting local media from quarantine (#10040)Dirk Klimpel2021-05-262-5/+30
|/ | | Signed-off-by: Dirk Klimpel dirk@klimpel.org
* 1.35.0rc1 v1.35.0rc1Erik Johnston2021-05-251-1/+1
|
* Add missing type hints to synapse.util (#9982)Patrick Cloke2021-05-246-23/+27
|
* Remove tls_fingerprints option (#9280)Jerin J Titus2021-05-243-61/+0
| | | | Signed-off-by: Jerin J Titus <72017981+jerinjtitus@users.noreply.github.com>
* Remove `keylen` from `LruCache`. (#9993)Richard van der Hoff2021-05-248-51/+73
| | | | | | | `keylen` seems to be a thing that is frequently incorrectly set, and we don't really need it. The only time it was used was to figure out if we had removed a subtree in `del_multi`, which we can do better by changing `TreeCache.pop` to return a different type (`TreeCacheNode`). Commits should be independently reviewable.
* Enable experimental spaces by default. (#10011)Patrick Cloke2021-05-242-2/+19
| | | | The previous spaces_enabled flag now defaults to true and is exposed in the sample config.
* Fix `get_state_ids_for_event` return type typo to match what the function ↵Eric Eastwood2021-05-241-1/+1
| | | | | actually does (#10050) It looks like a typo copy/paste from `get_state_for_event` above.
* Don't hammer the database for destination retry timings every ~5mins (#10036)Erik Johnston2021-05-216-60/+43
|
* Fix /upload 500'ing when presented a very large image (#10029)Michael Telatynski2021-05-212-0/+11
| | | | | | | | | | | | * Fix /upload 500'ing when presented a very large image Catch DecompressionBombError and re-raise as ThumbnailErrors * Set PIL's MAX_IMAGE_PIXELS to match homeserver.yaml to get it to bomb out quicker, to load less into memory in the case of super large images * Add changelog entry for 10029
* Remove unused properties from the SpaceSummaryHandler. (#10038)Patrick Cloke2021-05-211-2/+0
|
* Fixed removal of new presence stream states (#10014)Marek Matys2021-05-211-9/+9
| | | | | | | | | Fixes: https://github.com/matrix-org/synapse/issues/9962 This is a fix for above problem. I fixed it by swaping the order of insertion of new records and deletion of old ones. This ensures that we don't delete fresh database records as we do deletes before inserts. Signed-off-by: Marek Matys <themarcq@gmail.com>
* Add a batching queue implementation. (#10017)Erik Johnston2021-05-211-0/+153
|
* Add `Keyring.verify_events_for_server` and reduce memory usage (#10018)Erik Johnston2021-05-202-22/+93
| | | | | | Also add support for giving a callback to generate the JSON object to verify. This should reduce memory usage, as we no longer have the event in memory in dict form (which has a large memory footprint) for extend periods of time.
* Use ijson to parse the response to `/send_join`, reducing memory usage. (#9958)Erik Johnston2021-05-205-58/+223
| | | Instead of parsing the full response to `/send_join` into Python objects (which can be huge for large rooms) and *then* parsing that into events, we instead use ijson to stream parse the response directly into `EventBase` objects.
* Allow a user who could join a restricted room to see it in spaces summary. ↵Patrick Cloke2021-05-203-54/+253
| | | | | | (#9922) This finishes up the experimental implementation of MSC3083 by showing the restricted rooms in the spaces summary (from MSC2946).
* Refactor checking restricted join rules (#10007)Patrick Cloke2021-05-183-51/+49
| | | | | To be more consistent with similar code. The check now automatically raises an AuthError instead of passing back a boolean. It also absorbs some shared logic between callers.
* Use a database table to hold the users that should have full presence sent ↵Andrew Morgan2021-05-186-65/+245
| | | | to them, instead of something in-memory (#9823)
* Fix the allowed range of valid ordering characters for spaces. (#10002)Patrick Cloke2021-05-171-2/+2
| | | | \x7F was meant to be \0x7E (~) this was originally incorrect in MSC1772.
* Clarify comments in the space summary handler. (#9974)Patrick Cloke2021-05-171-5/+46
|
* Merge branch 'master' into developErik Johnston2021-05-171-1/+1
|\
| * 1.34.0Erik Johnston2021-05-171-1/+1
| |
* | Split multiplart email sending into a dedicated handler (#9977)Brendan Abolivier2021-05-174-90/+121
| | | | | | Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
* | Minor `@cachedList` enhancements (#9975)Richard van der Hoff2021-05-144-17/+16
| | | | | | | | | | | | - use a tuple rather than a list for the iterable that is passed into the wrapped function, for performance - test that we can pass an iterable and that keys are correctly deduped.
* | Remove unnecessary SystemRandom from SQLBaseStore (#9987)Dan Callahan2021-05-142-3/+2
| | | | | | | | | | | | | | | | It's not obvious that instances of SQLBaseStore each need their own instances of random.SystemRandom(); let's just use random directly. Introduced by 52839886d664576831462e033b88e5aba4c019e3 Signed-off-by: Dan Callahan <danc@element.io>
* | Remove superfluous call to bool() (#9986)Dan Callahan2021-05-141-1/+1
| | | | | | | | | | Our strtobool already returns a bool, so no need to re-cast here Signed-off-by: Dan Callahan <danc@element.io>
* | Simplify exception handling in is_ascii. (#9985)Dan Callahan2021-05-141-3/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | We can get away with just catching UnicodeError here. ⋮ +-- ValueError | +-- UnicodeError | +-- UnicodeDecodeError | +-- UnicodeEncodeError | +-- UnicodeTranslateError ⋮ https://docs.python.org/3/library/exceptions.html#exception-hierarchy Signed-off-by: Dan Callahan <danc@element.io>
* | Use Python's secrets module instead of random (#9984)Dan Callahan2021-05-141-8/+11
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Functionally identical, but more obviously cryptographically secure. ...Explicit is better than implicit? Avoids needing to know that SystemRandom() implies a CSPRNG, and complies with the big scary red box on the documentation for random: > Warning: > The pseudo-random generators of this module should not be used for > security purposes. For security or cryptographic uses, see the > secrets module. https://docs.python.org/3/library/random.html Signed-off-by: Dan Callahan <danc@element.io>
* | Support enabling opentracing by user (#9978)Richard van der Hoff2021-05-142-6/+36
|/ | | | | Add a config option which allows enabling opentracing by user id, eg for debugging requests made by a test user.
* 1.34.0rc1Brendan Abolivier2021-05-121-1/+1
|
* Correctly ratelimit invites when creating a room (#9968)Brendan Abolivier2021-05-123-12/+62
| | | | | * Correctly ratelimit invites when creating a room Also allow ratelimiting for more than one action at a time.
* Change the format of access tokens away from macaroons (#5588)Richard van der Hoff2021-05-123-10/+42
|
* Run cache_joined_hosts_for_event in background (#9951)Erik Johnston2021-05-121-5/+40
|
* Sort child events according to MSC1772 for the spaces summary API. (#9954)Patrick Cloke2021-05-111-2/+69
| | | | | | | | | This should help ensure that equivalent results are achieved between homeservers querying for the summary of a space. This implements modified MSC1772 rules, according to MSC2946. The different is that the origin_server_ts of the m.room.create event is not used as a tie-breaker since this might not be known if the homeserver is not part of the room.
* Support fetching the spaces summary via GET over federation. (#9947)Patrick Cloke2021-05-113-0/+28
| | | | | | | | | | | Per changes in MSC2946, the C-S and S-S APIs for spaces summary should use GET requests. Until this is stable, the POST endpoints still exist. This does not switch federation requests to use the GET version yet since it is newly added and already deployed servers might not support it. When switching to the stable endpoint we should switch to GET requests.
* Send the `m.room.create` stripped event with invites (support MSC1772). (#9966)Patrick Cloke2021-05-111-4/+2
| | | | | MSC1772 specifies the m.room.create event should be sent as part of the invite_state. This was done optionally behind an experimental flag, but is now done by default due to MSC1772 being approved.
* Merge branch 'master' into developBrendan Abolivier2021-05-114-69/+53
|\
| * 1.33.2Brendan Abolivier2021-05-111-1/+1
| |
| * Merge pull request from GHSA-x345-32rc-8h85Richard van der Hoff2021-05-113-68/+52
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * tests for push rule pattern matching * tests for acl pattern matching * factor out common `re.escape` * Factor out common re.compile * Factor out common anchoring code * add word_boundary support to `glob_to_regex` * Use `glob_to_regex` in push rule evaluator NB that this drops support for character classes. I don't think anyone ever used them. * Improve efficiency of globs with multiple wildcards The idea here is that we compress multiple `*` globs into a single `.*`. We also need to consider `?`, since `*?*` is as hard to implement efficiently as `**`. * add assertion on regex pattern * Fix mypy * Simplify glob_to_regex * Inline the glob_to_regex helper function Signed-off-by: Dan Callahan <danc@element.io> * Moar comments Signed-off-by: Dan Callahan <danc@element.io> Co-authored-by: Dan Callahan <danc@element.io>
* | Add config option to hide device names over federation (#9945)Aaron Raimist2021-05-112-1/+13
| | | | | | | | | | Now that cross signing exists there is much less of a need for other people to look at devices and verify them individually. This PR adds a config option to allow you to prevent device display names from being shared with other servers. Signed-off-by: Aaron Raimist <aaron@raim.ist>
* | Add debug logging for issue #9533 (#9959)Richard van der Hoff2021-05-115-2/+41
| | | | | | | | | | Hopefully this will help us track down where to-device messages are getting lost/delayed.
* | Fix `m.room_key_request` to-device messages (#9961)Richard van der Hoff2021-05-113-26/+31
| | | | | | fixes #9960
* | Improve performance of backfilling in large rooms. (#9935)Erik Johnston2021-05-101-69/+54
| | | | | | | | | | | | We were pulling the full auth chain for the room out of the DB each time we backfilled, which can be *huge* for large rooms and is totally unnecessary.
* | Always cache 'event_to_prev_state_group' (#9950)Erik Johnston2021-05-071-6/+7
| | | | | | Fixes regression in send PDU times introduced in #9905.
* | Merge remote-tracking branch 'origin/release-v1.33.2' into developErik Johnston2021-05-071-1/+1
|\|
| * Unpin attrs dep after new version has been released (#9946)Erik Johnston2021-05-071-1/+1
| | | | | | c.f. #9936
* | Reorganise the database schema directories (#9932)Richard van der Hoff2021-05-07282-44/+79
| | | | | | | | | | The hope here is that by moving all the schema files into synapse/storage/schema, it gets a bit easier for newcomers to navigate. It certainly got easier for me to write a helpful README. There's more to do on that front, but I'll follow up with other PRs for that.
* | Export jemalloc stats to prometheus when used (#9882)Erik Johnston2021-05-063-0/+199
| |
* | Revert "Leave out optional keys from /sync (#9919)" (#9940)Erik Johnston2021-05-061-44/+18
| | | | | | This reverts commit e9eb3549d32a6f93d07de8dbd5e1ebe54c8d8278.
* | Merge remote-tracking branch 'origin/master' into developErik Johnston2021-05-062-2/+3
|\|
| * 1.33.1 v1.33.1 github/release-v1.33.1 release-v1.33.1Erik Johnston2021-05-061-1/+1
| |
| * Pin attrs to <21.1.0 (#9937)Erik Johnston2021-05-061-1/+2
| | | | | | Fixes #9936
* | Follow-up to #9915 to correct the identifier for room types.Patrick Cloke2021-05-051-1/+1
| |
* | Include the time of the create event in Spaces Summary. (#9928)Patrick Cloke2021-05-051-0/+1
| | | | | | | | | | This is an update based on changes to MSC2946. The origin_server_ts of the m.room.create event is copied into the creation_ts field for each room returned from the spaces summary.
* | Increase perf of handling presence when joining large rooms. (#9916)Erik Johnston2021-05-051-72/+82
| |
* | Support stable MSC1772 spaces identifiers. (#9915)Patrick Cloke2021-05-052-2/+9
| | | | | | | | Support both the unstable and stable identifiers. A future release will disable the unstable identifiers.
* | Optionally track memory usage of each LruCache (#9881)Erik Johnston2021-05-056-1/+93
| | | | | | | | | | This will double count slightly in the presence of interned strings. It's off by default as it can consume a lot of resources.
generated by cgit-pink 1.4.1 (git 2.36.1) at 2024-11-15 01:44:58 +0000