summary refs log tree commit diff
path: root/synapse/handlers (follow)
Commit message (Collapse)AuthorAgeFilesLines
* Handle OTK uploads off master (#17271)Erik Johnston2024-06-061-29/+55
| | | And fallback keys uploads. Only device keys need handling on master
* Don't try and resync devices for down hosts (#17273)Erik Johnston2024-06-061-6/+18
| | | | It's just a waste of time if we won't even query the remote host as its marked as down.
* Always return OTK counts (#17275)Erik Johnston2024-06-061-4/+29
| | | Broke in https://github.com/element-hq/synapse/pull/17215
* Use fully-qualified `PersistedEventPosition` when returning `RoomsForUser` ↵Eric Eastwood2024-06-045-69/+8
| | | | | | | (#17265) Use fully-qualified `PersistedEventPosition` (`instance_name` and `stream_ordering`) when returning `RoomsForUser` to facilitate proper comparisons and `RoomStreamToken` generation. Spawning from https://github.com/element-hq/synapse/pull/17187 where we want to utilize this change
* Fix bug where typing replication breaks (#17252)Erik Johnston2024-05-311-3/+3
| | | | This can happen on restarts of the service, due to old rooms being pruned.
* Fix logging errors when receiving invalid User ID for key querys (#17250)Erik Johnston2024-05-311-0/+5
|
* In sync wait for worker to catch up since token (#17215)Erik Johnston2024-05-301-0/+35
| | | | | | | Otherwise things will get confused. An alternative would be to make sure that for lagging stream we don't return anything (and make sure the returned next_batch token doesn't go backwards). But that is a faff.
* Fix deduplicating of membership events to not create unused state groups. ↵Erik Johnston2024-05-302-35/+32
| | | | | | | | | | | | | (#17164) We try and deduplicate in two places: 1) really early on, and 2) just before we persist the event. The first case was broken due to it occuring before the profile information was added, and so it thought the event contents were different. The second case did catch it and handle it correctly, however doing so creates a redundant state group leading to bloat. Fixes #3791
* Don't invalidate all `get_relations_for_event` on history purge (#17083)Erik Johnston2024-05-291-1/+1
| | | | This is a tree cache already, so may as well move the room ID to the front and use that
* Ignore attempts to send to-device messages to bad users (#17240)Erik Johnston2024-05-291-0/+7
| | | | | | | | Currently sending a to-device message to a user ID with a dodgy destination is accepted, but then ends up spamming the logs when we try and send to the destination. An alternative would be to reject the request, but I'm slightly nervous that could break things.
* Handle duplicate OTK uploads racing (#17241)Erik Johnston2024-05-291-33/+45
| | | Currently this causes one of then to 500.
* Add Sliding Sync `/sync/e2ee` endpoint for To-Device messages (#17167)Eric Eastwood2024-05-231-10/+237
| | | | | | | | | | | This is being introduced as part of Sliding Sync but doesn't have any sliding window component. It's just a way to get E2EE events without having to sit through a big initial sync (`/sync` v2). And we can avoid encryption events being backed up by the main sync response or vice-versa. Part of some Sliding Sync simplification/experimentation. See [this discussion](https://github.com/element-hq/synapse/pull/17167#discussion_r1610495866) for why it may not be as useful as we thought. Based on: - https://github.com/matrix-org/matrix-spec-proposals/pull/3575 - https://github.com/matrix-org/matrix-spec-proposals/pull/3885 - https://github.com/matrix-org/matrix-spec-proposals/pull/3884
* Log exceptions when failing to auto-join new user according to the ↵reivilibre2024-05-221-1/+1
| | | | | | | `auto_join_rooms` option. (#17176) Would have been useful for tracking down #16878. Signed-off-by: Olivier 'reivilibre <oliverw@matrix.org>
* Reduce work of calculating outbound device pokes (#17211)Erik Johnston2024-05-221-0/+7
|
* Bring auto-accept invite logic into Synapse (#17147)devonh2024-05-211-1/+1
| | | | | | | | | | | | | | This PR ports the logic from the [synapse_auto_accept_invite](https://github.com/matrix-org/synapse-auto-accept-invite) module into synapse. I went with the naive approach of injecting the "module" next to where third party modules are currently loaded. If there is a better/preferred way to handle this, I'm all ears. It wasn't obvious to me if there was a better location to add this logic that would cleanly apply to all incoming invite events. Relies on https://github.com/element-hq/synapse/pull/17166 to fix linter errors.
* Improve perf of sync device lists (#17216)Erik Johnston2024-05-212-35/+25
| | | | | | | | Re-introduces #17191, and includes #17197 and #17214 The basic idea is to stop calling `get_rooms_for_user` everywhere, and instead use the table `device_lists_changes_in_room`. Commits reviewable one-by-one.
* Refactor `SyncResultBuilder` assembly to its own function (#17202)Eric Eastwood2024-05-161-116/+148
| | | | | | We will re-use `get_sync_result_builder(...)` in https://github.com/element-hq/synapse/pull/17167 Split out from https://github.com/element-hq/synapse/pull/17167
* Fix `joined_rooms`/`joined_room_ids` usage (#17208)Eric Eastwood2024-05-161-1/+1
| | | | | | | | This change was introduced in https://github.com/element-hq/synapse/pull/17203 But then https://github.com/element-hq/synapse/pull/17207 was reverted which brought back usage `joined_rooms` that needed to be updated. Wasn't caught because `develop` wasn't up to date before merging.
* Rename to be obvious: `joined_rooms` -> `joined_room_ids` (#17203)Eric Eastwood2024-05-161-2/+2
| | | Split out from https://github.com/element-hq/synapse/pull/17167
* Removed `request_key` from the `SyncConfig` (moved outside as its own ↵Eric Eastwood2024-05-161-3/+3
| | | | | | | | | function parameter) (#17201) Removed `request_key` from the `SyncConfig` (moved outside as its own function parameter) so it doesn't have to flow into `_generate_sync_entry_for_xxx` methods. This way we can separate the concerns of caching from generating the response and reuse the `_generate_sync_entry_for_xxx` functions as we see fit. Plus caching doesn't really have anything to do with the config of sync. Split from https://github.com/element-hq/synapse/pull/17167 Spawning from https://github.com/element-hq/synapse/pull/17167#discussion_r1601497279
* Revert "Improve perf of sync device lists" (#17207)Erik Johnston2024-05-161-6/+31
| | | Reverts element-hq/synapse#17191
* Fix bug where push rules would be empty in `/sync` (#17142)Erik Johnston2024-05-161-12/+8
| | | | | | Fixes #16987 Some old accounts seem to have an entry in global account data table for push rules, which we should ignore
* Refactor Sync handler to be able to return different sync responses ↵Eric Eastwood2024-05-161-7/+58
| | | | | | | | | | | | | | (`SyncVersion`) (#17200) Refactor Sync handler to be able to be able to return different sync responses (`SyncVersion`). Preparation to be able support sync v2 and a new Sliding Sync `/sync/e2ee` endpoint which returns a subset of sync v2. Split upon request: https://github.com/element-hq/synapse/pull/17167#discussion_r1601497279 Split from https://github.com/element-hq/synapse/pull/17167 where we will add `SyncVersion.E2EE_SYNC` and a new type of sync response.
* Improve perf of sync device lists (#17191)Erik Johnston2024-05-141-31/+6
| | | | | It's almost always more efficient to query the rooms that have device list changes, rather than looking at the list of all users whose devices have changed and then look for shared rooms.
* Allows CAS SSO flow to provide user IDs composed of numbers only (#17098)Aurélien Grimpard2024-05-141-0/+5
|
* Fix undiscovered linter errors (#17166)devonh2024-05-081-3/+11
| | | | | | Linter errors are showing up in #17147 that are unrelated to that PR. The errors do not currently show up on develop. This PR aims to resolve the linter errors separately from #17147.
* Add support for MSC3823 - Account Suspension (#17051)Shay2024-05-011-0/+30
|
* Apply user `email` & `picture` during OIDC registration if present & ↵devonh2024-04-291-0/+10
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | selected (#17120) This change will apply the `email` & `picture` provided by OIDC to the new user account when registering a new user via OIDC. If the user is directed to the account details form, this change makes sure they have been selected before applying them, otherwise they are omitted. In particular, this change ensures the values are carried through when Synapse has consent configured, and the redirect to the consent form/s are followed. I have tested everything manually. Including: - with/without consent configured - allowing/not allowing the use of email/avatar (via `sso_auth_account_details.html`) - with/without automatic account detail population (by un/commenting the `localpart_template` option in synapse config). ### Pull Request Checklist <!-- Please read https://element-hq.github.io/synapse/latest/development/contributing_guide.html before submitting your pull request --> * [X] Pull request is based on the develop branch * [X] Pull request includes a [changelog file](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#changelog). The entry should: - Be a short description of your change which makes sense to users. "Fixed a bug that prevented receiving messages from other servers." instead of "Moved X method from `EventStore` to `EventWorkerStore`.". - Use markdown where necessary, mostly for `code blocks`. - End with either a period (.) or an exclamation mark (!). - Start with a capital letter. - Feel free to credit yourself, by adding a sentence "Contributed by @github_username." or "Contributed by [Your Name]." to the end of the entry. * [X] [Code style](https://element-hq.github.io/synapse/latest/code_style.html) is correct (run the [linters](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#run-the-linters))
* Add support for MSC4115 (#17104)Richard van der Hoff2024-04-298-7/+40
| | | | Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
* Ensure that incoming to-device messages are not dropped (#17127)Richard van der Hoff2024-04-291-0/+3
| | | | | | | | | | | | | | | | | | | | ... when workers are unreachable, etc. Fixes https://github.com/element-hq/synapse/issues/17117. The general principle is just to make sure that we propagate any exceptions to the JsonResource, so that we return an error code to the sending server. That means that the sending server no longer considers the message safely sent, so it will retry later. In the issue, Erik mentions that an alternative solution would be to persist the to-device messages into a table so that they can be retried. This might be an improvement for performance, but even if we did that, we still need this mechanism, since we might be unable to reach the database. So, if we want to do that, it can be a later follow-up. --------- Co-authored-by: Erik Johnston <erik@matrix.org>
* Redact membership events if the user requested erasure upon deactivating ↵Till2024-04-251-1/+12
| | | | | (#17076) Fixes #15355 by redacting all membership events before leaving rooms.
* bugfix: make msc3967 idempotent (#16943)Kegan Dougal2024-04-151-0/+36
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | MSC3967 was updated recently to make it more robust to network failures: > there is an existing cross-signing master key and it exactly matches the cross-signing master key provided in the request body. If there are any additional keys provided in the request (self signing key, user signing key) they MUST also match the existing keys stored on the server. In other words, the request contains no new keys. If there are new keys, UIA MUST be performed. https://github.com/matrix-org/matrix-spec-proposals/blob/hughns/device-signing-upload-uia/proposals/3967-device-signing-upload-uia.md#proposal This covers the case where the 200 OK is lost in transit so the client retries the upload, only to then get UIA'd. Complement tests: https://github.com/matrix-org/complement/pull/713 - passing example https://github.com/element-hq/synapse/actions/runs/7976948122/job/21778795094?pr=16943#step:7:8820 ### Pull Request Checklist <!-- Please read https://element-hq.github.io/synapse/latest/development/contributing_guide.html before submitting your pull request --> * [x] Pull request is based on the develop branch * [x] Pull request includes a [changelog file](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#changelog). The entry should: - Be a short description of your change which makes sense to users. "Fixed a bug that prevented receiving messages from other servers." instead of "Moved X method from `EventStore` to `EventWorkerStore`.". - Use markdown where necessary, mostly for `code blocks`. - End with either a period (.) or an exclamation mark (!). - Start with a capital letter. - Feel free to credit yourself, by adding a sentence "Contributed by @github_username." or "Contributed by [Your Name]." to the end of the entry. * [x] [Code style](https://element-hq.github.io/synapse/latest/code_style.html) is correct (run the [linters](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#run-the-linters)) --------- Co-authored-by: reivilibre <oliverw@matrix.org>
* Also check if first event matches the last in prev batch (#17066)Erik Johnston2024-04-091-7/+13
| | | | | Refinement of #17064 cc @richvdh
* Fix PR #16677, a parameter was missing in a function call (#17033)Mathieu Velten2024-04-091-0/+1
| | | | Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
* Fixups to #17064 (#17065)Erik Johnston2024-04-081-0/+3
| | | | | Forget a line, and an empty batch is trivially linear. c.f. #17064
* Add back fast path for non-gappy syncs (#17064)Erik Johnston2024-04-081-0/+36
| | | | | | | | | | | PR #16942 removed an invalid optimisation that avoided pulling out state for non-gappy syncs. This causes a large increase in DB usage. c.f. #16941 for why that optimisation was wrong. However, we can still optimise in the simple case where the events in the timeline are a linear chain without any branching/merging of the DAG. cc. @richvdh
* Fix bug in calculating state for non-gappy syncs (#16942)Richard van der Hoff2024-04-041-54/+37
| | | | | | | | | Unfortunately, the optimisation we applied here for non-gappy syncs is not actually valid. Fixes https://github.com/element-hq/synapse/issues/16941. ~~Based on https://github.com/element-hq/synapse/pull/16930.~~ Requires https://github.com/matrix-org/sytest/pull/1374.
* `/sync`: fix bug in calculating `state` response (#16930)Richard van der Hoff2024-04-041-41/+13
| | | | | | | Fix a long-standing issue which could cause state to be omitted from the sync response if the last event was filtered out. Fixes: https://github.com/element-hq/synapse/issues/16928
* Fix bug in `/sync` response for archived rooms (#16932)Richard van der Hoff2024-04-041-14/+107
| | | | | | | | | | | | This PR fixes a very, very niche edge-case, but I've got some more work coming which will otherwise make the problem worse. The bug happens when the syncing user leaves a room, and has a sync filter which includes "left" rooms, but sets the timeline limit to 0. In that case, the state returned in the `state` section is calculated incorrectly. The fix is to pass a token corresponding to the point that the user leaves the room through to `compute_state_delta`.
* Fixups to new push stream (#17038)Erik Johnston2024-03-281-2/+4
| | | Follow on from #17037
* Add support for moving `/push_rules` off of main process (#17037)Erik Johnston2024-03-281-3/+16
|
* Add OIDC config to add extra parameters to the authorize URL (#16971)Mathieu Velten2024-03-221-6/+14
|
* Do not refuse to set read_marker if previous event_id is in wrong room (#16990)SpiritCroc2024-03-211-2/+2
|
* Fix reject knocks on deactivating account (#17010)Hanadi2024-03-211-10/+15
|
* OIDC: try to JWT decode userinfo response if JSON parsing failed (#16972)Mathieu Velten2024-03-211-4/+28
|
* Update power level default for public rooms (#16907)Shay2024-03-191-1/+1
|
* Improve event validation (#16908)Shay2024-03-192-1/+24
| | | As the title states.
* Pass module API to OIDC mapping provider (#16974)Mathieu Velten2024-03-191-3/+14
| | | | As done for SAML mapping provider, let's pass the module API to the OIDC one so the mapper can do more logic in its code.
* `/sync`: Fix edge-case in calculating the "device_lists" response (#16949)Richard van der Hoff2024-03-141-2/+9
| | | | | Fixes https://github.com/element-hq/synapse/issues/16948. If the `join` and the `leave` are in the same sync response, we need to count them as a "left" user.
* Split up `SyncHandler.compute_state_delta` (#16929)Richard van der Hoff2024-03-141-145/+236
| | | | | | This is a huge method, which melts my brain. This is a non-functional change which lays some groundwork for future work in this area.
* Improve lock performance when a lot of locks are waiting (#16840)Mathieu Velten2024-03-141-6/+9
| | | | | | | | | | | | When a lot of locks are waiting for a single lock, notifying all locks independently with `call_later` on each release is really costly and incurs some kind of async contention, where the CPU is spinning a lot for not much. The included test is taking around 30s before the change, and 0.5s after. It was found following failing tests with https://github.com/element-hq/synapse/pull/16827.
* Bump mypy from 1.5.1 to 1.8.0 (#16901)dependabot[bot]2024-03-132-3/+3
|
* Bump black from 23.10.1 to 24.2.0 (#16936)dependabot[bot]2024-03-1311-49/+51
|
* Don't lock up when joining large rooms (#16903)Erik Johnston2024-02-201-9/+17
| | | | Co-authored-by: Andrew Morgan <andrew@amorgan.xyz>
* Add a config to not send out device list updates for specific users (#16909)Erik Johnston2024-02-131-2/+11
| | | | | | | | | List of users not to send out device list updates for when they register new devices. This is useful to handle bot accounts. This is undocumented as its mostly a hack to test on matrix.org. Note: This will still send out device list updates if the device is later updated, e.g. end to end keys are added.
* Correctly mention previous copyright (#16820)Erik Johnston2024-01-2347-0/+60
| | | | | During the migration the automated script to update the copyright headers accidentally got rid of some of the existing copyright lines. Reinstate them.
* Allow room creation but not publishing to continue if room publication rules ↵Shay2024-01-221-4/+2
| | | | | | | | | | | | | are violated when creating a new room. (#16811) Prior to this PR, if a request to create a public (public as in published to the rooms directory) room violated the room list publication rules set in the [config](https://matrix-org.github.io/synapse/latest/usage/configuration/config_documentation.html#room_list_publication_rules), the request to create the room was denied and the room was not created. This PR changes the behavior such that when a request to create a room published to the directory violates room list publication rules, the room is still created but the room is not published to the directory.
* Faster load recents for sync (#16783)Erik Johnston2024-01-101-6/+8
| | | This hopefully reduces the amount of state we need to keep in memory
* Pull less state out if we fail to backfill (#16788)Erik Johnston2024-01-101-9/+12
| | | | | | | | | | | Sometimes we fail to fetch events during backfill due to missing state, and we often end up querying the same bad events periodically (as people backpaginate). In such cases its likely we will continue to fail to get the state, and therefore we should try *before* loading the state that we have from the DB (as otherwise it's wasted DB and memory). --------- Co-authored-by: reivilibre <oliverw@matrix.org>
* Reduce amount of state pulled out when querying federation hierachy (#16785)Erik Johnston2024-01-101-1/+11
| | | | | | | | | | | There are two changes here: 1. Only pull out the required state when handling the request. 2. Change the get filtered state return type to check that we're only querying state that was requested --------- Co-authored-by: reivilibre <oliverw@matrix.org>
* Faster partial join to room with complex auth graph (#7)Erik Johnston2024-01-101-49/+30
| | | | | | | | Instead of persisting outliers in a bunch of batches, let's just do them all at once. This is fine because all `_auth_and_persist_outliers_inner` is doing is checking the auth rules for each event, which requires the events to be topologically sorted by the auth graph.
* Filter out rooms from the room directory being served to other homeservers ↵reivilibre2024-01-081-50/+127
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | when those rooms block that homeserver by their Access Control Lists. (#16759) The idea here being that the directory server shouldn't advertise rooms to a requesting server is the requesting server would not be allowed to join or participate in the room. <!-- Fixes: # <!-- --> <!-- Supersedes: # <!-- --> <!-- Follows: # <!-- --> <!-- Part of: # <!-- --> Base: `develop` <!-- git-stack-base-branch:develop --> <!-- This pull request is commit-by-commit review friendly. <!-- --> <!-- This pull request is intended for commit-by-commit review. <!-- --> Original commit schedule, with full messages: <ol> <li> Pass `from_federation_origin` down into room list retrieval code </li> <li> Don't cache /publicRooms response for inbound federated requests </li> <li> fixup! Don't cache /publicRooms response for inbound federated requests </li> <li> Cap the number of /publicRooms entries to 100 </li> <li> Simplify code now that you can't request unlimited rooms </li> <li> Filter out rooms from federated requests that don't have the correct ACL </li> <li> Request a handful more when filtering ACLs so that we can try to avoid shortchanging the requester </li> </ol> --------- Signed-off-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org>
* Fix linting (#16780)Erik Johnston2024-01-051-1/+11
| | | Introduced in #16762
* Simplify internal metadata class. (#16762)Erik Johnston2024-01-053-5/+22
| | | | | | | | | We remove these fields as they're just duplicating data the event already stores, and (for reasons :shushing_face:) I'd like to simplify the class to only store simple types. I'm not entirely convinced that we shouldn't instead add helper methods to the event class to generate stream tokens, but I don't really think that's where they belong either
* Add recursion_depth to /relations if recursing (#16775)David Baker2024-01-041-0/+4
| | | | | This is an extra response parameter just added to MSC3981. In the current impl, the recursion depth is always 3, so this just returns a static 3 if the recurse parameter is supplied.
* Merge remote-tracking branch 'gitlab/clokep/license-license' into new_developErik Johnston2023-12-1350-522/+800
|\
| * Update license headersPatrick Cloke2023-11-2150-522/+800
| |
* | Fix upgrading a room without `events` field in power levels (#16725)David Robertson2023-12-051-1/+1
| |
* | Implement MSC4069: Inhibit profile propagation (#16636)Travis Ralston2023-12-041-2/+8
| | | | | | MSC: https://github.com/matrix-org/matrix-spec-proposals/pull/4069
* | ModuleAPI SSO auth callbacks (#15207)Andrew Yasinishyn2023-12-012-0/+24
| | | | | | Signed-off-by: Andrii Yasynyshyn yasinishyn.a.n@gmail.com
* | Drop unused tables & unneeded access token ID for events. (#16522)Patrick Cloke2023-12-011-6/+2
| |
* | Reduce DB load when forget on leave setting is disabled (#16668)Erik Johnston2023-11-291-3/+8
| | | | | | | | | | * Reduce DB load when forget on leave setting is disabled * Newsfile
* | Ignore `encryption_enabled_by_default_for_room_type` for notices room (#16677)Mathieu Velten2023-11-281-1/+7
|/
* Reduce task concurrency (#16656)Erik Johnston2023-11-171-1/+1
|
* Speed up deleting device messages (#16643)Erik Johnston2023-11-161-3/+5
| | | Keeping track of a lower bound of stream ID where we've deleted everything below makes the queries much faster. Otherwise, every time we scan for rows to delete we'd re-scan across all the rows that have previously deleted (until the next table VACUUM).
* Speed up persisting large number of outliers (#16649)Erik Johnston2023-11-161-11/+7
| | | Recalculating the roots tuple every iteration could be very expensive, so instead let's do a topological sort.
* Add an Admin API to temporarily grant the ability to update an existing ↵David Robertson2023-11-151-7/+13
| | | | cross-signing key without UIA (#16634)
* Use full GitHub links instead of bare issue numbers. (#16637)Patrick Cloke2023-11-154-8/+8
|
* Convert simple_select_one_txn and simple_select_one to return tuples. (#16612)Patrick Cloke2023-11-092-4/+5
|
* Return attrs for more media repo APIs. (#16611)Patrick Cloke2023-11-092-7/+10
|
* Remove remaining usage of cursor_to_dict. (#16564)Patrick Cloke2023-10-313-36/+35
|
* Merge pull request from GHSA-mp92-3jfm-3575Patrick Cloke2023-10-312-0/+9
|
* Claim local one-time-keys in bulk (#16565)David Robertson2023-10-301-0/+10
| | | | Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
* Claim fallback keys in bulk (#16570)David Robertson2023-10-301-0/+14
|
* Ensure local invited & knocking users leave before purge. (#16559)Patrick Cloke2023-10-271-3/+4
| | | | | This is mostly useful for federated rooms where some users would get stuck in the invite or knock state when the room was purged from their homeserver.
* Fix cross-worker ratelimiting (#16558)Erik Johnston2023-10-271-16/+57
| | | c.f. #16481
* Add new module API for adding custom fields to events `unsigned` section ↵Erik Johnston2023-10-276-18/+20
| | | | (#16549)
* Remove more usages of cursor_to_dict. (#16551)Patrick Cloke2023-10-262-12/+12
| | | Mostly to improve type safety.
* Add a new module API to update user presence state. (#16544)Patrick Cloke2023-10-263-33/+49
| | | | | | | | | | This adds a module API which allows a module to update a user's presence state/status message. This is useful for controlling presence from an external system. To fully control presence from the module the presence.enabled config parameter gains a new state of "untracked" which disables internal tracking of presence changes via user actions, etc. Only updates from the module will be persisted and sent down sync properly).
* Convert simple_select_list and simple_select_list_txn to return lists of ↵Patrick Cloke2023-10-262-6/+3
| | | | | tuples (#16505) This should use fewer allocations and improves type hints.
* Allow multiple workers to write to receipts stream. (#16432)Erik Johnston2023-10-254-30/+40
| | | Fixes #16417
* Mark sync as limited if there is a gap in the timeline (#16485)Erik Johnston2023-10-191-8/+44
| | | | | | | | This splits thinsg into two queries, but most of the time we won't have new event backwards extremities so this shouldn't actually add an extra RTT for the majority of cases. Note this removes the check for events with no prev events, but that was part of MSC2716 work that has since been removed.
* Improve performance of delete device messages query (#16492)Mathieu Velten2023-10-181-0/+2
|
* Convert DeviceLastConnectionInfo to attrs. (#16507)Patrick Cloke2023-10-171-16/+7
| | | To improve type safety & memory usage.
* Convert state delta processing from a dict to attrs. (#16469)Patrick Cloke2023-10-164-78/+73
| | | For improved type checking & memory usage.
* Remove useless async job to delete device messages on sync (#16491)Mathieu Velten2023-10-161-22/+0
|
* Revert "Drop unused tables & unneeded access token ID for events. (#16268)" ↵Patrick Cloke2023-10-121-2/+6
| | | | | | | | (#16465) This reverts commit cabd57746004fe2dacc11aa8d373854a3d25e306. There are additional usages of these tables which need to be removed first.
* Convert user_get_threepids response to attrs. (#16468)Patrick Cloke2023-10-113-5/+7
| | | This improves type annotations by not having a dictionary of Any values.
* Bump ruff from 0.0.290 to 0.0.292 (#16449)dependabot[bot]2023-10-095-8/+0
| | | | | | | | | | | | | | | | | | | | | | | | | * Bump ruff from 0.0.290 to 0.0.292 Bumps [ruff](https://github.com/astral-sh/ruff) from 0.0.290 to 0.0.292. - [Release notes](https://github.com/astral-sh/ruff/releases) - [Changelog](https://github.com/astral-sh/ruff/blob/main/BREAKING_CHANGES.md) - [Commits](https://github.com/astral-sh/ruff/compare/v0.0.290...v0.0.292) --- updated-dependencies: - dependency-name: ruff dependency-type: direct:development update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> * Fix up lint --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Erik Johnston <erik@matrix.org>
* Apply join rate limiter outside the lineariser (#16441)David Robertson2023-10-061-20/+23
|
* Drop unused tables & unneeded access token ID for events. (#16268)Patrick Cloke2023-10-061-6/+2
| | | | Drop the event_txn_id table and the tables related to MSC2716, which is no longer supported in Synapse.
* Stop sending incorrect knock_state_events. (#16403)Patrick Cloke2023-10-061-11/+2
| | | | | | | | | Synapse was incorrectly implemented with a knock_state_events property on some APIs (instead of knock_room_state). This was correct in Synapse 1.70.0, but *both* fields were sent to also be compatible with Synapse versions expecting the wrong field. Enough time has passed that only the correct field needs to be included/handled.
* Factor out `MultiWriter` token from `RoomStreamToken` (#16427)Erik Johnston2023-10-054-6/+5
|
* Some refactors around receipts stream (#16426)Erik Johnston2023-10-043-19/+16
|
* mypy plugin to check `@cached` return types (#14911)David Robertson2023-10-021-2/+2
| | | | | | | | Co-authored-by: David Robertson <davidr@element.io> Co-authored-by: Patrick Cloke <patrickc@matrix.org> Co-authored-by: Erik Johnston <erik@matrix.org> Assert that the return type of callables wrapped in @cached and @cachedList are cachable (aka immutable).
* Add a cache around server ACL checking (#16360)Patrick Cloke2023-09-262-0/+11
| | | | | * Pre-compiles the server ACLs onto an object per room and invalidates them when new events come in. * Converts the server ACL checking into Rust.
* Reduce calls to `send_presence_to_destinations` (#16385)Erik Johnston2023-09-261-15/+18
|
* Convert more cached return values to immutable types (#16356)Patrick Cloke2023-09-202-3/+13
|
* Return immutable objects for cachedList decorators (#16350)Patrick Cloke2023-09-196-33/+37
|
* Make cached account data/tags/admin types immutable (#16325)Patrick Cloke2023-09-182-20/+25
|
* Return an immutable value from get_latest_event_ids_in_room. (#16326)Patrick Cloke2023-09-181-5/+3
|
* Additional validation of receipts (#16327)Patrick Cloke2023-09-181-2/+24
| | | | Reject invalid receipts with a reasonable error message & expands tests for receipts.
* Set email charset as utf-8 rather than utf8 (#16329)José Joaquín Atria2023-09-181-2/+2
|
* Add automatic purge after all users forget a room (#15488)Mathieu Velten2023-09-153-390/+281
| | | | | | Also add restore of purge/shutdown rooms after a synapse restart. Co-authored-by: Eric Eastwood <erice@matrix.org> Co-authored-by: Erik Johnston <erikj@matrix.org>
* Speed up deleting to-device messages task (#16318)Erik Johnston2023-09-141-13/+14
|
* Refactor `get_user_by_id` (#16316)Erik Johnston2023-09-143-31/+26
|
* Fix using dehydrated devices (MSC2697) & refresh tokens (#16288)Hanadi2023-09-131-3/+4
| | | | | Refresh tokens were not correctly moved to the rehydrated device (similar to how the access token is currently handled). This resulted in invalid refresh tokens after rehydration.
* Don't schedule an async task on every sync (#16312)Erik Johnston2023-09-131-11/+26
|
* Fix deleting device inbox when using background worker (#16311)Erik Johnston2023-09-131-31/+31
| | | | | Introduced in #16240 The action for the task was only defined on the "master" handler, rather than the base worker one.
* Add the List-Unsubscribe header for notification emails. (#16274)Patrick Cloke2023-09-111-1/+9
| | | | | | | | Adds both the List-Unsubscribe (RFC2369) and List-Unsubscribe-Post (RFC8058) headers to push notification emails, which together should: * Show an "Unsubscribe" link in the MUA UI when viewing Synapse notification emails. * Enable "one-click" unsubscribe (the user never leaves their MUA, which automatically makes a POST request to the specified endpoint).
* Filter out down hosts when retrying fetching device lists (#16298)Erik Johnston2023-09-111-1/+14
|
* Recheck if remote device is cached before requesting it (#16252)Erik Johnston2023-09-071-6/+15
| | | | This fixes a bug where we could get stuck re-requesting the device over replication again and again.
* Handle "registration_enabled" parameter for CAS (#16262)Aurélien Grimpard2023-09-061-0/+2
| | | | Similar to OIDC, CAS providers can now disable registration such that only existing users are able to login via SSO.
* Allow modules to delete rooms. (#15997)reivilibre2023-09-062-3/+19
| | | | | | | | | | | | | | | * Allow user_id to be optional for room deletion * Add module API method to delete a room * Newsfile Signed-off-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org> * Don't worry about the case block=True && requester_user_id is None --------- Signed-off-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org>
* Delete device messages asynchronously and in staged batches (#16240)Mathieu Velten2023-09-063-6/+62
|
* Time out busy presence status & test multi-device busy (#16174)Patrick Cloke2023-09-051-1/+18
| | | | | | | Add a (long) timeout to when a "busy" device is considered not online. This does *not* match MSC3026, but is a reasonable thing for an implementation to do. Expands tests for the (unstable) busy presence with multiple devices.
* Track presence state per-device and combine to a user state. (#16066)Patrick Cloke2023-09-051-47/+232
| | | | | | | | | | | Tracks presence on an individual per-device basis and combine the per-device state into a per-user state. This should help in situations where a user has multiple devices with conflicting status (e.g. one is syncing with unavailable and one is syncing with online). The tie-breaking is done by priority: BUSY > ONLINE > UNAVAILABLE > OFFLINE
* Bump twisted from 22.10.0 to 23.8.0 (#16235)dependabot[bot]2023-09-051-6/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Bump twisted from 22.10.0 to 23.8.0 Bumps [twisted](https://github.com/twisted/twisted) from 22.10.0 to 23.8.0. - [Release notes](https://github.com/twisted/twisted/releases) - [Changelog](https://github.com/twisted/twisted/blob/trunk/NEWS.rst) - [Commits](https://github.com/twisted/twisted/compare/twisted-22.10.0...twisted-23.8.0) --- updated-dependencies: - dependency-name: twisted dependency-type: direct:production update-type: version-update:semver-major ... Signed-off-by: dependabot[bot] <support@github.com> * Fix types * Fix lint * Newsfile --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Erik Johnston <erik@matrix.org>
* Don't wake up destination transaction queue if they're not due for retry. ↵Erik Johnston2023-09-044-27/+36
| | | | (#16223)
* Add last_seen_ts to the admin users API (#16218)Mathieu Velten2023-09-041-0/+1
|
* Stop writing to the event_txn_id table (#16175)Patrick Cloke2023-08-301-13/+0
|
* Describe which rate limiter was hit in logs (#16135)David Robertson2023-08-305-26/+17
|
* Track currently syncing users by device for presence (#16172)Patrick Cloke2023-08-291-55/+100
| | | | | | | Refactoring to use both the user ID & the device ID when tracking the currently syncing users in the presence handler. This is done both locally and over replication. Note that the device ID is discarded but will be used in a future change.
* Bump ruff from 0.0.277 to 0.0.286 (#16198)dependabot[bot]2023-08-291-1/+1
| | | | Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* Support IPv6-only SMTP servers (#16155)Evilham2023-08-291-17/+11
| | | | Use Twisted HostnameEndpoint to connect to SMTP servers (instead of connectTCP/connectSSL) which properly supports IPv6-only servers.
* Pass the device ID around in the presence handler (#16171)Patrick Cloke2023-08-283-11/+45
| | | | | | Refactoring to pass the device ID (in addition to the user ID) through the presence handler (specifically the `user_syncing`, `set_state`, and `bump_presence_active_time` methods and their replication versions).
* Combine logic about not overriding BUSY presence. (#16170)Patrick Cloke2023-08-281-92/+63
| | | | | | | | | | | | | Simplify some of the presence code by reducing duplicated code between worker & non-worker modes. The main change is to push some of the logic from `user_syncing` into `set_state`. This is done by passing whether the user is setting the presence via a `/sync` with a new `is_sync` flag to `set_state`. If this is `true` some additional logic is performed: * Don't override `busy` presence. * Update the `last_user_sync_ts`. * Never update the status message.
* Add configuration setting for CAS protocol version (#15816)Aurélien Grimpard2023-08-241-1/+5
|
* Fix typechecking with twisted trunk (#16121)David Robertson2023-08-241-16/+16
|
* Only lock when we're backfilling (#16159)Erik Johnston2023-08-232-146/+156
|
* Disable caching in /sync corner case (#16080)Matthew Ma2023-08-171-10/+10
| | | Fixes #15502
* Rename pagination&purge locks and add comments explaining them (#16112)Mathieu Velten2023-08-164-13/+22
|
* Run pyupgrade for python 3.7 & 3.8. (#16110)Patrick Cloke2023-08-154-11/+16
|
* Allow customizing IdP name and icon for SAML and CAS (#16094)Gabriel Rodríguez2023-08-112-10/+12
|
* Clean-up presence code (#16092)Patrick Cloke2023-08-101-94/+75
| | | | | | | | Misc. clean-ups to: * Use keyword arguments. * Return early (reducing indentation) of some functions. * Removing duplicated / unused code. * Use wrap_as_background_process.
* Implements admin API to lock an user (MSC3939) (#15870)Mathieu Velten2023-08-102-1/+5
|
* Support MSC3814: Dehydrated Devices Part 2 (#16010)Shay2023-08-082-16/+11
|
* Fix deletion for Dehydrated Devices (#16046)Shay2023-08-041-0/+16
|
* Stabilize support for MSC3970: updated transaction semantics (scope to ↵Patrick Cloke2023-08-041-7/+5
| | | | | | | `device_id`) (#15629) For now this maintains compatible with old Synapses by falling back to using transaction semantics on a per-access token. A future version of Synapse will drop support for this.
* Trim whitespace when setting display names (#16031)Mohit Rathee2023-08-011-1/+1
|
* Combine duplicated code for calculating an event ID from a txn ID (#16023)Patrick Cloke2023-07-312-32/+35
| | | | Refactoring related to stabilization of MSC3970, refactor to combine code which has the same logic.
* Add ability to wait for locks and add locks to purge history / room deletion ↵Erik Johnston2023-07-314-25/+414
| | | | | (#15791) c.f. #13476
* Fix 404 on /profile when the display name is empty but not the avatar (#16012)Mathieu Velten2023-07-271-1/+1
|
* Support MSC3814: Dehydrated Devices (#15929)Shay2023-07-242-3/+109
| | | | | | Signed-off-by: Nicolas Werner <n.werner@famedly.com> Co-authored-by: Nicolas Werner <n.werner@famedly.com> Co-authored-by: Nicolas Werner <89468146+nico-famedly@users.noreply.github.com> Co-authored-by: Hubert Chathi <hubert@uhoreg.ca>
* Resync stale devices in background (#15975)Erik Johnston2023-07-241-1/+8
| | | | This is so we don't block responding to federation transaction while we try and fetch the device lists.
* Speed up updating state in large rooms (#15971)Erik Johnston2023-07-201-5/+4
| | | This should speed up updating state in rooms with lots of state.
* Reduce the amount of state we pull out (#15968)Erik Johnston2023-07-203-12/+13
|
* Support room version 11 (#15912)Patrick Cloke2023-07-184-6/+6
| | | | | And fix a bug in the implementation of the updated redaction format (MSC2174) where the top-level redacts field was not properly added for backwards-compatibility.
* Add a locality to a few presence metrics (#15952)Jason Little2023-07-181-13/+24
|
* Better warning in logs when we fail to fetch an alias (#15922)Eric Eastwood2023-07-111-1/+3
| | | | | | | | | | | | | **Before:** ``` Error retrieving alias ``` **After:** ``` Error retrieving alias #foo:bar -> 401 Unauthorized ``` *Spawning from creating the [manual testing strategy for the outbound federation proxy](https://github.com/matrix-org/synapse/pull/15773).*
* Add + as an allowed character for Matrix IDs (MSC4009) (#15911)Patrick Cloke2023-07-113-13/+6
|
* Remove experimental MSC2716 implementation to incrementally import history ↵Eric Eastwood2023-06-165-796/+34
| | | | | | | | | | | | into existing rooms (#15748) Context for why we're removing the implementation: - https://github.com/matrix-org/matrix-spec-proposals/pull/2716#issuecomment-1487441010 - https://github.com/matrix-org/matrix-spec-proposals/pull/2716#issuecomment-1504262734 Anyone wanting to continue MSC2716, should also address these leftover tasks: https://github.com/matrix-org/synapse/issues/10737 Closes https://github.com/matrix-org/synapse/issues/10737 in the fact that it is not longer necessary to track those things.
* Fix joining rooms through aliases where the alias server isn't a real ↵Tulir Asokan2023-06-141-1/+1
| | | | homeserver (#15776)
* Avoid backfill when we already have messages to return (#15737)Eric Eastwood2023-06-131-22/+115
| | | | | We now only block the client to backfill when we see a large gap in the events (more than 2 events missing in a row according to `depth`), more than 3 single-event holes, or not enough messages to fill the response. Otherwise, we return the messages directly to the client and backfill in the background for eventual consistency sake. Fix https://github.com/matrix-org/synapse/issues/15696
* Backfill in the background if we're doing it "just because" (#15710)Eric Eastwood2023-06-091-4/+14
| | | Fix https://github.com/matrix-org/synapse/issues/15702
* Check required power levels earlier in createRoom handler. (#15695)Grant McLean2023-06-071-14/+62
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Check required power levels earlier in createRoom handler. - If a server was configured to reject the creation of rooms with E2EE enabled (by specifying an unattainably high power level for "m.room.encryption" in default_power_level_content_override), the 403 error was not being triggered until after the room was created and before the "m.room.power_levels" was sent. This allowed a user to access the partially-configured room and complete the setup of E2EE and power levels manually. - This change causes the power level overrides to be checked earlier and the request to be rejected before the user gains access to the room. - A new `_validate_room_config` method is added to contain checks that should be run before a room is created. - The new test case confirms that a user request is rejected by the new validation method. Signed-off-by: Grant McLean <grant@catalyst.net.nz> * Add a changelog file. * Formatting fix for black. * Remove unneeded line from test. --------- Signed-off-by: Grant McLean <grant@catalyst.net.nz>
* Remove some unused `server_name` fields (#15723)Sean Quah2023-06-064-4/+0
| | | Signed-off-by: Sean Quah <seanq@matrix.org>
* Some house keeping on `maybe_backfill()` functions (#15709)Eric Eastwood2023-06-051-0/+17
|
* `N + 3`: Read from column `full_user_id` rather than `user_id` of tables ↵Shay2023-06-026-22/+14
| | | | `profiles` and `user_filters` (#15649)
* Add a catch-all * to the supported relation types when redacting (#15705)Mathieu Velten2023-06-021-5/+11
| | | This is an update to MSC3912 implementation
* Add stricter mypy options (#15694)Patrick Cloke2023-05-312-3/+3
| | | | Enable warn_unused_configs, strict_concatenate, disallow_subclassing_any, and disallow_incomplete_defs.
* Refactor config to be an experimental featureHugh Nimmo-Smith2023-05-301-2/+2
| | | | Also enforce you can't combine it with incompatible config options
* Disable account related endpoints when using OAuth delegationQuentin Gliech2023-05-301-1/+7
|
* Process previously failed backfill events in the background (#15585)Eric Eastwood2023-05-241-8/+62
| | | | | | | | | Process previously failed backfill events in the background because they are bound to fail again and we don't need to waste time holding up the request for something that is bound to fail again. Fix https://github.com/matrix-org/synapse/issues/13623 Follow-up to https://github.com/matrix-org/synapse/issues/13621 and https://github.com/matrix-org/synapse/issues/13622 Part of making `/messages` faster: https://github.com/matrix-org/synapse/issues/13356
* Add requesting user id parameter to key claim methods in ↵Shay2023-05-241-1/+2
| | | | `TransportLayerClient` (#15663)
* Consolidate logic to check for deactivated users. (#15634)Patrick Cloke2023-05-232-25/+8
| | | | | | | This moves the deactivated user check to the method which all login types call. Additionally updates the application service tests to be more realistic by removing invalid tests and fixing server names.
* Trace how many new events from the backfill response we need to process (#15633)Eric Eastwood2023-05-191-0/+5
| | | You can kinda derive this information from how many `_process_pulled_event` spans there are but it would be nice to quickly glance.
* Rename blacklist/whitelist internally. (#15620)Patrick Cloke2023-05-193-12/+10
| | | | Avoid renaming configuration settings for now and rename internal code to use blocklist and allowlist instead.
* Do not allow deactivated users to login with JWT. (#15624)Patrick Cloke2023-05-191-0/+118
| | | | | To improve the organization of this code it moves the JWT login checks to a separate handler and then fixes the bug (and a deprecation warning).
* Handle missing previous read marker event. (#15464)Nick Mills-Barrett2023-05-181-4/+14
| | | | | If the previous read marker is pointing to an event that no longer exists (e.g. due to retention) then assume that the newly given read marker is newer.
* Error if attempting to set m.push_rules account data, per MSC4010. (#15555)Patrick Cloke2023-05-092-11/+17
| | | | | m.push_rules, like m.fully_read, is a special account data type that cannot be set using the normal /account_data endpoint. Return an error instead of allowing data that will not be used to be stored.
* Use account data constants in more places. (#15554)Patrick Cloke2023-05-092-8/+9
|
* Check appservices for devices during a /user/devices query. (#15539)Patrick Cloke2023-05-051-0/+28
| | | | | | | | MSC3984 proxies /keys/query requests to appservices, but servers will can also requests devices / keys from the /user/devices endpoint. The formats are close enough that we can "proxy" that /user/devices to appservices (by calling /keys/query) and then change the format of the returned data before returning it over federation.
* Implement MSC4009 to widen the allowed Matrix ID grammar (#15536)Patrick Cloke2023-05-052-15/+18
| | | | | Behind a configuration flag this adds + to the list of allowed characters in Matrix IDs. The main feature this enables is using full E.164 phone numbers as Matrix IDs.
* Clean up and clarify "Create or modify Account" Admin API documentation (#15544)Andrew Morgan2023-05-051-2/+2
|
* Factor out an `is_mine_server_name` method (#15542)Sean Quah2023-05-056-8/+13
| | | | | | | | | | | | Add an `is_mine_server_name` method, similar to `is_mine_id`. Ideally we would use this consistently, instead of sometimes comparing against `hs.hostname` and other times reaching into `hs.config.server.server_name`. Also fix a bug in the tests where `hs.hostname` would sometimes differ from `hs.config.server.server_name`. Signed-off-by: Sean Quah <seanq@matrix.org>
* Move ThirdPartyEventRules into module_api/callbacks (#15535)Andrew Morgan2023-05-049-19/+28
|
* Update the base rules to remove the dont_notify action. (MSC3987) (#15534)Patrick Cloke2023-05-041-0/+2
| | | | | A dont_notify action is a no-op (and coalesce is undefined). These are both considered no-ops by the spec, per MSC3987 and the predefined push rules were updated to remove dont_notify from the list of actions.
* Add config option to forget rooms automatically when users leave them (#15224)Sean Quah2023-05-032-22/+154
| | | | | This is largely based off the stats and user directory updater code. Signed-off-by: Sean Quah <seanq@matrix.org>
* Initial implementation of MSC3981: recursive relations API (#15315)Patrick Cloke2023-05-021-0/+3
| | | | | | | | | | | Adds an optional keyword argument to the /relations API which will recurse a limited number of event relationships. This will cause the API to return not just the events related to the parent event, but also events related to those related to the parent event, etc. This is disabled by default behind an experimental configuration flag and is currently implemented using prefixed parameters.
* Add support for claiming multiple OTKs at once. (#15468)Patrick Cloke2023-04-272-16/+29
| | | | | | | MSC3983 provides a way to request multiple OTKs at once from appservices, this extends this concept to the Client-Server API. Note that this will likely be spit out into a separate MSC, but is currently part of MSC3983.
* Add column `full_user_id` to tables `profiles` and `user_filters`. (#15458)Shay2023-04-261-6/+2
|
* Add unstable /keys/claim endpoint which always returns fallback keys. (#15462)Patrick Cloke2023-04-252-15/+68
| | | | | | | | | | | | | It can be useful to always return the fallback key when attempting to claim keys. This adds an unstable endpoint for `/keys/claim` which always returns fallback keys in addition to one-time-keys. The fallback key(s) are not marked as "used" unless there are no corresponding OTKs. This is currently defined in MSC3983 (although likely to be split out to a separate MSC). The endpoint shape may change or be requested differently (i.e. a keyword parameter on the current endpoint), but the core logic should be reasonable.
* Experimental support for MSC3970: per-device transaction IDs (#15318)Quentin Gliech2023-04-252-12/+59
|
* Remove legacy code of single user device resync api (#15418)Alok Kumar Singh2023-04-213-64/+22
| | | | | * Removed single-user resync usage and updated it to use multi-user counterpart Signed-off-by: Alok Kumar Singh alokaks601@gmail.com
* Move Spam Checker callbacks to a dedicated file (#15453)Andrew Morgan2023-04-187-26/+44
|
* Delete pushers after calling on_logged_out module hook on device delete (#15410)Mathieu Velten2023-04-141-2/+4
|
* Implement MSC2175: remove the creator field from create events. (#15394)Patrick Cloke2023-04-063-12/+22
|
* Don't keep old stream_ordering_to_exterm around (#15382)Erik Johnston2023-04-061-0/+10
|
* Delete server-side backup keys when deactivating an account. (#15181)Shay2023-04-041-0/+3
|
* Fix a rare bug where initial /syncs would fail (#15383)Sean Quah2023-04-041-5/+19
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This change fixes a rare bug where initial /syncs would fail with a `KeyError` under the following circumstances: 1. A user fast joins a remote room. 2. The user is kicked from the room before the room's full state has been synced. 3. A second local user fast joins the room. 4. Events are backfilled into the room with a higher topological ordering than the original user's leave. They are assigned a negative stream ordering. It's not clear how backfill happened here, since it is expected to be equivalent to syncing the full state. 5. The second local user leaves the room before the room's full state has been synced. The homeserver does not complete the sync. 6. The original user performs an initial /sync with lazy_load_members enabled. * Because they were kicked from the room, the room is included in the /sync response even though the include_leave option is not specified. * To populate the room's timeline, `_load_filtered_recents` / `get_recent_events_for_room` fetches events with a lower stream ordering than the leave event and picks the ones with the highest topological orderings (which are most recent). This captures the backfilled events after the leave, since they have a negative stream ordering. These events are filtered out of the timeline, since the user was not in the room at the time and cannot view them. The sync code ends up with an empty timeline for the room that notably does not include the user's leave event. This seems buggy, but at least we don't disclose events the user isn't allowed to see. * Normally, `compute_state_delta` would fetch the state at the start and end of the room's timeline to generate the sync response. Since the timeline is empty, it fetches the state at `min(now, last event in the room)`, which corresponds with the second user's leave. The state during the entirety of the second user's membership does not include the membership for the first user because of partial state. This part is also questionable, since we are fetching state from outside the bounds of the user's membership. * `compute_state_delta` then tries and fails to find the user's membership in the auth events of timeline events. Because there is no timeline event whose auth events are expected to contain the user's membership, a `KeyError` is raised. Also contains a drive-by fix for a separate unlikely race condition. Signed-off-by: Sean Quah <seanq@matrix.org>
* Revert pruning of old devices (#15360)Erik Johnston2023-03-312-51/+3
| | | | | | | | | * Revert "Fix registering a device on an account with lots of devices (#15348)" This reverts commit f0d8f66eaaacfa75bed65bc5d0c602fbc5339c85. * Revert "Delete stale non-e2e devices for users, take 3 (#15183)" This reverts commit 78cdb72cd6b0e007c314d9fed9f629dfc5b937a6.
* Implement MSC3984 to proxy /keys/query requests to appservices. (#15321)Patrick Cloke2023-03-302-0/+77
| | | | | If enabled, for users which are exclusively owned by an application service then the appservice will be queried for devices in addition to any information stored in the Synapse database.
* Fix spinloop during partial state sync when a prev event is in backoff (#15351)Sean Quah2023-03-302-25/+35
| | | | | | | | Previously, we would spin in a tight loop until `update_state_for_partial_state_event` stopped raising `FederationPullAttemptBackoffError`s. Replace the spinloop with a wait until the backoff period has expired. Signed-off-by: Sean Quah <seanq@matrix.org>
* Add the ability to enable/disable registrations when in the OIDC flow (#14978)Warren Bailey2023-03-302-2/+16
| | | Signed-off-by: Warren Bailey <warren@warrenbailey.net>
* Fix registering a device on an account with lots of devices (#15348)Erik Johnston2023-03-291-0/+2
| | | Fixes up #15183
* Delete stale non-e2e devices for users, take 3 (#15183)Erik Johnston2023-03-292-3/+49
| | | | | This should help reduce the number of devices e.g. simple bots the repeatedly login rack up. We only delete non-e2e devices as they should be safe to delete, whereas if we delete e2e devices for a user we may accidentally break their ability to receive e2e keys for a message.
* Fix joining rooms you have been unbanned from (#15323)DeepBlueV7.X2023-03-292-53/+58
| | | | | | | | | | | | | | | | | | | | | | | * Fix joining rooms you have been unbanned from Since forever synapse did not allow you to join a room after you have been unbanned from it over federation. This was not actually because of the unban event not federating. Synapse simply used outdated state to validate the join transition. This skips the validation if we are not in the room and for that reason won't have the current room state. Fixes #1563 Signed-off-by: Nicolas Werner <nicolas.werner@hotmail.de> * Add changelog Signed-off-by: Nicolas Werner <nicolas.werner@hotmail.de> * Update changelog.d/15323.bugfix --------- Signed-off-by: Nicolas Werner <nicolas.werner@hotmail.de>
* Implement MSC3983 to proxy /keys/claim queries to appservices. (#15314)Patrick Cloke2023-03-282-9/+122
| | | | | | Experimental support for MSC3983 is behind a configuration flag. If enabled, for users which are exclusively owned by an application service then the appservice will be queried for one-time keys *if* there are none uploaded to Synapse.
* Prune old typing notifications (#15332)Erik Johnston2023-03-271-0/+25
| | | | | Rather than keeping them around forever in memory, slowing things down. Fixes #11750.
* Make cleaning up pushers depend on the device_id instead of the token_id ↵Quentin Gliech2023-03-243-4/+10
| | | | | | | | | | | | | | (#15280) This makes it so that we rely on the `device_id` to delete pushers on logout, instead of relying on the `access_token_id`. This ensures we're not removing pushers on token refresh, and prepares for a world without access token IDs (also known as the OIDC). This actually runs the `set_device_id_for_pushers` background update, which was forgotten in #13831. Note that for backwards compatibility it still deletes pushers based on the `access_token` until the background update finishes.
* Make `EventHandler.get_event` return `None` when the requested event is not ↵Andrew Morgan2023-03-211-4/+5
| | | | found (#15298)
* Fix error when sending message into deleted room. (#15235)Erik Johnston2023-03-211-2/+15
| | | | | | | | When a room is deleted in Synapse we remove the event forward extremities in the room, so if (say a bot) tries to send a message into the room we error out due to not being able to calculate prev events for the new event *before* we check if the sender is in the room. Fixes #8094
* Refresh remote profiles that have been marked as stale, in order to fill the ↵reivilibre2023-03-162-2/+244
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | user directory. [rei:userdirpriv] (#14756) * Scaffolding for background process to refresh profiles * Add scaffolding for background process to refresh profiles for a given server * Implement the code to select servers to refresh from * Ensure we don't build up multiple looping calls * Make `get_profile` able to respect backoffs * Add logic for refreshing users * When backing off, schedule a refresh when the backoff is over * Wake up the background processes when we receive an interesting state event * Add tests * Newsfile Signed-off-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org> * Add comment about 1<<62 --------- Signed-off-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org>
* Move Account Validity callbacks to a dedicated file (#15237)Andrew Morgan2023-03-161-85/+14
|
* Preparatory work to fix the user directory assuming that any remote ↵reivilibre2023-03-161-34/+47
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | membership state events represent a profile change. [rei:userdirpriv] (#14755) * Remove special-case method for new memberships only, use more generic method * Only collect profiles from state events in public rooms * Add a table to track stale remote user profiles * Add store methods to set and delete rows in this new table * Mark remote profiles as stale when a member state event comes in to a private room * Newsfile Signed-off-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org> * Simplify by removing Optionality of `event_id` * Replace names and avatars with None if they're set to dodgy things I think this makes more sense anyway. * Move schema delta to 74 (I missed the boat?) * Turns out these can be None after all --------- Signed-off-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org>
* Avoid unneeded work if auto-join rooms aren't configured. (#15262)Patrick Cloke2023-03-141-2/+8
| | | | | | It is not necessary to reach out to the database to check some parameters if the auto-join rooms are not configured, or (in some cases) if auto-create rooms is not configured.
* Merge branch 'release-v1.79' into developDavid Robertson2023-03-131-5/+24
|\
| * Refactor `filter_events_for_server` (#15240)David Robertson2023-03-101-5/+24
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Tweak docstring and type hint * Flip logic and provide better name * Separate decision from action * Track a set of strings, not EventBases * Require explicit boolean options from callers * Add explicit option for partial state rooms * Changelog * Rename param
* | Faster joins: Fix spurious errors on incremental sync (#15232)Sean Quah2023-03-091-2/+7
| | | | | | | | | | | | | | | | | | | | | | | | | | | | When pushing events in partial state rooms down incremental /sync, we try to find the `m.room.member` state event for their senders by digging through their auth events, so that we can present the membership to the client. Events usually have a membership event in their auth events, with the exception of the `m.room.create` event and a user's first join into the room. When implementing #13477, we took the case of a user's first join into account, but forgot to handle the `m.room.create` case. This change fixes that. Signed-off-by: Sean Quah <seanq@matrix.org>
* | Add topic and name events to group of events that are batch persisted when ↵Shay2023-03-081-56/+52
| | | | | | | | creating a room. (#15229)
* | Include the room ID in more purge room log lines. (#15222)Patrick Cloke2023-03-081-1/+1
| |
* | More speedups/fixes to creating batched events (#15195)Shay2023-03-072-3/+14
|/
* Pass the requester during event serialization. (#15174)Quentin Gliech2023-03-066-46/+93
| | | | This allows Synapse to properly include the transaction ID in the unsigned data of events.
* Experimental MSC3890 Implementation: Fix deleting account data when using an ↵Andrew Morgan2023-03-031-7/+0
| | | | account data writer worker (#14869)
* Add support for knocking to workers. (#15133)Dirk Klimpel2023-03-022-2/+6
|
* Implementation of MSC3967: Don't require UIA for initial upload of cross ↵Hugh Nimmo-Smith2023-03-021-0/+14
| | | | signing keys (#15077)
* Remove support for aggregating reactions (#15172)Richard van der Hoff2023-02-281-75/+1
| | | | | | | | | | It turns out that no clients rely on server-side aggregation of `m.annotation` relationships: it's just not very useful as currently implemented. It's also non-trivial to calculate. I want to remove it from MSC2677, so to keep the implementation in line, let's remove it here.
* Add module API callbacks for adding and deleting local 3PID associations (#15044Andrew Morgan2023-02-272-27/+42
|
* Batch up storing state groups when creating new room (#14918)Shay2023-02-244-29/+41
|
* Add information on uploaded media to user export command. (#15107)Dirk Klimpel2023-02-231-0/+38
|
* Bump black from 22.12.0 to 23.1.0 (#15103)dependabot[bot]2023-02-2210-16/+12
|
* Remove unused `room_alias` field from `/createRoom` response (#15093)David Robertson2023-02-222-21/+21
| | | | | | | | | | | | | | * Change `create_room` return type * Don't return room alias from /createRoom * Update other callsites * Fix up mypy complaints It looks like new_room_user_id is None iff new_room_id is None. It's a shame we haven't expressed this in a way that mypy can understand. * Changelog
* Add account data to export command (#14969)Dirk Klimpel2023-02-171-15/+34
| | | | | | | | | | | * Add account data to to export command * newsfile * remove not needed function * update newsfile * adopt #14973
* Mitigate a race where /make_join could 403 for restricted rooms (#15080)Sean Quah2023-02-171-1/+15
| | | | | | | | | | | | | | | | | | | | | | | | | Previously, when creating a join event in /make_join, we would decide whether to include additional fields to satisfy restricted room checks based on the current state of the room. Then, when building the event, we would capture the forward extremities of the room to use as prev events. This is subject to race conditions. For example, when leaving and rejoining a room, the following sequence of events leads to a misleading 403 response: 1. /make_join reads the current state of the room and sees that the user is still in the room. It decides to omit the field required for restricted room joins. 2. The leave event is persisted and the room's forward extremities are updated. 3. /make_join builds the event, using the post-leave forward extremities. The event then fails the restricted room checks. To mitigate the race, we move the read of the forward extremities closer to the read of the current state. Ideally, we would compute the state based off the chosen prev events, but that can involve state resolution, which is expensive. Signed-off-by: Sean Quah <seanq@matrix.org>
* Use mypy 1.0 (#15052)David Robertson2023-02-162-5/+15
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Update mypy and mypy-zope * Remove unused ignores These used to suppress ``` synapse/storage/engines/__init__.py:28: error: "__new__" must return a class instance (got "NoReturn") [misc] ``` and ``` synapse/http/matrixfederationclient.py:1270: error: "BaseException" has no attribute "reasons" [attr-defined] ``` (note that we check `hasattr(e, "reasons")` above) * Avoid empty body warnings, sometimes by marking methods as abstract E.g. ``` tests/handlers/test_register.py:58: error: Missing return statement [empty-body] tests/handlers/test_register.py:108: error: Missing return statement [empty-body] ``` * Suppress false positive about `JaegerConfig` Complaint was ``` synapse/logging/opentracing.py:450: error: Function "Type[Config]" could always be true in boolean context [truthy-function] ``` * Fix not calling `is_state()` Oops! ``` tests/rest/client/test_third_party_rules.py:428: error: Function "Callable[[], bool]" could always be true in boolean context [truthy-function] ``` * Suppress false positives from ParamSpecs ```` synapse/logging/opentracing.py:971: error: Argument 2 to "_custom_sync_async_decorator" has incompatible type "Callable[[Arg(Callable[P, R], 'func'), **P], _GeneratorContextManager[None]]"; expected "Callable[[Callable[P, R], **P], _GeneratorContextManager[None]]" [arg-type] synapse/logging/opentracing.py:1017: error: Argument 2 to "_custom_sync_async_decorator" has incompatible type "Callable[[Arg(Callable[P, R], 'func'), **P], _GeneratorContextManager[None]]"; expected "Callable[[Callable[P, R], **P], _GeneratorContextManager[None]]" [arg-type] ```` * Drive-by improvement to `wrapping_logic` annotation * Workaround false "unreachable" positives See https://github.com/Shoobx/mypy-zope/issues/91 ``` tests/http/test_proxyagent.py:626: error: Statement is unreachable [unreachable] tests/http/test_proxyagent.py:762: error: Statement is unreachable [unreachable] tests/http/test_proxyagent.py:826: error: Statement is unreachable [unreachable] tests/http/test_proxyagent.py:838: error: Statement is unreachable [unreachable] tests/http/test_proxyagent.py:845: error: Statement is unreachable [unreachable] tests/http/federation/test_matrix_federation_agent.py:151: error: Statement is unreachable [unreachable] tests/http/federation/test_matrix_federation_agent.py:452: error: Statement is unreachable [unreachable] tests/logging/test_remote_handler.py:60: error: Statement is unreachable [unreachable] tests/logging/test_remote_handler.py:93: error: Statement is unreachable [unreachable] tests/logging/test_remote_handler.py:127: error: Statement is unreachable [unreachable] tests/logging/test_remote_handler.py:152: error: Statement is unreachable [unreachable] ``` * Changelog * Tweak DBAPI2 Protocol to be accepted by mypy 1.0 Some extra context in: - https://github.com/matrix-org/python-canonicaljson/pull/57 - https://github.com/python/mypy/issues/6002 - https://mypy.readthedocs.io/en/latest/common_issues.html#covariant-subtyping-of-mutable-protocol-members-is-rejected * Pull in updated canonicaljson lib so the protocol check just works * Improve comments in opentracing I tried to workaround the ignores but found it too much trouble. I think the corresponding issue is https://github.com/python/mypy/issues/12909. The mypy repo has a PR claiming to fix this (https://github.com/python/mypy/pull/14677) which might mean this gets resolved soon? * Better annotation for INTERACTIVE_AUTH_CHECKERS * Drive-by AUTH_TYPE annotation, to remove an ignore
* Update the error code for duplicate annotation (#15075)Richard van der Hoff2023-02-151-1/+5
|
* Fix order of partial state tables when purging (#15068)David Robertson2023-02-141-0/+5
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Fix order of partial state tables when purging `partial_state_rooms` has an FK on `events` pointing to the join event we get from `/send_join`, so we must delete from that table before deleting from `events`. **NB:** It would be nice to cancel any resync processes for the room being purged. We do not do this at present. To do so reliably we'd need an internal HTTP "replication" endpoint, because the worker doing the resync process may be different to that handling the purge request. The first time the resync process tries to write data after the deletion it will fail because we have deleted necessary data e.g. auth events. AFAICS it will not retry the resync, so the only downside to not cancelling the resync is a scary-looking traceback. (This is presumably extremely race-sensitive.) * Changelog * admist(?) -> between * Warn about a race * Fix typo, thanks Sean Co-authored-by: Sean Quah <8349537+squahtx@users.noreply.github.com> --------- Co-authored-by: Sean Quah <8349537+squahtx@users.noreply.github.com>
* Faster joins: Omit device list updates from partial state rooms in /sync ↵Sean Quah2023-02-141-0/+5
| | | | | | | | | | | | (#15069) ...when lazy loading of members is not enabled. It's weird to notify a client that another user's device list has changed when the client doesn't think that they share a room. Note that when a room is un-partial stated, device list updates are emitted for every member in that room over /sync. Signed-off-by: Sean Quah <seanq@matrix.org>
* Skip calculating unread push actions in `/sync` when `enable_push` is false. ↵Erik Johnston2023-02-141-0/+8
| | | | (#14980)
* Tweak comment on `_is_local_room_accessible` as part of room visibility in ↵reivilibre2023-02-131-2/+2
| | | | `/hierarchy` to clarify the condition for a room being visible. (#14834)
* Refactor arguments of `try_unbind_threepid(_with_id_server)` from dict to ↵Andrew Morgan2023-02-133-33/+26
| | | | separate args (#15053)
* Faster joins: don't stall when a user joins during a fast join (#14606)Mathieu Velten2023-02-107-53/+151
| | | | | | | | | | | | | | | | Fixes #12801. Complement tests are at https://github.com/matrix-org/complement/pull/567. Avoid blocking on full state when handling a subsequent join into a partial state room. Also always perform a remote join into partial state rooms, since we do not know whether the joining user has been banned and want to avoid leaking history to banned users. Signed-off-by: Mathieu Velten <mathieuv@matrix.org> Co-authored-by: Sean Quah <seanq@matrix.org> Co-authored-by: David Robertson <davidr@element.io>
* Return read-only collections from `@cached` methods (#13755)Sean Quah2023-02-104-8/+8
| | | | | | | | | | | | | It's important that collections returned from `@cached` methods are not modified, otherwise future retrievals from the cache will return the modified collection. This applies to the return values from `@cached` methods and the values inside the dictionaries returned by `@cachedList` methods. It's not necessary for the dictionaries returned by `@cachedList` methods themselves to be read-only. Signed-off-by: Sean Quah <seanq@matrix.org> Co-authored-by: David Robertson <davidr@element.io>
* Avoid fetching unused account data in sync. (#14973)Patrick Cloke2023-02-104-48/+57
| | | | | | | The per-room account data is no longer unconditionally fetched, even if all rooms will be filtered out. Global account data will not be fetched if it will all be filtered out.
* Avoid mutating cached values in `_generate_sync_entry_for_account_data` (#15047)Sean Quah2023-02-101-0/+2
|
* Refactor get_user_devices_from_cache to avoid mutating cached values. (#15040)Patrick Cloke2023-02-101-4/+7
| | | | | | | | The previous version of the code could mutate a cached value, but only if the input requested all devices of a user *and* a specific device. To avoid this nonsensical situation we no longer fetch a specific device ID if all of a user's devices are returned.
* Add a class UnpersistedEventContext to allow for the batching up of storing ↵Shay2023-02-093-37/+70
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | state groups (#14675) * add class UnpersistedEventContext * modify create new client event to create unpersistedeventcontexts * persist event contexts after creation * fix tests to persist unpersisted event contexts * cleanup * misc lints + cleanup * changelog + fix comments * lints * fix batch insertion? * reduce redundant calculation * add unpersisted event classes * rework compute_event_context, split into function that returns unpersisted event context and then persists it * use calculate_context_info to create unpersisted event contexts * update typing * $%#^&* * black * fix comments and consolidate classes, use attr.s for class * requested changes * lint * requested changes * requested changes * refactor to be stupidly explicit * clearer renaming and flow * make partial state non-optional * update docstrings --------- Co-authored-by: Erik Johnston <erik@matrix.org>
* Avoid mutating cached room aliases. (#15038)Patrick Cloke2023-02-091-1/+2
| | | | | This might cause incorrect data in other callers which are not expecting the canonical alias to be added into the response.
* Add final type hint to synapse.server. (#15035)Patrick Cloke2023-02-091-1/+1
|