summary refs log tree commit diff
path: root/tests/rest/client (follow)
Commit message (Collapse)AuthorAgeFilesLines
* Drop MSC3244 as it was rejected HEAD developRory&2025-06-291-40/+0
|
* Remove 3pid and identity serversRory&2025-06-2810-1819/+376
|
* Remove SAML/2 authRory&2025-06-281-72/+2
|
* Remove CAS authRory&2025-06-281-216/+1
|
* Add media repository callbacks to module API to control media upload size ↵Hugh Nimmo-Smith2025-06-041-0/+57
| | | | | | | | (#18457) Adds new callbacks for media related functionality: - `get_media_config_for_user` - `is_user_allowed_to_upload_media_of_size`
* Bump ruff from 0.7.3 to 0.11.10 (#18451)dependabot[bot]2025-05-203-7/+7
| | | | | | Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Andrew Morgan <andrew@amorgan.xyz> Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
* Pass leave from remote invite rejection down Sliding Sync (#18375)Devon Hudson2025-05-081-0/+58
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Fixes #17753 ### Dev notes The `sliding_sync_membership_snapshots` and `sliding_sync_joined_rooms` database tables were added in https://github.com/element-hq/synapse/pull/17512 ### Pull Request Checklist <!-- Please read https://element-hq.github.io/synapse/latest/development/contributing_guide.html before submitting your pull request --> * [X] Pull request is based on the develop branch * [x] Pull request includes a [changelog file](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#changelog). The entry should: - Be a short description of your change which makes sense to users. "Fixed a bug that prevented receiving messages from other servers." instead of "Moved X method from `EventStore` to `EventWorkerStore`.". - Use markdown where necessary, mostly for `code blocks`. - End with either a period (.) or an exclamation mark (!). - Start with a capital letter. - Feel free to credit yourself, by adding a sentence "Contributed by @github_username." or "Contributed by [Your Name]." to the end of the entry. * [X] [Code style](https://element-hq.github.io/synapse/latest/code_style.html) is correct (run the [linters](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#run-the-linters)) --------- Co-authored-by: Erik Johnston <erik@matrix.org> Co-authored-by: Olivier 'reivilibre <oliverw@matrix.org> Co-authored-by: Eric Eastwood <erice@element.io>
* make tests tolerant to authlib 1.5.2 error messages (#18390)Florian Klink2025-05-051-10/+10
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | authlib 1.5.2 now single-quotes error messages in the claims, causing three tests to fail. Replace the comparison with a regex that accepts both single or double quotes. This succeeds the tests with both authlib 1.5.1 and 1.5.2. See https://github.com/NixOS/nixpkgs/pull/402797 for context. ### Pull Request Checklist <!-- Please read https://element-hq.github.io/synapse/latest/development/contributing_guide.html before submitting your pull request --> * [x] Pull request is based on the develop branch * [x] Pull request includes a [changelog file](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#changelog). The entry should: - Be a short description of your change which makes sense to users. "Fixed a bug that prevented receiving messages from other servers." instead of "Moved X method from `EventStore` to `EventWorkerStore`.". - Use markdown where necessary, mostly for `code blocks`. - End with either a period (.) or an exclamation mark (!). - Start with a capital letter. - Feel free to credit yourself, by adding a sentence "Contributed by @github_username." or "Contributed by [Your Name]." to the end of the entry. * [x] [Code style](https://element-hq.github.io/synapse/latest/code_style.html) is correct (run the [linters](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#run-the-linters))
* Store hashes of media files, and allow quarantining by hash. (#18277)Will Hunt2025-03-271-0/+3
| | | | | | | | | | | | | | | | This PR makes a few radical changes to media. This now stores the SHA256 hash of each file stored in the database (excluding thumbnails, more on that later). If a set of media is quarantined, any additional uploads of the same file contents or any other files with the same hash will be quarantined at the same time. Currently this does NOT: - De-duplicate media, although a future extension could be to do that. - Run any background jobs to identify the hashes of older files. This could also be a future extension, though the value of doing so is limited to combat the abuse of recent media. - Hash thumbnails. It's assumed that thumbnails are parented to some form of media, so you'd likely be wanting to quarantine the media and the thumbnail at the same time.
* Add a column `participant` to `room_memberships` table (#18068)Shay2025-03-181-0/+193
|
* MSC4140: don't cancel delayed state on own state (#17810)Andrew Ferrazzutti2025-03-171-26/+117
| | | | | | | When a user sends a state event, do not cancel their own delayed events for the same piece of state. For context, see [the relevant section in the MSC](https://github.com/matrix-org/matrix-spec-proposals/blob/a09a883d9a013ac4b6ffddebd7ea87a827d211b9/proposals/4140-delayed-events-futures.md#delayed-state-events-are-cancelled-by-a-more-recent-state-event).
* Add caching support to media endpoints (#18235)Erik Johnston2025-03-131-0/+110
| | | | | | | | | | | | | | | | | We do a few things in this PR to better support caching: 1. Change `Cache-Control` header to allow intermediary proxies to cache media *only* if they revalidate on every request. This means that the intermediary cache will still send the request to Synapse but with a `If-None-Match` header, at which point Synapse can check auth and respond with a 304 and empty content. 2. Add `ETag` response header to all media responses. We hardcode this to `1` since all media is immutable (beyond being deleted). 3. Check for `If-None-Match` header (after checking for auth), and if it matches then respond with a 304 and empty body. --------- Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
* Define delayed event ratelimit category (#18019)Andrew Ferrazzutti2025-02-252-0/+178
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Apply ratelimiting on delayed event management separately from messages. ### Pull Request Checklist <!-- Please read https://element-hq.github.io/synapse/latest/development/contributing_guide.html before submitting your pull request --> * [x] Pull request is based on the develop branch * [x] Pull request includes a [changelog file](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#changelog). The entry should: - Be a short description of your change which makes sense to users. "Fixed a bug that prevented receiving messages from other servers." instead of "Moved X method from `EventStore` to `EventWorkerStore`.". - Use markdown where necessary, mostly for `code blocks`. - End with either a period (.) or an exclamation mark (!). - Start with a capital letter. - Feel free to credit yourself, by adding a sentence "Contributed by @github_username." or "Contributed by [Your Name]." to the end of the entry. * [ ] [Code style](https://element-hq.github.io/synapse/latest/code_style.html) is correct (run the [linters](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#run-the-linters)) --------- Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
* Fix MSC4108 'rendez-vous' responses with some reverse proxy in the front of ↵Quentin Gliech2025-02-251-2/+4
| | | | | | | | | | | | | | | | | | | | Synapse (#18178) MSC4108 relies on ETag to determine if something has changed on the rendez-vous channel. Strong and correct ETag comparison works if the response body is bit-for-bit identical, which isn't the case if a proxy in the middle compresses the response on the fly. This adds a `no-transform` directive to the `Cache-Control` header, which tells proxies not to transform the response body. Additionally, some proxies (nginx) will switch to `Transfer-Encoding: chunked` if it doesn't know the Content-Length of the response, and 'weakening' the ETag if that's the case. I've added `Content-Length` headers to all responses, to hopefully solve that. This basically fixes QR-code login when nginx or cloudflare is involved, with gzip/zstd/deflate compression enabled.
* Disable room list publication by default (#18175)Erik Johnston2025-02-241-0/+6
| | | | | | This is in line with our general policy of ensuring that the default config is reasonably locked down. SyTest PR to fix tests: https://github.com/matrix-org/sytest/pull/1396
* Prevent suspended users from sending encrypted messages (#18157)Shay2025-02-211-2/+100
| | | Missed in the first round.
* Cleanup Python 3.8 leftovers (#17967)V024602025-02-105-8/+5
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Some small cleanups after Python3.8 became EOL. - Move some type imports from `typing_extensions` to `typing` - Remove the `abi3-py38` feature from pyo3 ### Pull Request Checklist <!-- Please read https://element-hq.github.io/synapse/latest/development/contributing_guide.html before submitting your pull request --> * [x] Pull request is based on the develop branch * [x] Pull request includes a [changelog file](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#changelog). The entry should: - Be a short description of your change which makes sense to users. "Fixed a bug that prevented receiving messages from other servers." instead of "Moved X method from `EventStore` to `EventWorkerStore`.". - Use markdown where necessary, mostly for `code blocks`. - End with either a period (.) or an exclamation mark (!). - Start with a capital letter. - Feel free to credit yourself, by adding a sentence "Contributed by @github_username." or "Contributed by [Your Name]." to the end of the entry. * [x] [Code style](https://element-hq.github.io/synapse/latest/code_style.html) is correct (run the [linters](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#run-the-linters)) --------- Co-authored-by: Quentin Gliech <quenting@element.io>
* Fix performance of `check_state_groups_and_bump_deletion` (#18141)Erik Johnston2025-02-071-2/+2
| | | | | | | | | | Regressed as part of https://github.com/element-hq/synapse/pull/18107 This does two things: 1. Only check if the state groups have been deleted when calculating the event context (as that's when we will insert them). This avoids lots of checks for read operations. 2. Don't lock the `state_groups` rows when doing the check. This adds overhead, and it doesn't prevent any races.
* Add locking to more safely delete state groups: Part 2 (#18130)Erik Johnston2025-02-031-1/+1
| | | | | | This actually makes it so that deleting state groups goes via the new mechanism. c.f. #18107
* Add locking to more safely delete state groups: Part 1 (#18107)Erik Johnston2025-02-031-2/+2
| | | | | | | | | | | | | | | | Currently we don't really have anything that stops us from deleting state groups when an in-flight event references it. This is a fairly rare race currently, but we want to be able to more aggressively delete state groups so it is important to address this to ensure that the database remains valid. This implements the locking, but doesn't actually use it. See the class docstring of the new data store for an explanation for how this works. --------- Co-authored-by: Devon Hudson <devon.dmytro@gmail.com>
* Fix join being denied after being invited over federation (#18075)Eric Eastwood2025-01-271-2/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This also happens for rejecting an invite. Basically, any out-of-band membership transition where we first get the membership as an `outlier` and then rely on federation filling us in to de-outlier it. This PR mainly addresses automated test flakiness, bots/scripts, and options within Synapse like [`auto_accept_invites`](https://element-hq.github.io/synapse/v1.122/usage/configuration/config_documentation.html#auto_accept_invites) that are able to react quickly (before federation is able to push us events), but also helps in generic scenarios where federation is lagging. I initially thought this might be a Synapse consistency issue (see issues labeled with [`Z-Read-After-Write`](https://github.com/matrix-org/synapse/labels/Z-Read-After-Write)) but it seems to be an event auth logic problem. Workers probably do increase the number of possible race condition scenarios that make this visible though (replication and cache invalidation lag). Fix https://github.com/element-hq/synapse/issues/15012 (probably fixes https://github.com/matrix-org/synapse/issues/15012 (https://github.com/element-hq/synapse/issues/15012)) Related to https://github.com/matrix-org/matrix-spec/issues/2062 Problems: 1. We don't consider [out-of-band membership](https://github.com/element-hq/synapse/blob/develop/docs/development/room-dag-concepts.md#out-of-band-membership-events) (outliers) in our `event_auth` logic even though we expose them in `/sync`. 1. (This PR doesn't address this point) Perhaps we should consider authing events in the persistence queue as events already in the queue could allow subsequent events to be allowed (events come through many channels: federation transaction, remote invite, remote join, local send). But this doesn't save us in the case where the event is more delayed over federation. ### What happened before? I wrote some Complement test that stresses this exact scenario and reproduces the problem: https://github.com/matrix-org/complement/pull/757 ``` COMPLEMENT_ALWAYS_PRINT_SERVER_LOGS=1 COMPLEMENT_DIR=../complement ./scripts-dev/complement.sh -run TestSynapseConsistency ``` We have `hs1` and `hs2` running in monolith mode (no workers): 1. `@charlie1:hs2` is invited and joins the room: 1. `hs1` invites `@charlie1:hs2` to a room which we receive on `hs2` as `PUT /_matrix/federation/v1/invite/{roomId}/{eventId}` (`on_invite_request(...)`) and the invite membership is persisted as an outlier. The `room_memberships` and `local_current_membership` database tables are also updated which means they are visible down `/sync` at this point. 1. `@charlie1:hs2` decides to join because it saw the invite down `/sync`. Because `hs2` is not yet in the room, this happens as a remote join `make_join`/`send_join` which comes back with all of the auth events needed to auth successfully and now `@charlie1:hs2` is successfully joined to the room. 1. `@charlie2:hs2` is invited and and tries to join the room: 1. `hs1` invites `@charlie2:hs2` to the room which we receive on `hs2` as `PUT /_matrix/federation/v1/invite/{roomId}/{eventId}` (`on_invite_request(...)`) and the invite membership is persisted as an outlier. The `room_memberships` and `local_current_membership` database tables are also updated which means they are visible down `/sync` at this point. 1. Because `hs2` is already participating in the room, we also see the invite come over federation in a transaction and we start processing it (not done yet, see below) 1. `@charlie2:hs2` decides to join because it saw the invite down `/sync`. Because `hs2`, is already in the room, this happens as a local join but we deny the event because our `event_auth` logic thinks that we have no membership in the room :x: (expected to be able to join because we saw the invite down `/sync`) 1. We finally finish processing the `@charlie2:hs2` invite event from and de-outlier it. - If this finished before we tried to join we would have been fine but this is the race condition that makes this situation visible. Logs for `hs2`: ``` 🗳️ on_invite_request: handling event <FrozenEventV3 event_id=$PRPCvdXdcqyjdUKP_NxGF2CcukmwOaoK0ZR1WiVOZVk, type=m.room.member, state_key=@user-2-charlie1:hs2, membership=invite, outlier=False> 🔦 _store_room_members_txn update room_memberships: <FrozenEventV3 event_id=$PRPCvdXdcqyjdUKP_NxGF2CcukmwOaoK0ZR1WiVOZVk, type=m.room.member, state_key=@user-2-charlie1:hs2, membership=invite, outlier=True> 🔦 _store_room_members_txn update local_current_membership: <FrozenEventV3 event_id=$PRPCvdXdcqyjdUKP_NxGF2CcukmwOaoK0ZR1WiVOZVk, type=m.room.member, state_key=@user-2-charlie1:hs2, membership=invite, outlier=True> 📨 Notifying about new event <FrozenEventV3 event_id=$PRPCvdXdcqyjdUKP_NxGF2CcukmwOaoK0ZR1WiVOZVk, type=m.room.member, state_key=@user-2-charlie1:hs2, membership=invite, outlier=True> ✅ on_invite_request: handled event <FrozenEventV3 event_id=$PRPCvdXdcqyjdUKP_NxGF2CcukmwOaoK0ZR1WiVOZVk, type=m.room.member, state_key=@user-2-charlie1:hs2, membership=invite, outlier=True> 🧲 do_invite_join for @user-2-charlie1:hs2 in !sfZVBdLUezpPWetrol:hs1 🔦 _store_room_members_txn update room_memberships: <FrozenEventV3 event_id=$bwv8LxFnqfpsw_rhR7OrTjtz09gaJ23MqstKOcs7ygA, type=m.room.member, state_key=@user-1-alice:hs1, membership=join, outlier=True> 🔦 _store_room_members_txn update room_memberships: <FrozenEventV3 event_id=$oju1ts3G3pz5O62IesrxX5is4LxAwU3WPr4xvid5ijI, type=m.room.member, state_key=@user-2-charlie1:hs2, membership=join, outlier=False> 📨 Notifying about new event <FrozenEventV3 event_id=$oju1ts3G3pz5O62IesrxX5is4LxAwU3WPr4xvid5ijI, type=m.room.member, state_key=@user-2-charlie1:hs2, membership=join, outlier=False> ... 🗳️ on_invite_request: handling event <FrozenEventV3 event_id=$O_54j7O--6xMsegY5EVZ9SA-mI4_iHJOIoRwYyeWIPY, type=m.room.member, state_key=@user-3-charlie2:hs2, membership=invite, outlier=False> 🔦 _store_room_members_txn update room_memberships: <FrozenEventV3 event_id=$O_54j7O--6xMsegY5EVZ9SA-mI4_iHJOIoRwYyeWIPY, type=m.room.member, state_key=@user-3-charlie2:hs2, membership=invite, outlier=True> 🔦 _store_room_members_txn update local_current_membership: <FrozenEventV3 event_id=$O_54j7O--6xMsegY5EVZ9SA-mI4_iHJOIoRwYyeWIPY, type=m.room.member, state_key=@user-3-charlie2:hs2, membership=invite, outlier=True> 📨 Notifying about new event <FrozenEventV3 event_id=$O_54j7O--6xMsegY5EVZ9SA-mI4_iHJOIoRwYyeWIPY, type=m.room.member, state_key=@user-3-charlie2:hs2, membership=invite, outlier=True> ✅ on_invite_request: handled event <FrozenEventV3 event_id=$O_54j7O--6xMsegY5EVZ9SA-mI4_iHJOIoRwYyeWIPY, type=m.room.member, state_key=@user-3-charlie2:hs2, membership=invite, outlier=True> 📬 handling received PDU in room !sfZVBdLUezpPWetrol:hs1: <FrozenEventV3 event_id=$O_54j7O--6xMsegY5EVZ9SA-mI4_iHJOIoRwYyeWIPY, type=m.room.member, state_key=@user-3-charlie2:hs2, membership=invite, outlier=False> 📮 handle_new_client_event: handling <FrozenEventV3 event_id=$WNVDTQrxy5tCdPQHMyHyIn7tE4NWqKsZ8Bn8R4WbBSA, type=m.room.member, state_key=@user-3-charlie2:hs2, membership=join, outlier=False> ❌ Denying new event <FrozenEventV3 event_id=$WNVDTQrxy5tCdPQHMyHyIn7tE4NWqKsZ8Bn8R4WbBSA, type=m.room.member, state_key=@user-3-charlie2:hs2, membership=join, outlier=False> because 403: You are not invited to this room. synapse.http.server - 130 - INFO - POST-16 - <SynapseRequest at 0x7f460c91fbf0 method='POST' uri='/_matrix/client/v3/join/%21sfZVBdLUezpPWetrol:hs1?server_name=hs1' clientproto='HTTP/1.0' site='8080'> SynapseError: 403 - You are not invited to this room. 📨 Notifying about new event <FrozenEventV3 event_id=$O_54j7O--6xMsegY5EVZ9SA-mI4_iHJOIoRwYyeWIPY, type=m.room.member, state_key=@user-3-charlie2:hs2, membership=invite, outlier=False> ✅ handled received PDU in room !sfZVBdLUezpPWetrol:hs1: <FrozenEventV3 event_id=$O_54j7O--6xMsegY5EVZ9SA-mI4_iHJOIoRwYyeWIPY, type=m.room.member, state_key=@user-3-charlie2:hs2, membership=invite, outlier=False> ```
* Ratelimit presence updates (#18000)Sven Mäder2025-01-241-0/+52
|
* Support the new `/auth_metadata` endpoint defined in MSC2965. (#18093)Quentin Gliech2025-01-212-79/+140
| | | | | | | See the updated MSC2965 --------- Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
* Implement MSC4133 to support custom profile fields. (#17488)Patrick Cloke2025-01-212-0/+340
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Implementation of [MSC4133](https://github.com/matrix-org/matrix-spec-proposals/pull/4133) to support custom profile fields. It is behind an experimental flag and includes tests. ### Pull Request Checklist <!-- Please read https://element-hq.github.io/synapse/latest/development/contributing_guide.html before submitting your pull request --> * [x] Pull request is based on the develop branch * [x] Pull request includes a [changelog file](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#changelog). The entry should: - Be a short description of your change which makes sense to users. "Fixed a bug that prevented receiving messages from other servers." instead of "Moved X method from `EventStore` to `EventWorkerStore`.". - Use markdown where necessary, mostly for `code blocks`. - End with either a period (.) or an exclamation mark (!). - Start with a capital letter. - Feel free to credit yourself, by adding a sentence "Contributed by @github_username." or "Contributed by [Your Name]." to the end of the entry. * [x] [Code style](https://element-hq.github.io/synapse/latest/code_style.html) is correct (run the [linters](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#run-the-linters)) --------- Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
* Bust `_membership_stream_cache` cache when current state changes (#17732)Eric Eastwood2025-01-081-18/+0
| | | | | | | | | | | | | | | | | | | | | | | This is particularly a problem in a state reset scenario where the membership might change without a corresponding event. This PR is targeting a scenario where a state reset happens which causes room membership to change. Previously, the cache would just hold onto stale data and now we properly bust the cache in this scenario. We have a few tests for these scenarios which you can see are now fixed because we can remove the `FIXME` where we were previously manually busting the cache in the test itself. This is a general Synapse thing so by it's nature it helps out Sliding Sync. Fix https://github.com/element-hq/synapse/issues/17368 Prerequisite for https://github.com/element-hq/synapse/issues/17929 --- Match when are busting `_curr_state_delta_stream_cache`
* Promote account suspension to stable (#17964)Travis Ralston2024-12-041-24/+8
| | | MSC: https://github.com/matrix-org/matrix-spec-proposals/pull/3823
* Support for MSC4190: device management for application services (#17705)Quentin Gliech2024-12-042-0/+209
| | | | | | | | This is an implementation of MSC4190, which allows appservices to manage their user's devices without /login & /logout. --------- Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
* Merge branch 'master' into developQuentin Gliech2024-12-031-3/+157
|\
| * Sliding Sync: Fix state leaking on incremental syncEric Eastwood2024-12-031-3/+157
| |
* | Sliding Sync: Include invite, ban, kick, targets when `$LAZY`-loading room ↵Eric Eastwood2024-12-021-3/+163
| | | | | | | | | | members (#17947) Part of https://github.com/element-hq/synapse/issues/17929
* | Consolidate SSO redirects through ↵Eric Eastwood2024-11-292-11/+140
|/ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | `/_matrix/client/v3/login/sso/redirect(/{idpId})` (#17972) Consolidate SSO redirects through `/_matrix/client/v3/login/sso/redirect(/{idpId})` Spawning from https://github.com/element-hq/sbg/pull/421#discussion_r1859497330 where we have a proxy that intercepts responses to `/_matrix/client/v3/login/sso/redirect(/{idpId})` in order to upgrade them to use OAuth 2.0 Pushed Authorization Requests (PAR). Instead of needing to intercept multiple endpoints that redirect to the authorization endpoint, it seems better to just have Synapse consolidate to a single flow. ### Testing strategy 1. Create a new OAuth application. I'll be using GitHub for example but there are [many options](https://github.com/matrix-org/synapse/blob/be65a8ec0195955c15fdb179c9158b187638e39a/docs/openid.md). Visit https://github.com/settings/developers -> **New OAuth App** - Application name: `Synapse local testing` - Homepage URL: `http://localhost:8008` - Authorization callback URL: `http://localhost:8008/_synapse/client/oidc/callback` 1. Update your Synapse `homeserver.yaml` ```yaml server_name: "my.synapse.server" public_baseurl: http://localhost:8008/ listeners: - port: 8008 bind_addresses: [ #'::1', '127.0.0.1' ] tls: false type: http x_forwarded: true resources: - names: [client, federation, metrics] compress: false # SSO login testing oidc_providers: - idp_id: github idp_name: Github idp_brand: "github" # optional: styling hint for clients discover: false issuer: "https://github.com/" client_id: "xxx" # TO BE FILLED client_secret: "xxx" # TO BE FILLED authorization_endpoint: "https://github.com/login/oauth/authorize" token_endpoint: "https://github.com/login/oauth/access_token" userinfo_endpoint: "https://api.github.com/user" scopes: ["read:user"] user_mapping_provider: config: subject_claim: "id" localpart_template: "{{ user.login }}" display_name_template: "{{ user.name }}" ``` 1. Start Synapse: `poetry run synapse_homeserver --config-path homeserver.yaml` 1. Visit `http://localhost:8008/_synapse/client/pick_idp?redirectUrl=http%3A%2F%2Fexample.com` 1. Choose GitHub 1. Notice that you're redirected to GitHub to sign in (`https://github.com/login/oauth/authorize?...`) Tested locally and works: 1. `http://localhost:8008/_synapse/client/pick_idp?idp=oidc-github&redirectUrl=http%3A//example.com` -> 1. `http://localhost:8008/_matrix/client/v3/login/sso/redirect/oidc-github?redirectUrl=http://example.com` -> 1. `https://github.com/login/oauth/authorize?response_type=code&client_id=xxx&redirect_uri=http%3A%2F%2Flocalhost%3A8008%2F_synapse%2Fclient%2Foidc%2Fcallback&scope=read%3Auser&state=xxx&nonce=xxx`
* Removal: Remove support for experimental msc3886 (#17638)Poruri Sai Rahul2024-11-131-9/+0
|
* Sliding Sync: Lazy-loading room members on incremental sync (remember ↵Eric Eastwood2024-11-041-4/+253
| | | | | | | | memberships) (#17809) Lazy-loading room members on incremental sync and remember which memberships we've sent down the connection before (up-to 100) Fix https://github.com/element-hq/synapse/issues/17804
* Support & use stable endpoints for MSC4151 (#17374)Travis Ralston2024-10-311-29/+2
| | | | | | | | | | | https://github.com/matrix-org/matrix-spec-proposals/pull/4151 has finished FCP. See https://github.com/element-hq/synapse/issues/17373 for unstable endpoint removal --------- Co-authored-by: Andrew Morgan <andrew@amorgan.xyz>
* Check if user is in room before being able to tag it (#17839)Lama2024-10-301-0/+95
| | | Fix #17819
* Add media tests for a CMYK JPEG image (#17786)Andrew Morgan2024-10-231-1/+3
|
* Sliding Sync: Reset `forgotten` status when membership changes (like ↵Eric Eastwood2024-10-222-2/+175
| | | | | | | | | | | | | | | | rejoining a room) (#17835) Reset `sliding_sync_membership_snapshots` -> `forgotten` status when membership changes (like rejoining a room). Fix https://github.com/element-hq/synapse/issues/17781 ### What was the problem before? Previously, if someone used `/forget` on one of their rooms, it would update `sliding_sync_membership_snapshots` as expected but when someone rejoined the room (or had any membership change), the upsert didn't overwrite and reset the `forgotten` status so it remained `forgotten` and invisible down the Sliding Sync endpoint.
* Correctly changes to required state config in sliding sync (#17785)Erik Johnston2024-10-141-0/+261
| | | | | | | | | | | | | | | | | | Fixes https://github.com/element-hq/synapse/issues/17698 This handles `required_state` changes by checking if new state has been added to the config, and if so fetching and returning that from the current state. This also takes care to ensure that given a state entry S that is added, removed and then re-added that we do *not* send S down a second time if there have been no changes to S in the current state. This is fine for Rust SDK (as it just remembers all state), but we might decide not to do this behaviour in the MSC. If we decide to always send down S then its easy enough to rip out all the code. --------- Co-authored-by: Eric Eastwood <eric.eastwood@beta.gouv.fr>
* Added display_name_claim in jwt_config which sets the user's display name ↵Nathan2024-10-091-0/+25
| | | | upon registration (#17708)
* Add missing license header (#17799)Andrew Ferrazzutti2024-10-082-1/+15
| | | | Co-authored-by: Erik Johnston <erik@matrix.org>
* Sliding sync: omit bump stamp when it is unchanged (#17788)Erik Johnston2024-10-081-0/+86
| | | This saves some DB lookups in rooms
* Don't say MSC4140 is supported when it's disabled (#17780)Andrew Ferrazzutti2024-10-041-1/+17
|
* Optimise notifier (#17765)Erik Johnston2024-09-301-10/+21
| | | | | | | The notifier is quite inefficient when it has to wake up many user streams all at once From a silly benchmark this takes the time to notify 1M user streams from ~30s to ~5s
* Support MSC3757: Restricting who can overwrite a state event (#17513)Andrew Ferrazzutti2024-09-261-0/+308
| | | | | | | | Link to the MSC: https://github.com/matrix-org/matrix-spec-proposals/pull/3757 --------- Co-authored-by: Quentin Gliech <quenting@element.io>
* Support MSC4140: Delayed events (Futures) (#17326)Andrew Ferrazzutti2024-09-232-0/+446
|
* Sliding sync: Add connection tracking to the `account_data` extension (#17695)Erik Johnston2024-09-191-14/+471
| | | | | | | | | | | This is basically exactly the same logic as for receipts. Essentially we just need to track which room account data we have and haven't sent down to clients, and use that when we pull stuff out. I think this just needs a couple of extra tests written --------- Co-authored-by: Eric Eastwood <eric.eastwood@beta.gouv.fr>
* Sliding Sync: Avoid fetching left rooms and add back `newly_left` rooms (#17725)Eric Eastwood2024-09-191-1/+478
| | | | | | | | | | | | | | | Performance optimization: We can avoid fetching rooms that the user has left themselves (which could be a significant amount), then only add back rooms that the user has `newly_left` (left in the token range of an incremental sync). It's a lot faster to fetch less rooms than fetch them all and throw them away in most cases. Since the user only leaves a room (or is state reset out) once in a blue moon, we can avoid a lot of work. Based on @erikjohnston's branch, erikj/ss_perf --------- Co-authored-by: Erik Johnston <erik@matrix.org>
* Fix sliding sync for rooms with unknown room version (#17733)Erik Johnston2024-09-191-0/+52
| | | Follow on from #17727
* Sliding Sync: No need to sort if the range is large enough to cover all of ↵Eric Eastwood2024-09-193-49/+67
| | | | | | | | | the rooms (#17731) No need to sort if the range is large enough to cover all of the rooms in the list. Previously, we would only do this optimization if the range was exactly large enough. Follow-up to https://github.com/element-hq/synapse/pull/17672
* Sliding sync: Ignore invites from ignored users (#17729)Eric Eastwood2024-09-181-1/+112
| | | `m.ignored_user_list` in account data
* Sliding Sync: Return room tags in account data extension (#17707)Eric Eastwood2024-09-161-38/+124
| | | | | | The account data extension was also updated to avoid copies when we pull the data out of the cache. Fix https://github.com/element-hq/synapse/issues/17694
* Sliding Sync: Support filtering by 'tags' / 'not_tags' in SSS (#17662)David Baker2024-09-121-1/+309
| | | | | | | | This appears to be enough to make Element Web work (or at least move it on to the next hurdle) --------- Co-authored-by: Eric Eastwood <eric.eastwood@beta.gouv.fr>
* Sliding Sync: Move filters tests to rest layer (#17703)Eric Eastwood2024-09-123-407/+1904
| | | | | | | | | | | | | Move filters tests to rest layer in order to test the new (with sliding sync tables) and fallback paths that Sliding Sync can use. Also found a bug in the new path because it's not being tested which is also fixed in this PR. We now take into account `has_known_state` when filtering. Spawning from https://github.com/element-hq/synapse/pull/17662#discussion_r1755574791. This should have been done when we started using the new sliding sync tables in https://github.com/element-hq/synapse/pull/17630
* import pydantic objects from the `_pydantic_compat` module (#17667)Éloi Rivard2024-09-111-7/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This PR changes `from pydantic import BaseModel` to `from synapse._pydantic_compat import BaseModel` (as well as `constr`, `conbytes`, `conint`, `confloat`). It allows `check_pydantic_models.py` to mock those pydantic objects only in the synapse module, and not interfere with pydantic objects in external dependencies. This should solve the CI problems for #17144, which breaks because `check_pydantic_models.py` patches pydantic models from [scim2-models](https://scim2-models.readthedocs.io/). /cc @DMRobertson @gotmax23 fixes #17659 ### Pull Request Checklist <!-- Please read https://element-hq.github.io/synapse/latest/development/contributing_guide.html before submitting your pull request --> * [x] Pull request is based on the develop branch * [x] Pull request includes a [changelog file](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#changelog). The entry should: - Be a short description of your change which makes sense to users. "Fixed a bug that prevented receiving messages from other servers." instead of "Moved X method from `EventStore` to `EventWorkerStore`.". - Use markdown where necessary, mostly for `code blocks`. - End with either a period (.) or an exclamation mark (!). - Start with a capital letter. - Feel free to credit yourself, by adding a sentence "Contributed by @github_username." or "Contributed by [Your Name]." to the end of the entry. * [x] [Code style](https://element-hq.github.io/synapse/latest/code_style.html) is correct (run the [linters](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#run-the-linters))
* Sliding Sync: Make sure we get up-to-date information from ↵Eric Eastwood2024-09-111-55/+136
| | | | | | | | | | | | | | | | | `get_sliding_sync_rooms_for_user(...)` (#17692) We need to bust the `get_sliding_sync_rooms_for_user` cache when the room encryption is updated and any other field that is used in the query. Follow-up to https://github.com/element-hq/synapse/pull/17630 - Bust cache for membership change (cross-reference `get_rooms_for_user`) - Bust cache for room `encryption` (cross-reference `get_room_encryption`) - Bust cache for `forgotten` (cross-reference `did_forget`/`get_forgotten_rooms_for_user`)
* Sliding sync: don't fetch room summary for named rooms. (#17683)Erik Johnston2024-09-111-26/+26
| | | | | | | | | | For rooms with a name we can skip fetching a full room summary, as we don't need to calculate heroes, and instead just fetch the room counts directly. This also changes things to not return counts and heroes for non-joined rooms. For left/banned rooms we were returning zero values anyway, and for invite/knock rooms we don't really want to leak such information (even if some of is included in the stripped state).
* Sliding Sync: Retrieve fewer events from DB in sync (#17688)Erik Johnston2024-09-101-30/+20
| | | | When using timeline limit of 1 we end up fetching 2 events from the DB purely to tell if the response was "limited" or not. Lets not do that.
* Sliding Sync: Speed up incremental sync by avoiding extra work (#17665)Eric Eastwood2024-09-091-5/+344
| | | | | Speed up incremental sync by avoiding extra work. We first look at the state delta changes and only fetch and calculate further derived things if they have changed.
* Fix bump stamp for non-joined rooms (#17674)Erik Johnston2024-09-061-0/+45
| | | | We should only look for bump stamps in joined rooms, otherwise we should just use the membership stream ordering.
* Small performance improvements for sliding sync (#17672)Erik Johnston2024-09-061-11/+8
| | | A couple of small performance improvements for sliding sync.
* Revert "Fix bump stamp for non-joined rooms"Erik Johnston2024-09-051-45/+0
| | | | This reverts commit f73c844403de00630fd773075cefe6f502b54e69.
* Fix bump stamp for non-joined roomsErik Johnston2024-09-051-0/+45
| | | | | We should only look for bump stamps in joined rooms, otherwise we should just use the membership stream ordering.
* Format files with Ruff (#17643)Quentin Gliech2024-09-0219-40/+70
| | | | | | I thought ruff check would also format, but it doesn't. This runs ruff format in CI and dev scripts. The first commit is just a run of `ruff format .` in the root directory.
* Sliding sync: use new DB tables (#17630)Erik Johnston2024-09-0113-3/+221
| | | | | | | | | | | Based on https://github.com/element-hq/synapse/pull/17629 Utilizing the new sliding sync tables added in https://github.com/element-hq/synapse/pull/17512 for fast acquisition of rooms for the user and filtering/sorting. --------- Co-authored-by: Eric Eastwood <eric.eastwood@beta.gouv.fr>
* MSC3861: load the issuer and account management URLs from OIDC discovery ↵Quentin Gliech2024-08-301-1/+19
| | | | | | | | | | | | | | (#17407) This will help mitigating any discrepancies between the issuer configured and the one returned by the OIDC provider. This also removes the need for configuring the `account_management_url` explicitely, as it will now be loaded from the OIDC discovery, as per MSC2965. Because we may now fetch stuff for the .well-known/matrix/client endpoint, this also transforms the client well-known resource to be asynchronous.
* Use custom stage UIA error for MAS cross-signing reset (#17509)Michael Telatynski2024-08-301-9/+3
| | | | | | Rather than 501 M_UNRECOGNISED Client side implementation at https://github.com/matrix-org/matrix-react-sdk/pull/12892/
* Sliding sync: Store the per-connection state in the database. (#17599)Erik Johnston2024-08-291-2/+8
| | | | | | | Based on #17600 --------- Co-authored-by: Eric Eastwood <eric.eastwood@beta.gouv.fr>
* Sliding Sync: Pre-populate room data for quick filtering/sorting (#17512)Eric Eastwood2024-08-291-13/+45
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Pre-populate room data for quick filtering/sorting in the Sliding Sync API Spawning from https://github.com/element-hq/synapse/pull/17450#discussion_r1697335578 This PR is acting as the Synapse version `N+1` step in the gradual migration being tracked by https://github.com/element-hq/synapse/issues/17623 Adding two new database tables: - `sliding_sync_joined_rooms`: A table for storing room meta data that the local server is still participating in. The info here can be shared across all `Membership.JOIN`. Keyed on `(room_id)` and updated when the relevant room current state changes or a new event is sent in the room. - `sliding_sync_membership_snapshots`: A table for storing a snapshot of room meta data at the time of the local user's membership. Keyed on `(room_id, user_id)` and only updated when a user's membership in a room changes. Also adds background updates to populate these tables with all of the existing data. We want to have the guarantee that if a row exists in the sliding sync tables, we are able to rely on it (accurate data). And if a row doesn't exist, we use a fallback to get the same info until the background updates fill in the rows or a new event comes in triggering it to be fully inserted. This means we need a couple extra things in place until we bump `SCHEMA_COMPAT_VERSION` and run the foreground update in the `N+2` part of the gradual migration. For context on why we can't rely on the tables without these things see [1]. 1. On start-up, block until we clear out any rows for the rooms that have had events since the max-`stream_ordering` of the `sliding_sync_joined_rooms` table (compare to max-`stream_ordering` of the `events` table). For `sliding_sync_membership_snapshots`, we can compare to the max-`stream_ordering` of `local_current_membership` - This accounts for when someone downgrades their Synapse version and then upgrades it again. This will ensure that we don't have any stale/out-of-date data in the `sliding_sync_joined_rooms`/`sliding_sync_membership_snapshots` tables since any new events sent in rooms would have also needed to be written to the sliding sync tables. For example a new event needs to bump `event_stream_ordering` in `sliding_sync_joined_rooms` table or some state in the room changing (like the room name). Or another example of someone's membership changing in a room affecting `sliding_sync_membership_snapshots`. 1. Add another background update that will catch-up with any rows that were just deleted from the sliding sync tables (based on the activity in the `events`/`local_current_membership`). The rooms that need recalculating are added to the `sliding_sync_joined_rooms_to_recalculate` table. 1. Making sure rows are fully inserted. Instead of partially inserting, we need to check if the row already exists and fully insert all data if not. All of this extra functionality can be removed once the `SCHEMA_COMPAT_VERSION` is bumped with support for the new sliding sync tables so people can no longer downgrade (the `N+2` part of the gradual migration). <details> <summary><sup>[1]</sup></summary> For `sliding_sync_joined_rooms`, since we partially insert rows as state comes in, we can't rely on the existence of the row for a given `room_id`. We can't even rely on looking at whether the background update has finished. There could still be partial rows from when someone reverted their Synapse version after the background update finished, had some state changes (or new rooms), then upgraded again and more state changes happen leaving a partial row. For `sliding_sync_membership_snapshots`, we insert items as a whole except for the `forgotten` column ~~so we can rely on rows existing and just need to always use a fallback for the `forgotten` data. We can't use the `forgotten` column in the table for the same reasons above about `sliding_sync_joined_rooms`.~~ We could have an out-of-date membership from when someone reverted their Synapse version. (same problems as outlined for `sliding_sync_joined_rooms` above) Discussed in an [internal meeting](https://docs.google.com/document/d/1MnuvPkaCkT_wviSQZ6YKBjiWciCBFMd-7hxyCO-OCbQ/edit#bookmark=id.dz5x6ef4mxz7) </details> ### TODO - [x] Update `stream_ordering`/`bump_stamp` - [x] Handle remote invites - [x] Handle state resets - [x] Consider adding `sender` so we can filter `LEAVE` memberships and distinguish from kicks. - [x] We should add it to be able to tell leaves from kicks - [x] Consider adding `tombstone` state to help address https://github.com/element-hq/synapse/issues/17540 - [x] We should add it `tombstone_successor_room_id` - [x] Consider adding `forgotten` status to avoid extra lookup/table-join on `room_memberships` - [x] We should add it - [x] Background update to fill in values for all joined rooms and non-join membership - [x] Clean-up tables when room is deleted - [ ] Make sure tables are useful to our use case - First explored in https://github.com/element-hq/synapse/compare/erikj/ss_use_new_tables - Also explored in https://github.com/element-hq/synapse/commit/76b5a576eb363496315dfd39510cad7d02b0fc73 - [x] Plan for how can we use this with a fallback - See plan discussed above in main area of the issue description - Discussed in an [internal meeting](https://docs.google.com/document/d/1MnuvPkaCkT_wviSQZ6YKBjiWciCBFMd-7hxyCO-OCbQ/edit#bookmark=id.dz5x6ef4mxz7) - [x] Plan for how we can rely on this new table without a fallback - Synapse version `N+1`: (this PR) Bump `SCHEMA_VERSION` to `87`. Add new tables and background update to backfill all rows. Since this is a new table, we don't have to add any `NOT VALID` constraints and validate them when the background update completes. Read from new tables with a fallback in cases where the rows aren't filled in yet. - Synapse version `N+2`: Bump `SCHEMA_VERSION` to `88` and bump `SCHEMA_COMPAT_VERSION` to `87` because we don't want people to downgrade and miss writes while they are on an older version. Add a foreground update to finish off the backfill so we can read from new tables without the fallback. Application code can now rely on the new tables being populated. - Discussed in an [internal meeting](https://docs.google.com/document/d/1MnuvPkaCkT_wviSQZ6YKBjiWciCBFMd-7hxyCO-OCbQ/edit#bookmark=id.hh7shg4cxdhj) ### Dev notes ``` SYNAPSE_TEST_LOG_LEVEL=INFO poetry run trial tests.storage.test_events.SlidingSyncPrePopulatedTablesTestCase SYNAPSE_POSTGRES=1 SYNAPSE_POSTGRES_USER=postgres SYNAPSE_TEST_LOG_LEVEL=INFO poetry run trial tests.storage.test_events.SlidingSyncPrePopulatedTablesTestCase ``` ``` SYNAPSE_TEST_LOG_LEVEL=INFO poetry run trial tests.handlers.test_sliding_sync.FilterRoomsTestCase ``` Reference: - [Development docs on background updates and worked examples of gradual migrations ](https://github.com/element-hq/synapse/blob/1dfa59b238cee0dc62163588cc9481896c288979/docs/development/database_schema.md#background-updates) - A real example of a gradual migration: https://github.com/matrix-org/synapse/pull/15649#discussion_r1213779514 - Adding `rooms.creator` field that needed a background update to backfill data, https://github.com/matrix-org/synapse/pull/10697 - Adding `rooms.room_version` that needed a background update to backfill data, https://github.com/matrix-org/synapse/pull/6729 - Adding `room_stats_state.room_type` that needed a background update to backfill data, https://github.com/matrix-org/synapse/pull/13031 - Tables from MSC2716: `insertion_events`, `insertion_event_edges`, `insertion_event_extremities`, `batch_events` - `current_state_events` updated in `synapse/storage/databases/main/events.py` --- ``` persist_event (adds to queue) _persist_event_batch _persist_events_and_state_updates (assigns `stream_ordering` to events) _persist_events_txn _store_event_txn _update_metadata_tables_txn _store_room_members_txn _update_current_state_txn ``` --- > Concatenated Indexes [...] (also known as multi-column, composite or combined index) > > [...] key consists of multiple columns. > > We can take advantage of the fact that the first index column is always usable for searching > > *-- https://use-the-index-luke.com/sql/where-clause/the-equals-operator/concatenated-keys* --- Dealing with `portdb` (`synapse/_scripts/synapse_port_db.py`), https://github.com/element-hq/synapse/pull/17512#discussion_r1725998219 --- <details> <summary>SQL queries:</summary> Both of these are equivalent and work in SQLite and Postgres Options 1: ```sql WITH data_table (room_id, user_id, membership_event_id, membership, event_stream_ordering, {", ".join(insert_keys)}) AS ( VALUES ( ?, ?, ?, (SELECT membership FROM room_memberships WHERE event_id = ?), (SELECT stream_ordering FROM events WHERE event_id = ?), {", ".join("?" for _ in insert_values)} ) ) INSERT INTO sliding_sync_non_join_memberships (room_id, user_id, membership_event_id, membership, event_stream_ordering, {", ".join(insert_keys)}) SELECT * FROM data_table WHERE membership != ? ON CONFLICT (room_id, user_id) DO UPDATE SET membership_event_id = EXCLUDED.membership_event_id, membership = EXCLUDED.membership, event_stream_ordering = EXCLUDED.event_stream_ordering, {", ".join(f"{key} = EXCLUDED.{key}" for key in insert_keys)} ``` Option 2: ```sql INSERT INTO sliding_sync_non_join_memberships (room_id, user_id, membership_event_id, membership, event_stream_ordering, {", ".join(insert_keys)}) SELECT column1 as room_id, column2 as user_id, column3 as membership_event_id, column4 as membership, column5 as event_stream_ordering, {", ".join("column" + str(i) for i in range(6, 6 + len(insert_keys)))} FROM ( VALUES ( ?, ?, ?, (SELECT membership FROM room_memberships WHERE event_id = ?), (SELECT stream_ordering FROM events WHERE event_id = ?), {", ".join("?" for _ in insert_values)} ) ) as v WHERE membership != ? ON CONFLICT (room_id, user_id) DO UPDATE SET membership_event_id = EXCLUDED.membership_event_id, membership = EXCLUDED.membership, event_stream_ordering = EXCLUDED.event_stream_ordering, {", ".join(f"{key} = EXCLUDED.{key}" for key in insert_keys)} ``` If we don't need the `membership` condition, we could use: ```sql INSERT INTO sliding_sync_non_join_memberships (room_id, membership_event_id, user_id, membership, event_stream_ordering, {", ".join(insert_keys)}) VALUES ( ?, ?, ?, (SELECT membership FROM room_memberships WHERE event_id = ?), (SELECT stream_ordering FROM events WHERE event_id = ?), {", ".join("?" for _ in insert_values)} ) ON CONFLICT (room_id, user_id) DO UPDATE SET membership_event_id = EXCLUDED.membership_event_id, membership = EXCLUDED.membership, event_stream_ordering = EXCLUDED.event_stream_ordering, {", ".join(f"{key} = EXCLUDED.{key}" for key in insert_keys)} ``` </details> ### Pull Request Checklist <!-- Please read https://element-hq.github.io/synapse/latest/development/contributing_guide.html before submitting your pull request --> * [x] Pull request is based on the develop branch * [x] Pull request includes a [changelog file](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#changelog). The entry should: - Be a short description of your change which makes sense to users. "Fixed a bug that prevented receiving messages from other servers." instead of "Moved X method from `EventStore` to `EventWorkerStore`.". - Use markdown where necessary, mostly for `code blocks`. - End with either a period (.) or an exclamation mark (!). - Start with a capital letter. - Feel free to credit yourself, by adding a sentence "Contributed by @github_username." or "Contributed by [Your Name]." to the end of the entry. * [x] [Code style](https://element-hq.github.io/synapse/latest/code_style.html) is correct (run the [linters](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#run-the-linters)) --------- Co-authored-by: Erik Johnston <erik@matrix.org>
* Sliding sync: Always send your own receipts down (#17617)Erik Johnston2024-08-291-0/+132
| | | | | | | | | | | When returning receipts in sliding sync for initial rooms we should always include our own receipts in the room (even if they don't match any timeline events). Reviewable commit-by-commit. --------- Co-authored-by: Eric Eastwood <eric.eastwood@beta.gouv.fr>
* Sliding Sync: Handle timeline limit changes (take 2) (#17579)Erik Johnston2024-08-201-0/+136
| | | | | | | | | | | | | | | This supersedes #17503, given the per-connection state is being heavily rewritten it felt easier to recreate the PR on top of that work. This correctly handles the case of timeline limits going up and down. This does not handle changes in `required_state`, but that can be done as a separate PR. Based on #17575. --------- Co-authored-by: Eric Eastwood <eric.eastwood@beta.gouv.fr>
* Sliding sync: Correctly track which read receipts we have or have not sent ↵Erik Johnston2024-08-193-4/+128
| | | | | | | | | | | down. (#17575) Add connection tracking to the receipts extension. Based on #17574 --------- Co-authored-by: Eric Eastwood <eric.eastwood@beta.gouv.fr>
* Sliding Sync: Exclude partially stated rooms if we must await full state ↵Eric Eastwood2024-08-131-30/+165
| | | | | | | | | | | | | (#17538) Previously, we just had very basic partial room exclusion based on whether we were lazy-loading room members. Now with this PR, we added `must_await_full_state(...)` with rules to check if we have a we're only requesting `required_state` which is completely satisfied even with partial state. Partially-stated rooms should have all state events except for remote membership events so if we require a remote membership event anywhere, then we need to return `True`.
* SSS: Implement PREVIOUSLY room tracking (#17535)Erik Johnston2024-08-081-72/+0
| | | | | | Implement tracking of rooms that have had updates that have not been sent down to clients. Simplified Sliding Sync (SSS)
* SS: Reset connection if token is unrecognized (#17529)Erik Johnston2024-08-061-17/+11
| | | | | | | | | | | | | | This triggers the client to start a new sliding sync connection. If we don't do this and the client asks for the full range of rooms, we end up sending down all rooms and their state from scratch (which can be very slow) This causes things like https://github.com/element-hq/element-x-ios/issues/3115 after we restart the server --------- Co-authored-by: Eric Eastwood <eric.eastwood@beta.gouv.fr>
* Sliding Sync: Easier to understand timeline assertions in tests (#17511)Eric Eastwood2024-08-051-21/+103
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Added `_assertTimelineEqual(...)` because I got fed up trying to understand the crazy diffs from the standard `self.assertEqual(...)`/`self.assertListEqual(...)` Before: ``` [FAIL] Traceback (most recent call last): File "/home/eric/Documents/github/element/synapse/tests/rest/client/sliding_sync/test_rooms_timeline.py", line 103, in test_rooms_limited_initial_sync self.assertListEqual( File "/usr/lib/python3.12/unittest/case.py", line 1091, in assertListEqual self.assertSequenceEqual(list1, list2, msg, seq_type=list) File "/usr/lib/python3.12/unittest/case.py", line 1073, in assertSequenceEqual self.fail(msg) twisted.trial.unittest.FailTest: Lists differ: ['$4QcmnzhdazSnDYcYSZCS_6-MWSzM_dN3RC7TRvW0w[95 chars]isM'] != ['$8N1XJ7e-3K_wxAanLVD3v8KQ96_B5Xj4huGkgy4N4[95 chars]nnU'] First differing element 0: '$4QcmnzhdazSnDYcYSZCS_6-MWSzM_dN3RC7TRvW0wWA' '$8N1XJ7e-3K_wxAanLVD3v8KQ96_B5Xj4huGkgy4N4-E' - ['$4QcmnzhdazSnDYcYSZCS_6-MWSzM_dN3RC7TRvW0wWA', - '$8N1XJ7e-3K_wxAanLVD3v8KQ96_B5Xj4huGkgy4N4-E', ? ^ + ['$8N1XJ7e-3K_wxAanLVD3v8KQ96_B5Xj4huGkgy4N4-E', ? ^ - '$q4PRxQ_pBZkQI1keYuZPTtExQ23DqpUI3-Lxwfj_isM'] + '$4QcmnzhdazSnDYcYSZCS_6-MWSzM_dN3RC7TRvW0wWA', + '$j3Xj-t2F1wH9kUHsI8X5yqS7hkdSyN2owaArfvk8nnU'] ``` After: ``` [FAIL] Traceback (most recent call last): File "/home/eric/Documents/github/element/synapse/tests/rest/client/sliding_sync/test_rooms_timeline.py", line 178, in test_rooms_limited_initial_sync self._assertTimelineEqual( File "/home/eric/Documents/github/element/synapse/tests/rest/client/sliding_sync/test_rooms_timeline.py", line 110, in _assertTimelineEqual self._assertListEqual( File "/home/eric/Documents/github/element/synapse/tests/rest/client/sliding_sync/test_rooms_timeline.py", line 79, in _assertListEqual self.fail(f"{diff_message}\n{message}") twisted.trial.unittest.FailTest: Items must Expected items to be in actual ('?' = missing expected items): [ (10, master) $w-BoqW1PQQFU4TzVJW5OIelugxh0mY12wrfw6mbC6D4 (m.room.message) activity4 (11, master) $sSidTZf1EOQmCVDU4mrH_1-bopMQhwcDUO2IhoemR6M (m.room.message) activity5 ? (12, master) $bgOcc3D-2QSkbk4aBxKVyOOQJGs7ZuncRJwG3cEANZg (m.room.member, @user1:test) join ] Actual ('+' = found expected items): [ + (11, master) $sSidTZf1EOQmCVDU4mrH_1-bopMQhwcDUO2IhoemR6M (m.room.message) activity5 + (10, master) $w-BoqW1PQQFU4TzVJW5OIelugxh0mY12wrfw6mbC6D4 (m.room.message) activity4 (9, master) $FmCNyc11YeFwiJ4an7_q6H0LCCjQOKd6UCr5VKeXXUw (m.room.message, None) activity3 ] ```
* Sliding Sync: Add typing notification extension (MSC3961) (#17505)Eric Eastwood2024-07-312-3/+501
| | | | | | | [MSC3961](https://github.com/matrix-org/matrix-spec-proposals/pull/3961): Sliding Sync Extension: Typing Notifications Based on [MSC3575](https://github.com/matrix-org/matrix-spec-proposals/pull/3575): Sliding Sync
* Sliding Sync: Split and move tests (#17504)Eric Eastwood2024-07-3114-5860/+6314
| | | | Split and move Sliding Sync tests so we have some more sane test file sizes
* Sliding Sync: Update filters to be robust against remote invite rooms (#17450)Eric Eastwood2024-07-301-0/+145
| | | | | | | | | | | | | | | Update `filters.is_encrypted` and `filters.types`/`filters.not_types` to be robust when dealing with remote invite rooms in Sliding Sync. Part of [MSC3575](https://github.com/matrix-org/matrix-spec-proposals/pull/3575): Sliding Sync Follow-up to https://github.com/element-hq/synapse/pull/17434 We now take into account current state, fallback to stripped state for invite/knock rooms, then historical state. If we can't determine the info needed to filter a room (either from state or stripped state), it is filtered out.
* Sliding Sync: Add receipts extension (MSC3960) (#17489)Eric Eastwood2024-07-301-205/+816
| | | | | | | [MSC3960](https://github.com/matrix-org/matrix-spec-proposals/pull/3960): Receipts extension Based on [MSC3575](https://github.com/matrix-org/matrix-spec-proposals/pull/3575): Sliding Sync
* Merge branch 'master' into developOlivier 'reivilibre2024-07-301-3/+2
|\
| * Upgrade locked dependency on Twisted to 24.7.0rc1. (#17502)reivilibre2024-07-301-3/+2
| | | | | | | | | | | | | | | | | | | | I also update the tests and HTTP Proxy code to fix it for this new Twisted release. Pulls in fix for https://github.com/twisted/twisted/security/advisories/GHSA-c8m8-j448-xjx7 Signed-off-by: Olivier 'reivilibre <oliverw@matrix.org>
* | Only send rooms with updates down sliding sync (#17479)Erik Johnston2024-07-301-27/+62
| | | | | | | | | | | | | | | | | | | | | | | | Rather than always including all rooms in range. Also adds a pre-filter to rooms that checks the stream change cache to see if anything might have happened. Based on #17447 --------- Co-authored-by: Eric Eastwood <eric.eastwood@beta.gouv.fr>
* | Sliding Sync: Track whether we have sent rooms down to clients (#17447)Erik Johnston2024-07-291-3/+449
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The basic idea is that we introduce a new token for a sliding sync connection, which stores the mapping of room to room "status" (i.e. have we sent the room down?). This token allows us to handle duplicate requests properly. In future it can be used to store more "per-connection" information safely. In future this should be migrated into the DB, so its important that we try to reduce the number of syncs where we need to update the per-connection information. In this PoC this only happens when we: a) send down a set of room for the first time, or b) we have previously sent down a room and there are updates but we are not sending the room down the sync (due to not falling in a list range) Co-authored-by: Eric Eastwood <eric.eastwood@beta.gouv.fr>
* | Refactor Sliding Sync tests to better utilize the ↵Eric Eastwood2024-07-251-1021/+725
| | | | | | | | | | | | | | | | `SlidingSyncBase.do_sync(...)` (pt. 2) (#17482) `SlidingSyncBase.do_sync()` for tests was first introduced in https://github.com/element-hq/synapse/pull/17452 Part 1: https://github.com/element-hq/synapse/pull/17481
* | Refactor Sliding Sync tests to better utilize the `SlidingSyncBase` (pt. 1) ↵Eric Eastwood2024-07-251-228/+106
| | | | | | | | | | | | | | | | (#17481) `SlidingSyncBase` for tests was first introduced in https://github.com/element-hq/synapse/pull/17452 Part 2: https://github.com/element-hq/synapse/pull/17482
* | Sliding Sync: Add Account Data extension (MSC3959) (#17477)Eric Eastwood2024-07-241-0/+786
| | | | | | | | | | | | Extensions based on [MSC3575](https://github.com/matrix-org/matrix-spec-proposals/pull/3575): Sliding Sync
* | Sliding Sync: ensure bump stamp ignores backfilled events (#17478)Erik Johnston2024-07-241-1/+121
| | | | | | | | | | | | Backfill events have a negative stream ordering, and so its not useful to use to compare with other (positive) stream orderings. Plus, the Rust SDK currently assumes `bump_stamp` is positive.
* | Use a new token format for sliding sync (#17452)Erik Johnston2024-07-241-191/+225
| | | | | | | | | | | | | | This is in preparation for adding per-connection state. --------- Co-authored-by: Eric Eastwood <eric.eastwood@beta.gouv.fr>
* | Sliding Sync: Add E2EE extension (MSC3884) (#17454)Eric Eastwood2024-07-221-4/+821
| | | | | | | | | | Spec: [MSC3884](https://github.com/matrix-org/matrix-spec-proposals/pull/3884) Based on [MSC3575](https://github.com/matrix-org/matrix-spec-proposals/pull/3575): Sliding Sync
* | SS: Implement `$ME` support (#17469)Erik Johnston2024-07-221-0/+74
| | | | | | `$ME` can be used as a substitute for the requester's user ID.
* | Prepare for authenticated media freeze (#17433)Shay2024-07-221-0/+209
| | | | | | | | | | | | As part of the rollout of [MSC3916](https://github.com/matrix-org/matrix-spec-proposals/blob/main/proposals/3916-authentication-for-media.md) this PR adds support for designating authenticated media and ensuring that authenticated media is not served over unauthenticated endpoints.
* | Order `heroes` by `stream_ordering` (as spec'ed) (#17435)Eric Eastwood2024-07-171-12/+9
| | | | | | | | | | | | | | | | | | | | | | | | | | The spec specifically mentions `stream_ordering` but that's a Synapse specific concept. In any case, the essence of the spec is basically the first 5 members of the room which `stream_ordering` accomplishes. Split off from https://github.com/element-hq/synapse/pull/17419#discussion_r1671342794 ## Spec compliance > This should be the first 5 members of the room, **ordered by stream ordering**, which are joined or invited. The list must never include the client’s own user ID. When no joined or invited members are available, this should consist of the banned and left users. > > *-- https://spec.matrix.org/v1.10/client-server-api/#_matrixclientv3sync_roomsummary* Related to https://github.com/matrix-org/matrix-spec/issues/1334
* | Handle remote download responses with `UNKNOWN_LENGTH` more gracefully (#17439)Shay2024-07-161-5/+45
| | | | | | | | | | | | | | | | | | | | | | | | | | | | Prior to this PR, remote downloads which did not provide a `content-length` were decremented from the remote download ratelimiter at the max allowable size, leading to excessive ratelimiting - see https://github.com/element-hq/synapse/issues/17394. This PR adds a linearizer to limit concurrent remote downloads to 6 per IP address, and decrements remote downloads without a `content-length` from the ratelimiter *after* the download is complete and the response length is known. Also adds logic to ensure that responses with a known length respect the `max_download_size`.
* | Add room subscriptions to Sliding Sync `/sync` (#17432)Eric Eastwood2024-07-151-63/+284
| | | | | | | | | | | | | | | | | | | | | | | | | | Add room subscriptions to Sliding Sync `/sync` Based on [MSC3575](https://github.com/matrix-org/matrix-spec-proposals/pull/3575): Sliding Sync Currently, you can only subscribe to rooms you have had *any* membership in before. In the future, we will allow `world_readable` rooms to be subscribed to without joining.
* | Add `is_dm` room field to Sliding Sync `/sync` (#17429)Eric Eastwood2024-07-111-0/+23
| | | | | | | | | | Based on [MSC3575](https://github.com/matrix-org/matrix-spec-proposals/pull/3575): Sliding Sync
* | Add `heroes` and room summary fields to Sliding Sync `/sync` (#17419)Eric Eastwood2024-07-111-12/+292
| | | | | | | | | | | | | | Additional room summary fields: `joined_count`, `invited_count` Based on [MSC3575](https://github.com/matrix-org/matrix-spec-proposals/pull/3575): Sliding Sync
* | Handle to-device extensions to Sliding Sync (#17416)Erik Johnston2024-07-101-2/+198
| | | | | | | | | | | | | | Implements MSC3885 --------- Co-authored-by: Eric Eastwood <eric.eastwood@beta.gouv.fr>
* | Add `rooms` `name` and `avatar` to Sliding Sync `/sync` (#17418)Eric Eastwood2024-07-091-0/+208
|/ | | Based on [MSC3575](https://github.com/matrix-org/matrix-spec-proposals/pull/3575): Sliding Sync
* Add `rooms.bump_stamp` to Sliding Sync `/sync` for easier client-side ↵Eric Eastwood2024-07-081-0/+96
| | | | | | | | | | | | | | | sorting (#17395) `bump_stamp` corresponds to the `stream_ordering` of the latest `DEFAULT_BUMP_EVENT_TYPES` in the room. This helps clients sort more readily without them needing to pull in a bunch of the timeline to determine the last activity. `bump_event_types` is a thing because for example, we don't want display name changes to mark the room as unread and bump it to the top. For encrypted rooms, we just have to consider any activity as a bump because we can't see the content and the client has to figure it out for themselves. Outside of Synapse, `bump_stamp` is just a free-form counter so other implementations could use `received_ts`or `origin_server_ts` (see the [*Security considerations* section in MSC3575 about the potential pitfalls of using `origin_server_ts`](https://github.com/matrix-org/matrix-spec-proposals/blob/kegan/sync-v3/proposals/3575-sync.md#security-considerations)). It doesn't have any guarantee about always going up. In the Synapse case, it could go down if an event was redacted/removed (or purged in cases of retention policies). In the future, we could add `bump_event_types` as [MSC3575](https://github.com/matrix-org/matrix-spec-proposals/pull/3575) mentions if people need to customize the event types. --- In the Sliding Sync proxy, a similar [`timestamp` field was added](https://github.com/matrix-org/sliding-sync/pull/247) for the same purpose but the name is not obvious what it pertains to or what it's for. The `timestamp` field was also added to Ruma in https://github.com/ruma/ruma/pull/1622
* Support MSC3916 by adding a federation /thumbnail endpoint and authenticated ↵Shay2024-07-081-71/+287
| | | | | | | | | | | | | | `_matrix/client/v1/media/thumbnail` endpoint (#17388) [MSC3916](https://github.com/matrix-org/matrix-spec-proposals/pull/3916) added the endpoints `_matrix/federation/v1/media/thumbnail` and the authenticated `_matrix/client/v1/media/thumbnail`. This PR implements those endpoints, along with stabilizing `_matrix/client/v1/media/config` and `_matrix/client/v1/media/preview_url`. Complement tests are at https://github.com/matrix-org/complement/pull/728
* Add `rooms.required_state` to Sliding Sync `/sync` (#17342)Eric Eastwood2024-07-041-1/+712
| | | Also handles excluding rooms with partial state when people are asking for room membership events unless it's `$LAZY` room membership.
* Return some room data in Sliding Sync `/sync` (#17320)Eric Eastwood2024-07-022-10/+1073
| | | | | | - Timeline events - Stripped `invite_state` Based on [MSC3575](https://github.com/matrix-org/matrix-spec-proposals/pull/3575): Sliding Sync
* Support MSC3916 by adding `_matrix/client/v1/media/download` endpoint (#17365)Shay2024-07-021-3/+606
|
* Fix sync waiting for an invalid token from the "future" (#17386)Erik Johnston2024-07-021-1/+3
| | | | | | | | | | | | | Fixes https://github.com/element-hq/synapse/issues/17274, hopefully. Basically, old versions of Synapse could advance streams without persisting anything in the DB (fixed in #17229). On restart those updates would get lost, and so the position of the stream would revert to an older position. If this happened across an upgrade to a later Synapse version which included #17215, then sync could get blocked indefinitely (until the stream advanced to the position in the token). We fix this by bounding the stream positions we'll wait for to the maximum position of the underlying stream ID generator.
* Add `is_invite` filtering to Sliding Sync `/sync` (#17335)Eric Eastwood2024-06-241-25/+123
| | | Based on [MSC3575](https://github.com/matrix-org/matrix-spec-proposals/pull/3575): Sliding Sync
* Add support for MSC3823 - Account Suspension Part 2 (#17255)Shay2024-06-241-0/+105
|
* Register sliding sync under a different path (#17331)Erik Johnston2024-06-191-1/+3
| | | As the API is slightly incompatible.
* Require the 'from' parameter for `/notifications` be an integer (#17283)Andrew Morgan2024-06-191-18/+153
| | | | Co-authored-by: Erik Johnston <erikj@element.io>
* Add `stream_ordering` sort to Sliding Sync `/sync` (#17293)Eric Eastwood2024-06-171-4/+57
| | | Sort is no longer configurable and we always sort rooms by the `stream_ordering` of the last event in the room or the point where the user can see up to in cases of leave/ban/invite/knock.
* Enable cross-signing key upload without UIA (#17284)Richard van der Hoff2024-06-141-65/+0
| | | | | | Per MSC3967, which is now stable, we should not require UIA when uploading cross-signing keys for the first time. Fixes: #17227
* Include user membership on events (#17282)Richard van der Hoff2024-06-131-1/+0
| | | | | MSC4115 has now completed FCP, so we can enable it by default and switch to the stable identifier.
* Add `is_dm` filtering to Sliding Sync `/sync` (#17277)Eric Eastwood2024-06-131-0/+127
| | | Based on [MSC3575](https://github.com/matrix-org/matrix-spec-proposals/pull/3575): Sliding Sync
* Bump `mypy` from 1.8.0 to 1.9.0 (#17297)Andrew Morgan2024-06-131-4/+24
| | | | Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* Add report room API (MSC4151) (#17270)Travis Ralston2024-06-121-2/+91
| | | | | | | | https://github.com/matrix-org/matrix-spec-proposals/pull/4151 This is intended to be enabled by default for immediate use. When FCP is complete, the unstable endpoint will be dropped and stable endpoint supported instead - no backwards compatibility is expected for the unstable endpoint.
* Reorganize Pydantic models and types used in handlers (#17279)Eric Eastwood2024-06-101-1/+1
| | | | | | Spawning from https://github.com/element-hq/synapse/pull/17187#discussion_r1619492779 around wanting to put `SlidingSyncBody` (parse the request in the rest layer), `SlidingSyncConfig` (from the rest layer, pass to the handler), `SlidingSyncResponse` (pass the response from the handler back to the rest layer to respond) somewhere that doesn't contaminate the imports and cause circular import issues. - Moved Pydantic parsing models to `synapse/types/rest` - Moved handler types to `synapse/types/handlers`
* Add Sliding Sync `/sync` endpoint (initial implementation) (#17187)Eric Eastwood2024-06-062-2/+137
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Based on [MSC3575](https://github.com/matrix-org/matrix-spec-proposals/pull/3575): Sliding Sync This iteration only focuses on returning the list of room IDs in the sliding window API (without sorting/filtering). Rooms appear in the Sliding sync response based on: - `invite`, `join`, `knock`, `ban` membership events - Kicks (`leave` membership events where `sender` is different from the `user_id`/`state_key`) - `newly_left` (rooms that were left during the given token range, > `from_token` and <= `to_token`) - In order for bans/kicks to not show up, you need to `/forget` those rooms. This doesn't modify the event itself though and only adds the `forgotten` flag to `room_memberships` in Synapse. There isn't a way to tell when a room was forgotten at the moment so we can't factor it into the from/to range. ### Example request `POST http://localhost:8008/_matrix/client/unstable/org.matrix.msc3575/sync` ```json { "lists": { "foo-list": { "ranges": [ [0, 99] ], "sort": [ "by_notification_level", "by_recency", "by_name" ], "required_state": [ ["m.room.join_rules", ""], ["m.room.history_visibility", ""], ["m.space.child", "*"] ], "timeline_limit": 100 } } } ``` Response: ```json { "next_pos": "s58_224_0_13_10_1_1_16_0_1", "lists": { "foo-list": { "count": 1, "ops": [ { "op": "SYNC", "range": [0, 99], "room_ids": [ "!MmgikIyFzsuvtnbvVG:my.synapse.linux.server" ] } ] } }, "rooms": {}, "extensions": {} } ```
* Fix slipped logging context when media rejected (#17239)Erik Johnston2024-05-291-7/+7
| | | | | | | When a module rejects a piece of media we end up trying to close the same logging context twice. Instead of fixing the existing code we refactor to use an async context manager, which is easier to write correctly.
* Support MSC3916 by adding unstable media endpoints to `_matrix/client` (#17213)Shay2024-05-241-0/+1609
| | | | | | | | | | [MSC3916](https://github.com/matrix-org/matrix-spec-proposals/blob/rav/authentication-for-media/proposals/3916-authentication-for-media.md) adds new media endpoints under `_matrix/client`. This PR adds the `/preview_url`, `/config`, and `/thumbnail` endpoints. `/download` will be added in a follow-up PR once the work for the federation `/download` endpoint is complete (see https://github.com/element-hq/synapse/pull/17172). Should be reviewable commit-by-commit.
* Add Sliding Sync `/sync/e2ee` endpoint for To-Device messages (#17167)Eric Eastwood2024-05-233-165/+449
| | | | | | | | | | | This is being introduced as part of Sliding Sync but doesn't have any sliding window component. It's just a way to get E2EE events without having to sit through a big initial sync (`/sync` v2). And we can avoid encryption events being backed up by the main sync response or vice-versa. Part of some Sliding Sync simplification/experimentation. See [this discussion](https://github.com/element-hq/synapse/pull/17167#discussion_r1610495866) for why it may not be as useful as we thought. Based on: - https://github.com/matrix-org/matrix-spec-proposals/pull/3575 - https://github.com/matrix-org/matrix-spec-proposals/pull/3885 - https://github.com/matrix-org/matrix-spec-proposals/pull/3884
* Bring auto-accept invite logic into Synapse (#17147)devonh2024-05-211-0/+2
| | | | | | | | | | | | | | This PR ports the logic from the [synapse_auto_accept_invite](https://github.com/matrix-org/synapse-auto-accept-invite) module into synapse. I went with the naive approach of injecting the "module" next to where third party modules are currently loaded. If there is a better/preferred way to handle this, I'm all ears. It wasn't obvious to me if there was a better location to add this logic that would cleanly apply to all incoming invite events. Relies on https://github.com/element-hq/synapse/pull/17166 to fix linter errors.
* Add support for MSC3823 - Account Suspension (#17051)Shay2024-05-011-3/+66
|
* Apply user `email` & `picture` during OIDC registration if present & ↵devonh2024-04-291-14/+190
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | selected (#17120) This change will apply the `email` & `picture` provided by OIDC to the new user account when registering a new user via OIDC. If the user is directed to the account details form, this change makes sure they have been selected before applying them, otherwise they are omitted. In particular, this change ensures the values are carried through when Synapse has consent configured, and the redirect to the consent form/s are followed. I have tested everything manually. Including: - with/without consent configured - allowing/not allowing the use of email/avatar (via `sso_auth_account_details.html`) - with/without automatic account detail population (by un/commenting the `localpart_template` option in synapse config). ### Pull Request Checklist <!-- Please read https://element-hq.github.io/synapse/latest/development/contributing_guide.html before submitting your pull request --> * [X] Pull request is based on the develop branch * [X] Pull request includes a [changelog file](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#changelog). The entry should: - Be a short description of your change which makes sense to users. "Fixed a bug that prevented receiving messages from other servers." instead of "Moved X method from `EventStore` to `EventWorkerStore`.". - Use markdown where necessary, mostly for `code blocks`. - End with either a period (.) or an exclamation mark (!). - Start with a capital letter. - Feel free to credit yourself, by adding a sentence "Contributed by @github_username." or "Contributed by [Your Name]." to the end of the entry. * [X] [Code style](https://element-hq.github.io/synapse/latest/code_style.html) is correct (run the [linters](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#run-the-linters))
* Add support for MSC4115 (#17104)Richard van der Hoff2024-04-291-1/+6
| | | | Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
* MSC4108 implementation (#17056)Quentin Gliech2024-04-251-1/+400
| | | | | | Co-authored-by: Hugh Nimmo-Smith <hughns@element.io> Co-authored-by: Hugh Nimmo-Smith <hughns@users.noreply.github.com> Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
* Send an email if the address is already bound to an user account (#16819)mcalinghee2024-04-231-0/+9
| | | | Co-authored-by: Mathieu Velten <mathieu.velten@beta.gouv.fr> Co-authored-by: Olivier D <odelcroi@gmail.com>
* Parse json validation (#16923)Gordan Trevis2024-04-181-0/+52
| | | | Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
* Support for MSC4108 via delegation (#17086)Quentin Gliech2024-04-171-4/+30
| | | | | | | This adds support for MSC4108 via delegation, similar to what has been done for MSC3886 --------- Co-authored-by: Hugh Nimmo-Smith <hughns@element.io>
* Stabliize support for MSC3981: recurse /relations (#17023)Patrick Cloke2024-04-091-6/+3
| | | | | | | See [MSC3981](https://github.com/matrix-org/matrix-spec-proposals/pull/3981), this pretty much just removes flags though. Part of #17021
* Fix bug in `/sync` response for archived rooms (#16932)Richard van der Hoff2024-04-041-7/+11
| | | | | | | | | | | | This PR fixes a very, very niche edge-case, but I've got some more work coming which will otherwise make the problem worse. The bug happens when the syncing user leaves a room, and has a sync filter which includes "left" rooms, but sets the timeline limit to 0. In that case, the state returned in the `state` section is calculated incorrectly. The fix is to pass a token corresponding to the point that the user leaves the room through to `compute_state_delta`.
* Do not refuse to set read_marker if previous event_id is in wrong room (#16990)SpiritCroc2024-03-211-4/+4
|
* Update power level default for public rooms (#16907)Shay2024-03-191-0/+18
|
* Bump mypy from 1.5.1 to 1.8.0 (#16901)dependabot[bot]2024-03-131-1/+1
|
* Bump black from 23.10.1 to 24.2.0 (#16936)dependabot[bot]2024-03-132-10/+8
|
* Stabilize support for Retry-After header (MSC4014) (#16947)Patrick Cloke2024-03-081-3/+0
|
* Correctly mention previous copyright (#16820)Erik Johnston2024-01-2333-0/+38
| | | | | During the migration the automated script to update the copyright headers accidentally got rid of some of the existing copyright lines. Reinstate them.
* Handle wildcard type filters properly (#14984)Mo Balaa2024-01-222-1/+29
|
* Fix email verification redirection (#16761)FadhlanR2024-01-021-3/+40
| | | | | | Previously, the response status of `HTMLResource` was hardcoded as `200`. However, for proper redirection after the user verifies their email, we require the status to be `302`. This PR addresses that issue by using `code` as response status.
* Merge remote-tracking branch 'gitlab/clokep/license-license' into new_developErik Johnston2023-12-1340-412/+640
|\
| * Update license headersPatrick Cloke2023-11-2140-412/+640
| |
* | Expose OIDC discovery information under the CSAPI (#16726)David Robertson2023-12-061-0/+59
| | | | | | | | Co-authored-by: Quentin Gliech <quenting@element.io>
* | Fix upgrading a room without `events` field in power levels (#16725)David Robertson2023-12-051-0/+28
| |
* | Implement MSC4069: Inhibit profile propagation (#16636)Travis Ralston2023-12-041-0/+160
| | | | | | MSC: https://github.com/matrix-org/matrix-spec-proposals/pull/4069
* | Keep track of `user_ips` and `monthly_active_users` when delegating auth ↵David Robertson2023-11-231-7/+1
|/ | | | | | | | | | | | | (#16672) * Describe `insert_client_ip` * Pull out client_ips and MAU tracking to BaseAuth * Define HAS_AUTHLIB once in tests sick of copypasting * Track ips and token usage when delegating auth * Test that we track MAU and user_ips * Don't track `__oidc_admin`
* Add an Admin API to temporarily grant the ability to update an existing ↵David Robertson2023-11-151-2/+186
| | | | cross-signing key without UIA (#16634)
* Use full GitHub links instead of bare issue numbers. (#16637)Patrick Cloke2023-11-154-5/+8
|
* Convert simple_select_one_txn and simple_select_one to return tuples. (#16612)Patrick Cloke2023-11-092-10/+10
|
* Add new module API for adding custom fields to events `unsigned` section ↵Erik Johnston2023-10-271-1/+1
| | | | (#16549)
* Add a new module API to update user presence state. (#16544)Patrick Cloke2023-10-261-2/+17
| | | | | | | | | | This adds a module API which allows a module to update a user's presence state/status message. This is useful for controlling presence from an external system. To fully control presence from the module the presence.enabled config parameter gains a new state of "untracked" which disables internal tracking of presence changes via user actions, etc. Only updates from the module will be persisted and sent down sync properly).
* Apply join rate limiter outside the lineariser (#16441)David Robertson2023-10-061-0/+24
|
* Register media servlets via regex. (#16419)Patrick Cloke2023-10-061-4/+2
| | | | | This converts the media servlet URLs in the same way as (most) of the rest of Synapse. This will give more flexibility in the versions each endpoint exists under.
* Add support for pydantic v2 via pydantic.v1 compat module (#16332)Maxwell G2023-09-251-1/+7
| | | While maintaining support with pydantic v1.
* Additional validation of receipts (#16327)Patrick Cloke2023-09-182-161/+214
| | | | Reject invalid receipts with a reasonable error message & expands tests for receipts.
* Add automatic purge after all users forget a room (#15488)Mathieu Velten2023-09-151-5/+1
| | | | | | Also add restore of purge/shutdown rooms after a synapse restart. Co-authored-by: Eric Eastwood <erice@matrix.org> Co-authored-by: Erik Johnston <erikj@matrix.org>
* Update ruff config (#16283)Patrick Cloke2023-09-083-8/+8
| | | Enable additional checks & clean-up unneeded configuration.
* Avoid temporary storage of sensitive information. (#16272)Patrick Cloke2023-09-081-0/+13
| | | | During the UI auth process, avoid storing sensitive information into the database.
* Bump mypy-zope & mypy. (#16188)Patrick Cloke2023-08-296-12/+12
|
* Prepare unit tests for Python 3.12 (#16099)V024602023-08-253-25/+28
|
* Replace simple_async_mock with AsyncMock (#16180)Patrick Cloke2023-08-251-3/+2
| | | | Python 3.8 has a native AsyncMock, use it instead of a custom implementation.
* Replace make_awaitable with AsyncMock (#16179)Patrick Cloke2023-08-246-53/+47
| | | | Python 3.8 provides a native AsyncMock, we can replace the homegrown version we have.
* Add `Retry-After` to M_LIMIT_EXCEEDED error responses (#16136)Will Hunt2023-08-241-6/+18
| | | Implements MSC4041 behind an experimental configuration flag.
* Run pyupgrade for python 3.7 & 3.8. (#16110)Patrick Cloke2023-08-153-27/+27
|
* Support MSC3814: Dehydrated Devices Part 2 (#16010)Shay2023-08-081-3/+74
|
* Fix deletion for Dehydrated Devices (#16046)Shay2023-08-041-1/+138
|
* Add forward-compatibility for the redacts property (MSC2174). (#16013)Patrick Cloke2023-08-021-17/+50
| | | | | | | | The location of the redacts field changes in room version 11. Ensure it is copied to the *new* location for *old* room versions for forwards-compatibility with clients. Note that copying it to the *old* location for the *new* room version was previously handled.
* Trim whitespace when setting display names (#16031)Mohit Rathee2023-08-011-0/+12
|
* Add ability to wait for locks and add locks to purge history / room deletion ↵Erik Johnston2023-07-311-2/+2
| | | | | (#15791) c.f. #13476
* Support MSC3814: Dehydrated Devices (#15929)Shay2023-07-241-2/+148
| | | | | | Signed-off-by: Nicolas Werner <n.werner@famedly.com> Co-authored-by: Nicolas Werner <n.werner@famedly.com> Co-authored-by: Nicolas Werner <89468146+nico-famedly@users.noreply.github.com> Co-authored-by: Hubert Chathi <hubert@uhoreg.ca>
* Support room version 11 (#15912)Patrick Cloke2023-07-181-2/+19
| | | | | And fix a bug in the implementation of the updated redaction format (MSC2174) where the top-level redacts field was not properly added for backwards-compatibility.
* Re-introduce the outbound federation proxy (#15913)Eric Eastwood2023-07-182-3/+0
| | | | | Allow configuring the set of workers to proxy outbound federation traffic through (`outbound_federation_restricted_to`). This is useful when you have a worker setup with `federation_sender` instances responsible for sending outbound federation requests and want to make sure *all* outbound federation traffic goes through those instances. Before this change, the generic workers would still contact federation themselves for things like profile lookups, backfill, etc. This PR allows you to set more strict access controls/firewall for all workers and only allow the `federation_sender`'s to contact the outside world.
* Revert "Federation outbound proxy" (#15910)Eric Eastwood2023-07-102-0/+3
| | | | | | Revert "Federation outbound proxy (#15773)" This reverts commit b07b14b494ae1dd564b4c44f844c9a9545b3d08a.
* Federation outbound proxy (#15773)Eric Eastwood2023-07-052-3/+0
| | | | | | | Allow configuring the set of workers to proxy outbound federation traffic through (`outbound_federation_restricted_to`). This is useful when you have a worker setup with `federation_sender` instances responsible for sending outbound federation requests and want to make sure *all* outbound federation traffic goes through those instances. Before this change, the generic workers would still contact federation themselves for things like profile lookups, backfill, etc. This PR allows you to set more strict access controls/firewall for all workers and only allow the `federation_sender`'s to contact the outside world. The original code is from @erikjohnston's branches which I've gotten in-shape to merge.
* Add login spam checker API (#15838)Erik Johnston2023-06-261-2/+106
|
* Remove experimental MSC2716 implementation to incrementally import history ↵Eric Eastwood2023-06-161-302/+0
| | | | | | | | | | | | into existing rooms (#15748) Context for why we're removing the implementation: - https://github.com/matrix-org/matrix-spec-proposals/pull/2716#issuecomment-1487441010 - https://github.com/matrix-org/matrix-spec-proposals/pull/2716#issuecomment-1504262734 Anyone wanting to continue MSC2716, should also address these leftover tasks: https://github.com/matrix-org/synapse/issues/10737 Closes https://github.com/matrix-org/synapse/issues/10737 in the fact that it is not longer necessary to track those things.
* push rules: fix internal conversion from _type to value (#15781)Mathieu Velten2023-06-161-0/+67
| | | Also fix wrong rule names for `is_user_mention` and `is_room_mention`.
* Clear event caches when we purge history (#15609)Erik Johnston2023-06-081-3/+0
| | | | | | | This should help a little with #13476 --------- Co-authored-by: Patrick Cloke <patrickc@matrix.org>
* Check required power levels earlier in createRoom handler. (#15695)Grant McLean2023-06-071-0/+37
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Check required power levels earlier in createRoom handler. - If a server was configured to reject the creation of rooms with E2EE enabled (by specifying an unattainably high power level for "m.room.encryption" in default_power_level_content_override), the 403 error was not being triggered until after the room was created and before the "m.room.power_levels" was sent. This allowed a user to access the partially-configured room and complete the setup of E2EE and power levels manually. - This change causes the power level overrides to be checked earlier and the request to be rejected before the user gains access to the room. - A new `_validate_room_config` method is added to contain checks that should be run before a room is created. - The new test case confirms that a user request is rejected by the new validation method. Signed-off-by: Grant McLean <grant@catalyst.net.nz> * Add a changelog file. * Formatting fix for black. * Remove unneeded line from test. --------- Signed-off-by: Grant McLean <grant@catalyst.net.nz>
* `N + 3`: Read from column `full_user_id` rather than `user_id` of tables ↵Shay2023-06-021-1/+3
| | | | `profiles` and `user_filters` (#15649)
* Add a catch-all * to the supported relation types when redacting (#15705)Mathieu Velten2023-06-021-3/+101
| | | This is an update to MSC3912 implementation
* Implement stable support for MSC3882 to allow an existing device/session to ↵Hugh Nimmo-Smith2023-06-013-18/+104
| | | | | | | | generate a login token for use on a new device/session (#15388) Implements stable support for MSC3882; this involves updating Synapse's support to match the MSC / the spec says. Continue to support the unstable version to allow clients to transition.
* Do not allow deactivated users to login with JWT. (#15624)Patrick Cloke2023-05-191-2/+18
| | | | | To improve the organization of this code it moves the JWT login checks to a separate handler and then fixes the bug (and a deprecation warning).
* Handle missing previous read marker event. (#15464)Nick Mills-Barrett2023-05-181-0/+147
| | | | | If the previous read marker is pointing to an event that no longer exists (e.g. due to retention) then assume that the newly given read marker is newer.
* Update Mutual Rooms (MSC2666) implementation (#15621)Jonathan de Jong2023-05-181-2/+4
| | | | | | | | To track changes in MSC2666: - The change from `/mutual_rooms/{user_id}` to `/mutual_rooms?user_id={user_id}`. - The addition of `next_batch_token` (and logic). - Unstable flag now being `uk.half-shot.msc2666.query_mutual_rooms`. - The error code when your own user is requested.
* Move ThirdPartyEventRules into module_api/callbacks (#15535)Andrew Morgan2023-05-041-21/+35
|
* Initial implementation of MSC3981: recursive relations API (#15315)Patrick Cloke2023-05-021-0/+120
| | | | | | | | | | | Adds an optional keyword argument to the /relations API which will recurse a limited number of event relationships. This will cause the API to return not just the events related to the parent event, but also events related to those related to the parent event, etc. This is disabled by default behind an experimental configuration flag and is currently implemented using prefixed parameters.
* Add column `full_user_id` to tables `profiles` and `user_filters`. (#15458)Shay2023-04-261-1/+3
|
* Move Spam Checker callbacks to a dedicated file (#15453)Andrew Morgan2023-04-181-7/+19
|
* Implement MSC2174: move redacts to a content property. (#15395)Patrick Cloke2023-04-131-2/+37
| | | | | | | This moves `redacts` from being a top-level property to a `content` property in a new room version. MSC2176 (which was previously implemented) states to not `redact` this property.
* Delete server-side backup keys when deactivating an account. (#15181)Shay2023-04-041-0/+157
|
* Revert pruning of old devices (#15360)Erik Johnston2023-03-311-47/+0
| | | | | | | | | * Revert "Fix registering a device on an account with lots of devices (#15348)" This reverts commit f0d8f66eaaacfa75bed65bc5d0c602fbc5339c85. * Revert "Delete stale non-e2e devices for users, take 3 (#15183)" This reverts commit 78cdb72cd6b0e007c314d9fed9f629dfc5b937a6.
* Fix registering a device on an account with lots of devices (#15348)Erik Johnston2023-03-291-0/+47
| | | Fixes up #15183
* Make `POST /_matrix/client/v3/rooms/{roomId}/report/{eventId}` endpoint ↵Andrew Morgan2023-03-211-0/+37
| | | | return 404 if event exists, but the user lacks access (#15300)
* Make `EventHandler.get_event` return `None` when the requested event is not ↵Andrew Morgan2023-03-211-0/+5
| | | | found (#15298)
* Move Account Validity callbacks to a dedicated file (#15237)Andrew Morgan2023-03-161-3/+2
|
* Merge branch 'release-v1.79' into developDavid Robertson2023-03-131-9/+10
|\
| * Fix missing conditional for registering ↵Andrew Morgan2023-03-101-9/+10
| | | | | | | | `on_remove_user_third_party_identifier` module api callbacks (#15227
* | Pass the Requester down to the HttpTransactionCache. (#15200)Quentin Gliech2023-03-071-13/+42
|/
* Stop applying edits to event contents (MSC3925). (#15193)Patrick Cloke2023-03-061-49/+10
| | | | | | | | | | | Enables MSC3925 support by default, which: * Includes the full edit event in the bundled aggregations of an edited event. * Stops modifying the original event's content to return the new content from the edit event. This is a backwards-incompatible change that is considered to be "correct" by the spec.
* Implementation of MSC3967: Don't require UIA for initial upload of cross ↵Hugh Nimmo-Smith2023-03-021-0/+141
| | | | signing keys (#15077)
* Remove support for aggregating reactions (#15172)Richard van der Hoff2023-02-281-150/+28
| | | | | | | | | | It turns out that no clients rely on server-side aggregation of `m.annotation` relationships: it's just not very useful as currently implemented. It's also non-trivial to calculate. I want to remove it from MSC2677, so to keep the implementation in line, let's remove it here.
* Add module API callbacks for adding and deleting local 3PID associations (#15044Andrew Morgan2023-02-271-0/+121
|
* Batch up storing state groups when creating new room (#14918)Shay2023-02-241-2/+2
|
* Bump black from 22.12.0 to 23.1.0 (#15103)dependabot[bot]2023-02-2217-41/+5
|
* Fix-up type hints in tests/server.py. (#15084)Patrick Cloke2023-02-172-31/+41
| | | | | This file was being ignored by mypy, we remove that and add the missing type hints & deal with any fallout.
* Use mypy 1.0 (#15052)David Robertson2023-02-162-1/+4
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Update mypy and mypy-zope * Remove unused ignores These used to suppress ``` synapse/storage/engines/__init__.py:28: error: "__new__" must return a class instance (got "NoReturn") [misc] ``` and ``` synapse/http/matrixfederationclient.py:1270: error: "BaseException" has no attribute "reasons" [attr-defined] ``` (note that we check `hasattr(e, "reasons")` above) * Avoid empty body warnings, sometimes by marking methods as abstract E.g. ``` tests/handlers/test_register.py:58: error: Missing return statement [empty-body] tests/handlers/test_register.py:108: error: Missing return statement [empty-body] ``` * Suppress false positive about `JaegerConfig` Complaint was ``` synapse/logging/opentracing.py:450: error: Function "Type[Config]" could always be true in boolean context [truthy-function] ``` * Fix not calling `is_state()` Oops! ``` tests/rest/client/test_third_party_rules.py:428: error: Function "Callable[[], bool]" could always be true in boolean context [truthy-function] ``` * Suppress false positives from ParamSpecs ```` synapse/logging/opentracing.py:971: error: Argument 2 to "_custom_sync_async_decorator" has incompatible type "Callable[[Arg(Callable[P, R], 'func'), **P], _GeneratorContextManager[None]]"; expected "Callable[[Callable[P, R], **P], _GeneratorContextManager[None]]" [arg-type] synapse/logging/opentracing.py:1017: error: Argument 2 to "_custom_sync_async_decorator" has incompatible type "Callable[[Arg(Callable[P, R], 'func'), **P], _GeneratorContextManager[None]]"; expected "Callable[[Callable[P, R], **P], _GeneratorContextManager[None]]" [arg-type] ```` * Drive-by improvement to `wrapping_logic` annotation * Workaround false "unreachable" positives See https://github.com/Shoobx/mypy-zope/issues/91 ``` tests/http/test_proxyagent.py:626: error: Statement is unreachable [unreachable] tests/http/test_proxyagent.py:762: error: Statement is unreachable [unreachable] tests/http/test_proxyagent.py:826: error: Statement is unreachable [unreachable] tests/http/test_proxyagent.py:838: error: Statement is unreachable [unreachable] tests/http/test_proxyagent.py:845: error: Statement is unreachable [unreachable] tests/http/federation/test_matrix_federation_agent.py:151: error: Statement is unreachable [unreachable] tests/http/federation/test_matrix_federation_agent.py:452: error: Statement is unreachable [unreachable] tests/logging/test_remote_handler.py:60: error: Statement is unreachable [unreachable] tests/logging/test_remote_handler.py:93: error: Statement is unreachable [unreachable] tests/logging/test_remote_handler.py:127: error: Statement is unreachable [unreachable] tests/logging/test_remote_handler.py:152: error: Statement is unreachable [unreachable] ``` * Changelog * Tweak DBAPI2 Protocol to be accepted by mypy 1.0 Some extra context in: - https://github.com/matrix-org/python-canonicaljson/pull/57 - https://github.com/python/mypy/issues/6002 - https://mypy.readthedocs.io/en/latest/common_issues.html#covariant-subtyping-of-mutable-protocol-members-is-rejected * Pull in updated canonicaljson lib so the protocol check just works * Improve comments in opentracing I tried to workaround the ignores but found it too much trouble. I think the corresponding issue is https://github.com/python/mypy/issues/12909. The mypy repo has a PR claiming to fix this (https://github.com/python/mypy/pull/14677) which might mean this gets resolved soon? * Better annotation for INTERACTIVE_AUTH_CHECKERS * Drive-by AUTH_TYPE annotation, to remove an ignore
* Add final type hint to tests.unittest. (#15072)Patrick Cloke2023-02-148-20/+29
| | | | Adds a return type to HomeServerTestCase.make_homeserver and deal with any variables which are no longer Any.
* Prevent clients from reporting nonexistent events. (#13779)reivilibre2023-02-141-0/+12
|
* Add missing type hints in tests (#14879)Patrick Cloke2023-01-261-14/+28
| | | | * FIx-up type hints in tests.logging. * Add missing type hints to test_transactions.
* Fix type hints in knocking tests. (#14887)Andrew Morgan2023-01-251-3/+1
|
* Faster joins: omit partial rooms from eager syncs until the resync completes ↵David Robertson2023-01-232-6/+8
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | (#14870) * Allow `AbstractSet` in `StrCollection` Or else frozensets are excluded. This will be useful in an upcoming commit where I plan to change a function that accepts `List[str]` to accept `StrCollection` instead. * `rooms_to_exclude` -> `rooms_to_exclude_globally` I am about to make use of this exclusion mechanism to exclude rooms for a specific user and a specific sync. This rename helps to clarify the distinction between the global config and the rooms to exclude for a specific sync. * Better function names for internal sync methods * Track a list of excluded rooms on SyncResultBuilder I plan to feed a list of partially stated rooms for this sync to ignore * Exclude partial state rooms during eager sync using the mechanism established in the previous commit * Track un-partial-state stream in sync tokens So that we can work out which rooms have become fully-stated during a given sync period. * Fix mutation of `@cached` return value This was fouling up a complement test added alongside this PR. Excluding a room would mean the set of forgotten rooms in the cache would be extended. This means that room could be erroneously considered forgotten in the future. Introduced in #12310, Synapse 1.57.0. I don't think this had any user-visible side effects (until now). * SyncResultBuilder: track rooms to force as newly joined Similar plan as before. We've omitted rooms from certain sync responses; now we establish the mechanism to reintroduce them into future syncs. * Read new field, to present rooms as newly joined * Force un-partial-stated rooms to be newly-joined for eager incremental syncs only, provided they're still fully stated * Notify user stream listeners to wake up long polling syncs * Changelog * Typo fix Co-authored-by: Sean Quah <8349537+squahtx@users.noreply.github.com> * Unnecessary list cast Co-authored-by: Sean Quah <8349537+squahtx@users.noreply.github.com> * Rephrase comment Co-authored-by: Sean Quah <8349537+squahtx@users.noreply.github.com> * Another comment Co-authored-by: Sean Quah <8349537+squahtx@users.noreply.github.com> * Fixup merge(?) * Poke notifier when receiving un-partial-stated msg over replication * Fixup merge whoops Thanks MV :) Co-authored-by: Mathieu Velen <mathieuv@matrix.org> Co-authored-by: Mathieu Velten <mathieuv@matrix.org> Co-authored-by: Sean Quah <8349537+squahtx@users.noreply.github.com>
* Change default room version to 10. Implements MSC3904 (#14111)Catalan Lover2023-01-181-3/+9
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Change Documentation to have v10 as default room version * Change Default Room version to 10 * Add changelog entry for default room version swap * Add changelog entry for v10 default room version in docs * Clarify doc changelog entry Co-authored-by: David Robertson <david.m.robertson1@gmail.com> * Improve Documentation changes. Co-authored-by: David Robertson <david.m.robertson1@gmail.com> * Update Changelog entry to have correct format Co-authored-by: David Robertson <david.m.robertson1@gmail.com> * Update Spec Version to 1.5 * Only need 1 changelog. * Fix test. * Update "Changed in" line Co-authored-by: David Robertson <david.m.robertson1@gmail.com> Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com> Co-authored-by: Patrick Cloke <patrickc@matrix.org>
* Implement MSC3925: changes to bundling of edits (#14811)Richard van der Hoff2023-01-101-55/+130
| | | | | | | | Two parts to this: * Bundle the whole of the replacement with any edited events. This is backwards-compatible so I haven't put it behind a flag. * Optionally, inhibit server-side replacement of edited events. This has scope to break things, so it is currently disabled by default.
* Disable sending confirmation email when 3pid is disabled #14682 (#14725)Jeyachandran Rathnam2023-01-091-25/+5
| | | | | | | | | | | * Fixes #12277 :Disable sending confirmation email when 3pid is disabled * Fix test_add_email_if_disabled test case to reflect changes to enable_3pid_changes flag * Add changelog file * Rename newsfragment. Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
* Respond with proper error responses on unknown paths. (#14621)Patrick Cloke2022-12-082-3/+3
| | | | Returns a proper 404 with an errcode of M_RECOGNIZED for unknown endpoints per MSC3743.
* Reject receipt requests with invalid room or event IDs. (#14632)Nick Mills-Barrett2022-12-071-0/+76
| | | | If the room or event IDs are empty or of an invalid form they should be rejected.
* Move MSC3030 `/timestamp_to_event` endpoint to stable v1 location (#14471)Eric Eastwood2022-11-281-6/+1
| | | | | | | | Fix https://github.com/matrix-org/synapse/issues/14390 - Client API: `/_matrix/client/unstable/org.matrix.msc3030/rooms/<roomID>/timestamp_to_event?ts=<timestamp>&dir=<direction>` -> `/_matrix/client/v1/rooms/<roomID>/timestamp_to_event?ts=<timestamp>&dir=<direction>` - Federation API: `/_matrix/federation/unstable/org.matrix.msc3030/timestamp_to_event/<roomID>?ts=<timestamp>&dir=<direction>` -> `/_matrix/federation/v1/timestamp_to_event/<roomID>?ts=<timestamp>&dir=<direction>` Complement test changes: https://github.com/matrix-org/complement/pull/559
* Batch fetch bundled references (#14508)Patrick Cloke2022-11-221-2/+2
| | | | | | | | | Avoid an n+1 query problem and fetch the bundled aggregations for m.reference relations in a single query instead of a query per event. This applies similar logic for as was previously done for edits in 8b309adb436c162510ed1402f33b8741d71fc058 (#11660; threads in b65acead428653b988351ae8d7b22127a22039cd (#11752); and annotations in 1799a54a545618782840a60950ef4b64da9ee24d (#14491).
* Batch fetch bundled annotations (#14491)Patrick Cloke2022-11-221-2/+2
| | | | | | | | Avoid an n+1 query problem and fetch the bundled aggregations for m.annotation relations in a single query instead of a query per event. This applies similar logic for as was previously done for edits in 8b309adb436c162510ed1402f33b8741d71fc058 (#11660) and threads in b65acead428653b988351ae8d7b22127a22039cd (#11752).
* Correctly create power level event during initial room creation (#14361)Shay2022-11-071-2/+2
|
* Fix /refresh endpoint version (#14364)Tulir Asokan2022-11-041-8/+8
|
* Implement MSC3912: Relation-based redactions (#14260)Brendan Abolivier2022-11-032-4/+306
| | | Co-authored-by: Sean Quah <8349537+squahtx@users.noreply.github.com>
* Fix dehydrated device REST checks (#14336)David Robertson2022-10-311-0/+34
|
* Support OIDC backchannel logouts (#11414)Quentin Gliech2022-10-312-28/+417
| | | | | | | If configured an OIDC IdP can log a user's session out of Synapse when they log out of the identity provider. The IdP sends a request directly to Synapse (and must be configured with an endpoint) when a user logs out.
* Merge branch 'master' into developOlivier Wilkinson (reivilibre)2022-10-281-3/+51
|\
| * Fix room creation being rate limited too aggressively since Synapse v1.69.0. ↵reivilibre2022-10-281-3/+51
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | (#14314) * Introduce a test for the old behaviour which we want to restore * Reintroduce the old behaviour in a simpler way * Newsfile Signed-off-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org> * Use 1 credit instead of 2 for creating a room: be more lenient than before Notably, the UI in Element Web was still broken after restoring to prior behaviour. After discussion, we agreed that it would be sensible to increase the limit. Signed-off-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org>
* | Merge branch 'release-v1.70' into developOlivier Wilkinson (reivilibre)2022-10-251-33/+77
|\|
| * Properly update the threads table when thread events are redacted. (#14248)Patrick Cloke2022-10-211-33/+77
| | | | | | | | | | | | | | | | | | When the last event in a thread is redacted we need to update the threads table: * Find the new latest event in the thread and store it into the table; or * Remove the thread from the table if it is no longer a thread (i.e. all events in the thread were redacted).
* | Refactor OIDC tests to better mimic an actual OIDC provider. (#13910)Quentin Gliech2022-10-253-88/+120
|/ | | | | | | | | This implements a fake OIDC server, which intercepts calls to the HTTP client. Improves accuracy of tests by covering more internal methods. One particular example was the ID token validation, which previously mocked. This uncovered an incorrect dependency: Synapse actually requires at least authlib 0.15.1, not 0.14.0.
* Fix MSC3030 `/timestamp_to_event` returning `outliers` that it has no idea ↵Eric Eastwood2022-10-181-0/+65
| | | | | | | | | | | | | | | | | | | | | | | | | | | whether are near a gap or not (#14215) Fix MSC3030 `/timestamp_to_event` endpoint returning `outliers` that it has no idea whether are near a gap or not (and therefore unable to determine whether it's actually the closest event). The reason Synapse doesn't know whether an `outlier` is next to a gap is because our gap checks rely on entries in the `event_edges`, `event_forward_extremeties`, and `event_backward_extremities` tables which is [not the case for `outliers`](https://github.com/matrix-org/synapse/blob/2c63cdcc3f1aa4625e947de3c23e0a8133c61286/docs/development/room-dag-concepts.md#outliers). Also fixes MSC3030 Complement `can_paginate_after_getting_remote_event_from_timestamp_to_event_endpoint` test flake. Although this acted flakey in Complement, if `sync_partial_state` raced and beat us before `/timestamp_to_event`, then even if we retried the failing `/context` request it wouldn't work until we made this Synapse change. With this PR, Synapse will never return an `outlier` event so that test will always go and ask over federation. Fix https://github.com/matrix-org/synapse/issues/13944 ### Why did this fail before? Why was it flakey? Sleuthing the server logs on the [CI failure](https://github.com/matrix-org/synapse/actions/runs/3149623842/jobs/5121449357#step:5:5805), it looks like `hs2:/timestamp_to_event` found `$NP6-oU7mIFVyhtKfGvfrEQX949hQX-T-gvuauG6eurU` as an `outlier` event locally. Then when we went and asked for it via `/context`, since it's an `outlier`, it was filtered out of the results -> `You don't have permission to access that event.` This is reproducible when `sync_partial_state` races and persists `$NP6-oU7mIFVyhtKfGvfrEQX949hQX-T-gvuauG6eurU` as an `outlier` before we evaluate `get_event_for_timestamp(...)`. To consistently reproduce locally, just add a delay at the [start of `get_event_for_timestamp(...)`](https://github.com/matrix-org/synapse/blob/cb20b885cb4bd1648581dd043a184d86fc8c7a00/synapse/handlers/room.py#L1470-L1496) so it always runs after `sync_partial_state` completes. ```py from twisted.internet import task as twisted_task d = twisted_task.deferLater(self.hs.get_reactor(), 3.5) await d ``` In a run where it passes, on `hs2`, `get_event_for_timestamp(...)` finds a different event locally which is next to a gap and we request from a closer one from `hs1` which gets backfilled. And since the backfilled event is not an `outlier`, it's returned as expected during `/context`. With this PR, Synapse will never return an `outlier` event so that test will always go and ask over federation.
* Implementation of HTTP 307 response for MSC3886 POST endpoint (#14018)Hugh Nimmo-Smith2022-10-181-0/+45
| | | | Co-authored-by: reivilibre <olivier@librepush.net> Co-authored-by: Andrew Morgan <andrewm@element.io>
* Support filtering the /messages API by relation type (MSC3874). (#14148)Patrick Cloke2022-10-172-138/+8
| | | Gated behind an experimental configuration flag.
* Do not allow a None-limit on PaginationConfig. (#14146)Patrick Cloke2022-10-141-1/+2
| | | | | | | The callers either set a default limit or manually handle a None-limit later on (by setting a default value). Update the callers to always instantiate PaginationConfig with a default limit and then assume the limit is non-None.
* Stabilize the threads API. (#14175)Patrick Cloke2022-10-141-18/+29
| | | | | | | Stabilize the threads API (MSC3856) by supporting (only) the v1 path for the endpoint. This also marks the API as safe for workers since it is a read-only API.
* Add an API for listing threads in a room. (#13394)Patrick Cloke2022-10-131-0/+151
| | | | | | | | | Implement the /threads endpoint from MSC3856. This is currently unstable and behind an experimental configuration flag. It includes a background update to backfill data, results from the /threads endpoint will be partial until that finishes.
* Fix handling of public rooms filter with a network tuple. (#14053)Patrick Cloke2022-10-051-10/+31
| | | | | | | | Fixes two related bugs: * The handling of `[null]` for a `room_types` filter was incorrect. * The ordering of arguments when providing both a network tuple and room type field was incorrect.
* Do not return unspecced original_event field when using the stable ↵Patrick Cloke2022-10-031-5/+8
| | | | | | | | | | /relations endpoint. (#14025) Keep the old behavior (of including the original_event field) for any requests to the /unstable version of the endpoint, but do not include the field when the /v1 version is used. This should avoid new clients from depending on this field, but will not help with current dependencies.
* Add query parameter `ts` to allow appservices set the `origin_server_ts` for ↵lukasdenk2022-10-031-2/+117
| | | | | | | state events. (#11866) MSC3316 declares that both /rooms/{roomId}/send and /rooms/{roomId}/state should accept a ts parameter for appservices. This change expands support to /state and adds tests.
* Skip filtering during push if there are no push actions (#13992)Erik Johnston2022-09-301-2/+2
|
* Allow admins to require a manual approval process before new accounts can be ↵Brendan Abolivier2022-09-294-5/+113
| | | | used (using MSC3866) (#13556)
* Expose MSC3882 only be under an unstable endpoint. (#13868)Hugh Nimmo-Smith2022-09-291-7/+9
|
* Prepatory work for batching events to send (#13487)Shay2022-09-281-2/+2
| | | This PR begins work on batching up events during the creation of a room. The PR splits out the creation and sending/persisting of the events. The first three events in the creation of the room-creating the room, joining the creator to the room, and the power levels event are sent sequentially, while the subsequent events are created and collected to be sent at the end of the function. This is currently done by appending them to a list and then iterating over the list to send, the next step (after this PR) would be to send and persist the collected events as a batch.
* Support the stable dir parameter for /relations. (#13920)Patrick Cloke2022-09-271-4/+2
| | | | | | | Since MSC3715 has passed FCP, the stable parameter can be used. This currently falls back to the unstable parameter if the stable parameter is not provided (and MSC3715 support is enabled in the configuration).
* Properly paginate forward in the /relations API. (#13840)Patrick Cloke2022-09-221-1/+28
| | | | | This fixes a bug where the `/relations` API with `dir=f` would skip the first item of each page (except the first page), causing incomplete data to be returned to the client.
* Implementation of MSC3882 login token request (#13722)Hugh Nimmo-Smith2022-09-211-0/+132
|
* A third batch of Pydantic validation for rest/client/account.py (#13736)David Robertson2022-09-151-3/+26
|
* Cancel the processing of key query requests when they time out. (#13680)reivilibre2022-09-071-0/+29
|
* Remove support for unstable private read receipts (#13653)Šimon Brandner2022-09-011-26/+11
| | | Signed-off-by: Šimon Brandner <simon.bra.ag@gmail.com>
* Drop support for calling `/_matrix/client/v3/rooms/{roomId}/invite` without ↵Jacek Kuśnierz2022-08-313-3/+25
| | | | | | | an `id_access_token` (#13241) Fixes #13206 Signed-off-by: Jacek Kusnierz jacek.kusnierz@tum.de
* Directly lookup local membership instead of getting all members in a room ↵Eric Eastwood2022-08-241-6/+6
| | | | | first (`get_users_in_room` mis-use) (#13608) See https://github.com/matrix-org/synapse/pull/13575#discussion_r953023755
* Drop support for delegating email validation, round 2 (#13596)David Robertson2022-08-231-1/+1
|
* `synapse.api.auth.Auth` cleanup: make permission-related methods use ↵Quentin Gliech2022-08-222-5/+5
| | | | | | | | | `Requester` instead of the `UserID` (#13024) Part of #13019 This changes all the permission-related methods to rely on the Requester instead of the UserID. This is a first step towards enabling scoped access tokens at some point, since I expect the Requester to have scope-related informations in it. It also changes methods which figure out the user/device/appservice out of the access token to return a Requester instead of something else. This avoids having store-related objects in the methods signatures.
generated by cgit-magenta 1.5.1 (git 2.48.1) at 2025-08-13 02:32:45 +0000