summary refs log tree commit diff
path: root/tests/rest (follow)
Commit message (Collapse)AuthorAgeFilesLines
* Fix sync waiting for an invalid token from the "future" (#17386)Erik Johnston2024-07-021-1/+3
| | | | | | | | | | | | | Fixes https://github.com/element-hq/synapse/issues/17274, hopefully. Basically, old versions of Synapse could advance streams without persisting anything in the DB (fixed in #17229). On restart those updates would get lost, and so the position of the stream would revert to an older position. If this happened across an upgrade to a later Synapse version which included #17215, then sync could get blocked indefinitely (until the stream advanced to the position in the token). We fix this by bounding the stream positions we'll wait for to the maximum position of the underlying stream ID generator.
* Add `is_invite` filtering to Sliding Sync `/sync` (#17335)Eric Eastwood2024-06-241-25/+123
| | | Based on [MSC3575](https://github.com/matrix-org/matrix-spec-proposals/pull/3575): Sliding Sync
* Add support for MSC3823 - Account Suspension Part 2 (#17255)Shay2024-06-242-0/+189
|
* Register sliding sync under a different path (#17331)Erik Johnston2024-06-191-1/+3
| | | As the API is slightly incompatible.
* Filter added to Admin-API GET /rooms (#17276)Alexander Fechler2024-06-191-0/+77
|
* Require the 'from' parameter for `/notifications` be an integer (#17283)Andrew Morgan2024-06-191-18/+153
| | | | Co-authored-by: Erik Johnston <erikj@element.io>
* Add `stream_ordering` sort to Sliding Sync `/sync` (#17293)Eric Eastwood2024-06-171-4/+57
| | | Sort is no longer configurable and we always sort rooms by the `stream_ordering` of the last event in the room or the point where the user can see up to in cases of leave/ban/invite/knock.
* Enable cross-signing key upload without UIA (#17284)Richard van der Hoff2024-06-142-69/+0
| | | | | | Per MSC3967, which is now stable, we should not require UIA when uploading cross-signing keys for the first time. Fixes: #17227
* Include user membership on events (#17282)Richard van der Hoff2024-06-131-1/+0
| | | | | MSC4115 has now completed FCP, so we can enable it by default and switch to the stable identifier.
* Add `is_dm` filtering to Sliding Sync `/sync` (#17277)Eric Eastwood2024-06-131-0/+127
| | | Based on [MSC3575](https://github.com/matrix-org/matrix-spec-proposals/pull/3575): Sliding Sync
* Bump `mypy` from 1.8.0 to 1.9.0 (#17297)Andrew Morgan2024-06-131-4/+24
| | | | Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* Add report room API (MSC4151) (#17270)Travis Ralston2024-06-122-5/+94
| | | | | | | | https://github.com/matrix-org/matrix-spec-proposals/pull/4151 This is intended to be enabled by default for immediate use. When FCP is complete, the unstable endpoint will be dropped and stable endpoint supported instead - no backwards compatibility is expected for the unstable endpoint.
* Reorganize Pydantic models and types used in handlers (#17279)Eric Eastwood2024-06-101-1/+1
| | | | | | Spawning from https://github.com/element-hq/synapse/pull/17187#discussion_r1619492779 around wanting to put `SlidingSyncBody` (parse the request in the rest layer), `SlidingSyncConfig` (from the rest layer, pass to the handler), `SlidingSyncResponse` (pass the response from the handler back to the rest layer to respond) somewhere that doesn't contaminate the imports and cause circular import issues. - Moved Pydantic parsing models to `synapse/types/rest` - Moved handler types to `synapse/types/handlers`
* Add Sliding Sync `/sync` endpoint (initial implementation) (#17187)Eric Eastwood2024-06-062-2/+137
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Based on [MSC3575](https://github.com/matrix-org/matrix-spec-proposals/pull/3575): Sliding Sync This iteration only focuses on returning the list of room IDs in the sliding window API (without sorting/filtering). Rooms appear in the Sliding sync response based on: - `invite`, `join`, `knock`, `ban` membership events - Kicks (`leave` membership events where `sender` is different from the `user_id`/`state_key`) - `newly_left` (rooms that were left during the given token range, > `from_token` and <= `to_token`) - In order for bans/kicks to not show up, you need to `/forget` those rooms. This doesn't modify the event itself though and only adds the `forgotten` flag to `room_memberships` in Synapse. There isn't a way to tell when a room was forgotten at the moment so we can't factor it into the from/to range. ### Example request `POST http://localhost:8008/_matrix/client/unstable/org.matrix.msc3575/sync` ```json { "lists": { "foo-list": { "ranges": [ [0, 99] ], "sort": [ "by_notification_level", "by_recency", "by_name" ], "required_state": [ ["m.room.join_rules", ""], ["m.room.history_visibility", ""], ["m.space.child", "*"] ], "timeline_limit": 100 } } } ``` Response: ```json { "next_pos": "s58_224_0_13_10_1_1_16_0_1", "lists": { "foo-list": { "count": 1, "ops": [ { "op": "SYNC", "range": [0, 99], "room_ids": [ "!MmgikIyFzsuvtnbvVG:my.synapse.linux.server" ] } ] } }, "rooms": {}, "extensions": {} } ```
* Fix slipped logging context when media rejected (#17239)Erik Johnston2024-05-292-14/+14
| | | | | | | When a module rejects a piece of media we end up trying to close the same logging context twice. Instead of fixing the existing code we refactor to use an async context manager, which is easier to write correctly.
* Support MSC3916 by adding unstable media endpoints to `_matrix/client` (#17213)Shay2024-05-241-0/+1609
| | | | | | | | | | [MSC3916](https://github.com/matrix-org/matrix-spec-proposals/blob/rav/authentication-for-media/proposals/3916-authentication-for-media.md) adds new media endpoints under `_matrix/client`. This PR adds the `/preview_url`, `/config`, and `/thumbnail` endpoints. `/download` will be added in a follow-up PR once the work for the federation `/download` endpoint is complete (see https://github.com/element-hq/synapse/pull/17172). Should be reviewable commit-by-commit.
* Add Sliding Sync `/sync/e2ee` endpoint for To-Device messages (#17167)Eric Eastwood2024-05-233-165/+449
| | | | | | | | | | | This is being introduced as part of Sliding Sync but doesn't have any sliding window component. It's just a way to get E2EE events without having to sit through a big initial sync (`/sync` v2). And we can avoid encryption events being backed up by the main sync response or vice-versa. Part of some Sliding Sync simplification/experimentation. See [this discussion](https://github.com/element-hq/synapse/pull/17167#discussion_r1610495866) for why it may not be as useful as we thought. Based on: - https://github.com/matrix-org/matrix-spec-proposals/pull/3575 - https://github.com/matrix-org/matrix-spec-proposals/pull/3885 - https://github.com/matrix-org/matrix-spec-proposals/pull/3884
* Bring auto-accept invite logic into Synapse (#17147)devonh2024-05-211-0/+2
| | | | | | | | | | | | | | This PR ports the logic from the [synapse_auto_accept_invite](https://github.com/matrix-org/synapse-auto-accept-invite) module into synapse. I went with the naive approach of injecting the "module" next to where third party modules are currently loaded. If there is a better/preferred way to handle this, I'm all ears. It wasn't obvious to me if there was a better location to add this logic that would cleanly apply to all incoming invite events. Relies on https://github.com/element-hq/synapse/pull/17166 to fix linter errors.
* An federation whitelist query endpoint extension (#16848)Erik Johnston2024-05-133-0/+143
| | | | | | | | | | This is to allow clients to query the configured federation whitelist. Disabled by default. --------- Co-authored-by: Devon Hudson <devonhudson@librem.one> Co-authored-by: devonh <devon.dmytro@gmail.com> Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
* Add support for MSC3823 - Account Suspension (#17051)Shay2024-05-011-3/+66
|
* Apply user `email` & `picture` during OIDC registration if present & ↵devonh2024-04-291-14/+190
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | selected (#17120) This change will apply the `email` & `picture` provided by OIDC to the new user account when registering a new user via OIDC. If the user is directed to the account details form, this change makes sure they have been selected before applying them, otherwise they are omitted. In particular, this change ensures the values are carried through when Synapse has consent configured, and the redirect to the consent form/s are followed. I have tested everything manually. Including: - with/without consent configured - allowing/not allowing the use of email/avatar (via `sso_auth_account_details.html`) - with/without automatic account detail population (by un/commenting the `localpart_template` option in synapse config). ### Pull Request Checklist <!-- Please read https://element-hq.github.io/synapse/latest/development/contributing_guide.html before submitting your pull request --> * [X] Pull request is based on the develop branch * [X] Pull request includes a [changelog file](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#changelog). The entry should: - Be a short description of your change which makes sense to users. "Fixed a bug that prevented receiving messages from other servers." instead of "Moved X method from `EventStore` to `EventWorkerStore`.". - Use markdown where necessary, mostly for `code blocks`. - End with either a period (.) or an exclamation mark (!). - Start with a capital letter. - Feel free to credit yourself, by adding a sentence "Contributed by @github_username." or "Contributed by [Your Name]." to the end of the entry. * [X] [Code style](https://element-hq.github.io/synapse/latest/code_style.html) is correct (run the [linters](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#run-the-linters))
* Add support for MSC4115 (#17104)Richard van der Hoff2024-04-291-1/+6
| | | | Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
* Fix filtering of rooms when supplying the `destination` query parameter to ↵Andrew Morgan2024-04-261-3/+64
| | | | `/_synapse/admin/v1/federation/destinations/<destination>/rooms` (#17077)
* MSC4108 implementation (#17056)Quentin Gliech2024-04-251-1/+400
| | | | | | Co-authored-by: Hugh Nimmo-Smith <hughns@element.io> Co-authored-by: Hugh Nimmo-Smith <hughns@users.noreply.github.com> Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
* Send an email if the address is already bound to an user account (#16819)mcalinghee2024-04-231-0/+9
| | | | Co-authored-by: Mathieu Velten <mathieu.velten@beta.gouv.fr> Co-authored-by: Olivier D <odelcroi@gmail.com>
* Parse json validation (#16923)Gordan Trevis2024-04-182-0/+113
| | | | Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
* Support for MSC4108 via delegation (#17086)Quentin Gliech2024-04-171-4/+30
| | | | | | | This adds support for MSC4108 via delegation, similar to what has been done for MSC3886 --------- Co-authored-by: Hugh Nimmo-Smith <hughns@element.io>
* Parse Integer negative value validation (#16920)Gordan Trevis2024-04-161-2/+3
|
* Stabliize support for MSC3981: recurse /relations (#17023)Patrick Cloke2024-04-091-6/+3
| | | | | | | See [MSC3981](https://github.com/matrix-org/matrix-spec-proposals/pull/3981), this pretty much just removes flags though. Part of #17021
* Fix bug in `/sync` response for archived rooms (#16932)Richard van der Hoff2024-04-041-7/+11
| | | | | | | | | | | | This PR fixes a very, very niche edge-case, but I've got some more work coming which will otherwise make the problem worse. The bug happens when the syncing user leaves a room, and has a sync filter which includes "left" rooms, but sets the timeline limit to 0. In that case, the state returned in the `state` section is calculated incorrectly. The fix is to pass a token corresponding to the point that the user leaves the room through to `compute_state_delta`.
* Do not refuse to set read_marker if previous event_id is in wrong room (#16990)SpiritCroc2024-03-211-4/+4
|
* Update power level default for public rooms (#16907)Shay2024-03-191-0/+18
|
* Bump mypy from 1.5.1 to 1.8.0 (#16901)dependabot[bot]2024-03-131-1/+1
|
* Bump black from 23.10.1 to 24.2.0 (#16936)dependabot[bot]2024-03-132-10/+8
|
* deactivated flag refactored to filter deactivated users. (#16874)Alexander Fechler2024-03-111-3/+53
| | | | Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
* Stabilize support for Retry-After header (MSC4014) (#16947)Patrick Cloke2024-03-081-3/+0
|
* Correctly mention previous copyright (#16820)Erik Johnston2024-01-2351-0/+58
| | | | | During the migration the automated script to update the copyright headers accidentally got rid of some of the existing copyright lines. Reinstate them.
* Handle wildcard type filters properly (#14984)Mo Balaa2024-01-222-1/+29
|
* Search non ASCII display names using Admin API (#16767)Adam Jędrzejewski2024-01-041-0/+23
| | | | | Closes #16370 Signed-off-by: Adam Jedrzejewski <adamjedrzejewski@icloud.com>
* Fix email verification redirection (#16761)FadhlanR2024-01-021-3/+40
| | | | | | Previously, the response status of `HTMLResource` was hardcoded as `200`. However, for proper redirection after the user verifies their email, we require the status to be `302`. This PR addresses that issue by using `code` as response status.
* Enable user without password (#16770)Dirk Klimpel2024-01-021-8/+23
| | | | | | | | | | | Closes: - https://github.com/matrix-org/synapse/issues/10397 - #10397 An administrator should know whether he wants to set a password or not. There are many uses cases where a blank password is required. - Use of only some users with SSO. - Use of bots with password, users with SSO
* Merge remote-tracking branch 'gitlab/clokep/license-license' into new_developErik Johnston2023-12-1361-625/+976
|\
| * Update license headersPatrick Cloke2023-11-2162-635/+992
| |
* | Add avatar and topic settings for server notice room (#16679)Mathieu Velten2023-12-121-0/+109
| |
* | Expose OIDC discovery information under the CSAPI (#16726)David Robertson2023-12-061-0/+59
| | | | | | | | Co-authored-by: Quentin Gliech <quenting@element.io>
* | Fix upgrading a room without `events` field in power levels (#16725)David Robertson2023-12-051-0/+28
| |
* | Server notices: add an autojoin setting for the notices room (#16699)Mathieu Velten2023-12-041-0/+27
| | | | | | | | Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
* | Implement MSC4069: Inhibit profile propagation (#16636)Travis Ralston2023-12-041-0/+160
| | | | | | MSC: https://github.com/matrix-org/matrix-spec-proposals/pull/4069
* | Move media retention tests out of rest tests (#16684)David Robertson2023-11-271-294/+0
| | | | | | | | | | | | | | | | * Move media retention tests out of rest tests AFAICS this doesn't make any HTTP requests and so it ought not to belong in `tests.rest`. * Changelog
* | Keep track of `user_ips` and `monthly_active_users` when delegating auth ↵David Robertson2023-11-233-21/+3
|/ | | | | | | | | | | | | (#16672) * Describe `insert_client_ip` * Pull out client_ips and MAU tracking to BaseAuth * Define HAS_AUTHLIB once in tests sick of copypasting * Track ips and token usage when delegating auth * Test that we track MAU and user_ips * Don't track `__oidc_admin`
* Add an Admin API to temporarily grant the ability to update an existing ↵David Robertson2023-11-152-2/+242
| | | | cross-signing key without UIA (#16634)
* Use full GitHub links instead of bare issue numbers. (#16637)Patrick Cloke2023-11-155-6/+9
|
* Convert simple_select_one_txn and simple_select_one to return tuples. (#16612)Patrick Cloke2023-11-093-11/+11
|
* Return attrs for more media repo APIs. (#16611)Patrick Cloke2023-11-092-18/+18
|
* Ensure local invited & knocking users leave before purge. (#16559)Patrick Cloke2023-10-271-1/+52
| | | | | This is mostly useful for federated rooms where some users would get stuck in the invite or knock state when the room was purged from their homeserver.
* Add new module API for adding custom fields to events `unsigned` section ↵Erik Johnston2023-10-271-1/+1
| | | | (#16549)
* Add a new module API to update user presence state. (#16544)Patrick Cloke2023-10-261-2/+17
| | | | | | | | | | This adds a module API which allows a module to update a user's presence state/status message. This is useful for controlling presence from an external system. To fully control presence from the module the presence.enabled config parameter gains a new state of "untracked" which disables internal tracking of presence changes via user actions, etc. Only updates from the module will be persisted and sent down sync properly).
* Apply join rate limiter outside the lineariser (#16441)David Robertson2023-10-061-0/+24
|
* Register media servlets via regex. (#16419)Patrick Cloke2023-10-066-161/+134
| | | | | This converts the media servlet URLs in the same way as (most) of the rest of Synapse. This will give more flexibility in the versions each endpoint exists under.
* Remove Python version from `/_synapse/admin/v1/server_version` (#16380)MomentQYC2023-10-021-3/+1
| | | | There's no reason to expose the full Python version over what is frequently a public API.
* Add support for pydantic v2 via pydantic.v1 compat module (#16332)Maxwell G2023-09-251-1/+7
| | | While maintaining support with pydantic v1.
* Filter locked users in the admin API (#16328)Hanadi2023-09-181-0/+26
| | | | Co-authored-by: Hanadi Tamimi <hanadi.tamimi@sdui.de>
* Additional validation of receipts (#16327)Patrick Cloke2023-09-182-161/+214
| | | | Reject invalid receipts with a reasonable error message & expands tests for receipts.
* Add automatic purge after all users forget a room (#15488)Mathieu Velten2023-09-153-23/+162
| | | | | | Also add restore of purge/shutdown rooms after a synapse restart. Co-authored-by: Eric Eastwood <erice@matrix.org> Co-authored-by: Erik Johnston <erikj@matrix.org>
* Update ruff config (#16283)Patrick Cloke2023-09-084-11/+11
| | | Enable additional checks & clean-up unneeded configuration.
* Avoid temporary storage of sensitive information. (#16272)Patrick Cloke2023-09-081-0/+13
| | | | During the UI auth process, avoid storing sensitive information into the database.
* Add last_seen_ts to the admin users API (#16218)Mathieu Velten2023-09-041-0/+60
|
* Bump mypy-zope & mypy. (#16188)Patrick Cloke2023-08-298-15/+15
|
* Prepare unit tests for Python 3.12 (#16099)V024602023-08-253-25/+28
|
* Replace simple_async_mock with AsyncMock (#16180)Patrick Cloke2023-08-251-3/+2
| | | | Python 3.8 has a native AsyncMock, use it instead of a custom implementation.
* Replace make_awaitable with AsyncMock (#16179)Patrick Cloke2023-08-247-61/+55
| | | | Python 3.8 provides a native AsyncMock, we can replace the homegrown version we have.
* Add `Retry-After` to M_LIMIT_EXCEEDED error responses (#16136)Will Hunt2023-08-241-6/+18
| | | Implements MSC4041 behind an experimental configuration flag.
* Allow filtering for admins in the list accounts admin API (#16114)Alexander Fechler2023-08-181-0/+38
|
* Run pyupgrade for python 3.7 & 3.8. (#16110)Patrick Cloke2023-08-153-27/+27
|
* Implements admin API to lock an user (MSC3939) (#15870)Mathieu Velten2023-08-101-1/+110
|
* Support MSC3814: Dehydrated Devices Part 2 (#16010)Shay2023-08-081-3/+74
|
* Fix deletion for Dehydrated Devices (#16046)Shay2023-08-041-1/+138
|
* Add forward-compatibility for the redacts property (MSC2174). (#16013)Patrick Cloke2023-08-021-17/+50
| | | | | | | | The location of the redacts field changes in room version 11. Ensure it is copied to the *new* location for *old* room versions for forwards-compatibility with clients. Note that copying it to the *old* location for the *new* room version was previously handled.
* Trim whitespace when setting display names (#16031)Mohit Rathee2023-08-011-0/+12
|
* Add ability to wait for locks and add locks to purge history / room deletion ↵Erik Johnston2023-07-311-2/+2
| | | | | (#15791) c.f. #13476
* Support MSC3814: Dehydrated Devices (#15929)Shay2023-07-241-2/+148
| | | | | | Signed-off-by: Nicolas Werner <n.werner@famedly.com> Co-authored-by: Nicolas Werner <n.werner@famedly.com> Co-authored-by: Nicolas Werner <89468146+nico-famedly@users.noreply.github.com> Co-authored-by: Hubert Chathi <hubert@uhoreg.ca>
* Actually stop reading from column `user_id` of tables `profiles` (#15955)Shay2023-07-231-1/+1
|
* Support room version 11 (#15912)Patrick Cloke2023-07-181-2/+19
| | | | | And fix a bug in the implementation of the updated redaction format (MSC2174) where the top-level redacts field was not properly added for backwards-compatibility.
* Re-introduce the outbound federation proxy (#15913)Eric Eastwood2023-07-182-3/+0
| | | | | Allow configuring the set of workers to proxy outbound federation traffic through (`outbound_federation_restricted_to`). This is useful when you have a worker setup with `federation_sender` instances responsible for sending outbound federation requests and want to make sure *all* outbound federation traffic goes through those instances. Before this change, the generic workers would still contact federation themselves for things like profile lookups, backfill, etc. This PR allows you to set more strict access controls/firewall for all workers and only allow the `federation_sender`'s to contact the outside world.
* Mark get_user_in_directory private since only used in tests (#15884)Mathieu Velten2023-07-121-3/+3
|
* Revert "Federation outbound proxy" (#15910)Eric Eastwood2023-07-102-0/+3
| | | | | | Revert "Federation outbound proxy (#15773)" This reverts commit b07b14b494ae1dd564b4c44f844c9a9545b3d08a.
* Federation outbound proxy (#15773)Eric Eastwood2023-07-052-3/+0
| | | | | | | Allow configuring the set of workers to proxy outbound federation traffic through (`outbound_federation_restricted_to`). This is useful when you have a worker setup with `federation_sender` instances responsible for sending outbound federation requests and want to make sure *all* outbound federation traffic goes through those instances. Before this change, the generic workers would still contact federation themselves for things like profile lookups, backfill, etc. This PR allows you to set more strict access controls/firewall for all workers and only allow the `federation_sender`'s to contact the outside world. The original code is from @erikjohnston's branches which I've gotten in-shape to merge.
* Add not_user_type param to the list accounts admin API (#15844)Michael Weimann2023-07-041-0/+78
| | | Signed-off-by: Michael Weimann <michaelw@element.io>
* Add login spam checker API (#15838)Erik Johnston2023-06-261-2/+106
|
* Remove experimental MSC2716 implementation to incrementally import history ↵Eric Eastwood2023-06-161-302/+0
| | | | | | | | | | | | into existing rooms (#15748) Context for why we're removing the implementation: - https://github.com/matrix-org/matrix-spec-proposals/pull/2716#issuecomment-1487441010 - https://github.com/matrix-org/matrix-spec-proposals/pull/2716#issuecomment-1504262734 Anyone wanting to continue MSC2716, should also address these leftover tasks: https://github.com/matrix-org/synapse/issues/10737 Closes https://github.com/matrix-org/synapse/issues/10737 in the fact that it is not longer necessary to track those things.
* push rules: fix internal conversion from _type to value (#15781)Mathieu Velten2023-06-161-0/+67
| | | Also fix wrong rule names for `is_user_mention` and `is_room_mention`.
* Clear event caches when we purge history (#15609)Erik Johnston2023-06-081-3/+0
| | | | | | | This should help a little with #13476 --------- Co-authored-by: Patrick Cloke <patrickc@matrix.org>
* Check required power levels earlier in createRoom handler. (#15695)Grant McLean2023-06-071-0/+37
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Check required power levels earlier in createRoom handler. - If a server was configured to reject the creation of rooms with E2EE enabled (by specifying an unattainably high power level for "m.room.encryption" in default_power_level_content_override), the 403 error was not being triggered until after the room was created and before the "m.room.power_levels" was sent. This allowed a user to access the partially-configured room and complete the setup of E2EE and power levels manually. - This change causes the power level overrides to be checked earlier and the request to be rejected before the user gains access to the room. - A new `_validate_room_config` method is added to contain checks that should be run before a room is created. - The new test case confirms that a user request is rejected by the new validation method. Signed-off-by: Grant McLean <grant@catalyst.net.nz> * Add a changelog file. * Formatting fix for black. * Remove unneeded line from test. --------- Signed-off-by: Grant McLean <grant@catalyst.net.nz>
* `N + 3`: Read from column `full_user_id` rather than `user_id` of tables ↵Shay2023-06-021-1/+3
| | | | `profiles` and `user_filters` (#15649)
* Add a catch-all * to the supported relation types when redacting (#15705)Mathieu Velten2023-06-021-3/+101
| | | This is an update to MSC3912 implementation
* Implement stable support for MSC3882 to allow an existing device/session to ↵Hugh Nimmo-Smith2023-06-013-18/+104
| | | | | | | | generate a login token for use on a new device/session (#15388) Implements stable support for MSC3882; this involves updating Synapse's support to match the MSC / the spec says. Continue to support the unstable version to allow clients to transition.
* Add stubs package for lxml. (#15697)Patrick Cloke2023-05-311-1/+1
| | | | | | | The stubs have some issues so this has some generous cast and ignores in it, but it is better than not having stubs. Note that confusing that Element is a function which creates _Element instances (and similarly for Comment).
* Tests for JWKS endpointHugh Nimmo-Smith2023-05-301-0/+106
|
* Refactor config to be an experimental featureHugh Nimmo-Smith2023-05-301-7/+10
| | | | Also enforce you can't combine it with incompatible config options
* Test MSC2965 implementation: well-known discovery documentHugh Nimmo-Smith2023-05-301-0/+38
|
* Rename blacklist/whitelist internally. (#15620)Patrick Cloke2023-05-191-27/+21
| | | | Avoid renaming configuration settings for now and rename internal code to use blocklist and allowlist instead.
* Do not allow deactivated users to login with JWT. (#15624)Patrick Cloke2023-05-191-2/+18
| | | | | To improve the organization of this code it moves the JWT login checks to a separate handler and then fixes the bug (and a deprecation warning).
* Handle missing previous read marker event. (#15464)Nick Mills-Barrett2023-05-181-0/+147
| | | | | If the previous read marker is pointing to an event that no longer exists (e.g. due to retention) then assume that the newly given read marker is newer.
* Update Mutual Rooms (MSC2666) implementation (#15621)Jonathan de Jong2023-05-181-2/+4
| | | | | | | | To track changes in MSC2666: - The change from `/mutual_rooms/{user_id}` to `/mutual_rooms?user_id={user_id}`. - The addition of `next_batch_token` (and logic). - Unstable flag now being `uk.half-shot.msc2666.query_mutual_rooms`. - The error code when your own user is requested.
* Apply url_preview_url_blacklist to oEmbed and pre-cached images (#15601)Patrick Cloke2023-05-161-4/+190
| | | | | | | | There are two situations which were previously not properly checked: 1. If the requested URL was replaced with an oEmbed URL, then the oEmbed URL was not checked against url_preview_url_blacklist. 2. Follow-up URLs (either via autodiscovery of oEmbed or to pre-cache images) were not checked against url_preview_url_blacklist.
* Add config option to prevent media downloads from listed domains. (#15197)Travis Ralston2023-05-091-0/+139
| | | | | | | This stops media (and thumbnails) from being accessed from the listed domains. It does not delete any already locally cached media, but will prevent accessing it. Note that admin APIs are unaffected by this change.
* Move ThirdPartyEventRules into module_api/callbacks (#15535)Andrew Morgan2023-05-041-21/+35
|
* Remove references to supporting per-user flag for msc2654 (#15522)Shay2023-05-021-6/+2
|
* Initial implementation of MSC3981: recursive relations API (#15315)Patrick Cloke2023-05-021-0/+120
| | | | | | | | | | | Adds an optional keyword argument to the /relations API which will recurse a limited number of event relationships. This will cause the API to return not just the events related to the parent event, but also events related to those related to the parent event, etc. This is disabled by default behind an experimental configuration flag and is currently implemented using prefixed parameters.
* Add an admin API endpoint to support per-user feature flags (#15344)Shay2023-04-281-0/+127
|
* Add column `full_user_id` to tables `profiles` and `user_filters`. (#15458)Shay2023-04-262-7/+27
|
* Move Spam Checker callbacks to a dedicated file (#15453)Andrew Morgan2023-04-181-7/+19
|
* Implement MSC2174: move redacts to a content property. (#15395)Patrick Cloke2023-04-131-2/+37
| | | | | | | This moves `redacts` from being a top-level property to a `content` property in a new room version. MSC2176 (which was previously implemented) states to not `redact` this property.
* Delete server-side backup keys when deactivating an account. (#15181)Shay2023-04-041-0/+157
|
* Revert pruning of old devices (#15360)Erik Johnston2023-03-311-47/+0
| | | | | | | | | * Revert "Fix registering a device on an account with lots of devices (#15348)" This reverts commit f0d8f66eaaacfa75bed65bc5d0c602fbc5339c85. * Revert "Delete stale non-e2e devices for users, take 3 (#15183)" This reverts commit 78cdb72cd6b0e007c314d9fed9f629dfc5b937a6.
* Fix registering a device on an account with lots of devices (#15348)Erik Johnston2023-03-291-0/+47
| | | Fixes up #15183
* Make cleaning up pushers depend on the device_id instead of the token_id ↵Quentin Gliech2023-03-241-2/+2
| | | | | | | | | | | | | | (#15280) This makes it so that we rely on the `device_id` to delete pushers on logout, instead of relying on the `access_token_id`. This ensures we're not removing pushers on token refresh, and prepares for a world without access token IDs (also known as the OIDC). This actually runs the `set_device_id_for_pushers` background update, which was forgotten in #13831. Note that for backwards compatibility it still deletes pushers based on the `access_token` until the background update finishes.
* Make `POST /_matrix/client/v3/rooms/{roomId}/report/{eventId}` endpoint ↵Andrew Morgan2023-03-211-0/+37
| | | | return 404 if event exists, but the user lacks access (#15300)
* Make `EventHandler.get_event` return `None` when the requested event is not ↵Andrew Morgan2023-03-211-0/+5
| | | | found (#15298)
* Fix error when sending message into deleted room. (#15235)Erik Johnston2023-03-211-0/+15
| | | | | | | | When a room is deleted in Synapse we remove the event forward extremities in the room, so if (say a bot) tries to send a message into the room we error out due to not being able to calculate prev events for the new event *before* we check if the sender is in the room. Fixes #8094
* Separate HTTP preview code and URL previewer. (#15269)Patrick Cloke2023-03-201-18/+16
| | | Separates REST layer code from the actual URL previewing.
* Move Account Validity callbacks to a dedicated file (#15237)Andrew Morgan2023-03-161-3/+2
|
* Merge branch 'release-v1.79' into developDavid Robertson2023-03-131-9/+10
|\
| * Fix missing conditional for registering ↵Andrew Morgan2023-03-101-9/+10
| | | | | | | | `on_remove_user_third_party_identifier` module api callbacks (#15227
* | Pass the Requester down to the HttpTransactionCache. (#15200)Quentin Gliech2023-03-071-13/+42
|/
* Stop applying edits to event contents (MSC3925). (#15193)Patrick Cloke2023-03-061-49/+10
| | | | | | | | | | | Enables MSC3925 support by default, which: * Includes the full edit event in the bundled aggregations of an edited event. * Stops modifying the original event's content to return the new content from the edit event. This is a backwards-incompatible change that is considered to be "correct" by the spec.
* Implementation of MSC3967: Don't require UIA for initial upload of cross ↵Hugh Nimmo-Smith2023-03-021-0/+141
| | | | signing keys (#15077)
* Remove support for aggregating reactions (#15172)Richard van der Hoff2023-02-281-150/+28
| | | | | | | | | | It turns out that no clients rely on server-side aggregation of `m.annotation` relationships: it's just not very useful as currently implemented. It's also non-trivial to calculate. I want to remove it from MSC2677, so to keep the implementation in line, let's remove it here.
* Admin API endpoint to delete a reported event (#15116)Dirk Klimpel2023-02-281-2/+141
| | | | | | | | | | | | | | | | | | | | | * Admin api to delete event report * lint + tests * newsfile * Apply suggestions from code review Co-authored-by: David Robertson <david.m.robertson1@gmail.com> * revert changes - move to WorkerStore * update unit test * Note that timestamp is in millseconds --------- Co-authored-by: David Robertson <david.m.robertson1@gmail.com>
* Add module API callbacks for adding and deleting local 3PID associations (#15044Andrew Morgan2023-02-271-0/+121
|
* Refactor media modules. (#15146)Patrick Cloke2023-02-279-2147/+5
| | | | | | | * Removes the `v1` directory from `test.rest.media.v1`. * Moves the non-REST code from `synapse.rest.media.v1` to `synapse.media`. * Flatten the `v1` directory from `synapse.rest.media`, but leave compatiblity with 3rd party media repositories and spam checkers.
* Batch up storing state groups when creating new room (#14918)Shay2023-02-241-2/+2
|
* Do not fail completely if oEmbed autodiscovery fails. (#15092)Patrick Cloke2023-02-231-3/+41
| | | | | | Previously if an autodiscovered oEmbed request failed (e.g. the oEmbed endpoint is down or does not exist) then the entire URL preview would fail. Instead we now return everything we can, even if this additional request fails.
* Bump black from 22.12.0 to 23.1.0 (#15103)dependabot[bot]2023-02-2224-58/+5
|
* Fix-up type hints in tests/server.py. (#15084)Patrick Cloke2023-02-172-31/+41
| | | | | This file was being ignored by mypy, we remove that and add the missing type hints & deal with any fallout.
* Use mypy 1.0 (#15052)David Robertson2023-02-162-1/+4
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Update mypy and mypy-zope * Remove unused ignores These used to suppress ``` synapse/storage/engines/__init__.py:28: error: "__new__" must return a class instance (got "NoReturn") [misc] ``` and ``` synapse/http/matrixfederationclient.py:1270: error: "BaseException" has no attribute "reasons" [attr-defined] ``` (note that we check `hasattr(e, "reasons")` above) * Avoid empty body warnings, sometimes by marking methods as abstract E.g. ``` tests/handlers/test_register.py:58: error: Missing return statement [empty-body] tests/handlers/test_register.py:108: error: Missing return statement [empty-body] ``` * Suppress false positive about `JaegerConfig` Complaint was ``` synapse/logging/opentracing.py:450: error: Function "Type[Config]" could always be true in boolean context [truthy-function] ``` * Fix not calling `is_state()` Oops! ``` tests/rest/client/test_third_party_rules.py:428: error: Function "Callable[[], bool]" could always be true in boolean context [truthy-function] ``` * Suppress false positives from ParamSpecs ```` synapse/logging/opentracing.py:971: error: Argument 2 to "_custom_sync_async_decorator" has incompatible type "Callable[[Arg(Callable[P, R], 'func'), **P], _GeneratorContextManager[None]]"; expected "Callable[[Callable[P, R], **P], _GeneratorContextManager[None]]" [arg-type] synapse/logging/opentracing.py:1017: error: Argument 2 to "_custom_sync_async_decorator" has incompatible type "Callable[[Arg(Callable[P, R], 'func'), **P], _GeneratorContextManager[None]]"; expected "Callable[[Callable[P, R], **P], _GeneratorContextManager[None]]" [arg-type] ```` * Drive-by improvement to `wrapping_logic` annotation * Workaround false "unreachable" positives See https://github.com/Shoobx/mypy-zope/issues/91 ``` tests/http/test_proxyagent.py:626: error: Statement is unreachable [unreachable] tests/http/test_proxyagent.py:762: error: Statement is unreachable [unreachable] tests/http/test_proxyagent.py:826: error: Statement is unreachable [unreachable] tests/http/test_proxyagent.py:838: error: Statement is unreachable [unreachable] tests/http/test_proxyagent.py:845: error: Statement is unreachable [unreachable] tests/http/federation/test_matrix_federation_agent.py:151: error: Statement is unreachable [unreachable] tests/http/federation/test_matrix_federation_agent.py:452: error: Statement is unreachable [unreachable] tests/logging/test_remote_handler.py:60: error: Statement is unreachable [unreachable] tests/logging/test_remote_handler.py:93: error: Statement is unreachable [unreachable] tests/logging/test_remote_handler.py:127: error: Statement is unreachable [unreachable] tests/logging/test_remote_handler.py:152: error: Statement is unreachable [unreachable] ``` * Changelog * Tweak DBAPI2 Protocol to be accepted by mypy 1.0 Some extra context in: - https://github.com/matrix-org/python-canonicaljson/pull/57 - https://github.com/python/mypy/issues/6002 - https://mypy.readthedocs.io/en/latest/common_issues.html#covariant-subtyping-of-mutable-protocol-members-is-rejected * Pull in updated canonicaljson lib so the protocol check just works * Improve comments in opentracing I tried to workaround the ignores but found it too much trouble. I think the corresponding issue is https://github.com/python/mypy/issues/12909. The mypy repo has a PR claiming to fix this (https://github.com/python/mypy/pull/14677) which might mean this gets resolved soon? * Better annotation for INTERACTIVE_AUTH_CHECKERS * Drive-by AUTH_TYPE annotation, to remove an ignore
* Add final type hint to tests.unittest. (#15072)Patrick Cloke2023-02-1410-26/+43
| | | | Adds a return type to HomeServerTestCase.make_homeserver and deal with any variables which are no longer Any.
* Prevent clients from reporting nonexistent events. (#13779)reivilibre2023-02-141-0/+12
|
* Return read-only collections from `@cached` methods (#13755)Sean Quah2023-02-101-2/+2
| | | | | | | | | | | | | It's important that collections returned from `@cached` methods are not modified, otherwise future retrievals from the cache will return the modified collection. This applies to the return values from `@cached` methods and the values inside the dictionaries returned by `@cachedList` methods. It's not necessary for the dictionaries returned by `@cachedList` methods themselves to be read-only. Signed-off-by: Sean Quah <seanq@matrix.org> Co-authored-by: David Robertson <davidr@element.io>
* Add a class UnpersistedEventContext to allow for the batching up of storing ↵Shay2023-02-091-1/+3
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | state groups (#14675) * add class UnpersistedEventContext * modify create new client event to create unpersistedeventcontexts * persist event contexts after creation * fix tests to persist unpersisted event contexts * cleanup * misc lints + cleanup * changelog + fix comments * lints * fix batch insertion? * reduce redundant calculation * add unpersisted event classes * rework compute_event_context, split into function that returns unpersisted event context and then persists it * use calculate_context_info to create unpersisted event contexts * update typing * $%#^&* * black * fix comments and consolidate classes, use attr.s for class * requested changes * lint * requested changes * requested changes * refactor to be stupidly explicit * clearer renaming and flow * make partial state non-optional * update docstrings --------- Co-authored-by: Erik Johnston <erik@matrix.org>
* Respond correctly to unknown methods on known endpoints (#14605)Patrick Cloke2023-02-091-3/+6
| | | | Respond with a 405 error if a request is received on a known endpoint, but to an unknown method, per MSC3743.
* Typecheck tests.rest.media.v1.test_media_storage (#15008)David Robertson2023-02-071-18/+31
| | | | | | | | | | | * Fix MediaStorage type hint * Typecheck tests.rest.media.v1.test_media_storage * Changelog * Remove assert and make the comment succinct * Fix syntax for olddeps
* Add helper to parse an enum from query args & use it. (#14956)Patrick Cloke2023-02-011-1/+4
| | | | | | | | The `parse_enum` helper pulls an enum value from the query string (by delegating down to the parse_string helper with values generated from the enum). This is used to pull out "f" and "b" in most places and then we thread the resulting Direction enum throughout more code.
* Add missing type hints in tests (#14879)Patrick Cloke2023-01-261-14/+28
| | | | * FIx-up type hints in tests.logging. * Add missing type hints to test_transactions.
* Fix type hints in knocking tests. (#14887)Andrew Morgan2023-01-251-3/+1
|
* Faster joins: omit partial rooms from eager syncs until the resync completes ↵David Robertson2023-01-233-8/+10
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | (#14870) * Allow `AbstractSet` in `StrCollection` Or else frozensets are excluded. This will be useful in an upcoming commit where I plan to change a function that accepts `List[str]` to accept `StrCollection` instead. * `rooms_to_exclude` -> `rooms_to_exclude_globally` I am about to make use of this exclusion mechanism to exclude rooms for a specific user and a specific sync. This rename helps to clarify the distinction between the global config and the rooms to exclude for a specific sync. * Better function names for internal sync methods * Track a list of excluded rooms on SyncResultBuilder I plan to feed a list of partially stated rooms for this sync to ignore * Exclude partial state rooms during eager sync using the mechanism established in the previous commit * Track un-partial-state stream in sync tokens So that we can work out which rooms have become fully-stated during a given sync period. * Fix mutation of `@cached` return value This was fouling up a complement test added alongside this PR. Excluding a room would mean the set of forgotten rooms in the cache would be extended. This means that room could be erroneously considered forgotten in the future. Introduced in #12310, Synapse 1.57.0. I don't think this had any user-visible side effects (until now). * SyncResultBuilder: track rooms to force as newly joined Similar plan as before. We've omitted rooms from certain sync responses; now we establish the mechanism to reintroduce them into future syncs. * Read new field, to present rooms as newly joined * Force un-partial-stated rooms to be newly-joined for eager incremental syncs only, provided they're still fully stated * Notify user stream listeners to wake up long polling syncs * Changelog * Typo fix Co-authored-by: Sean Quah <8349537+squahtx@users.noreply.github.com> * Unnecessary list cast Co-authored-by: Sean Quah <8349537+squahtx@users.noreply.github.com> * Rephrase comment Co-authored-by: Sean Quah <8349537+squahtx@users.noreply.github.com> * Another comment Co-authored-by: Sean Quah <8349537+squahtx@users.noreply.github.com> * Fixup merge(?) * Poke notifier when receiving un-partial-stated msg over replication * Fixup merge whoops Thanks MV :) Co-authored-by: Mathieu Velen <mathieuv@matrix.org> Co-authored-by: Mathieu Velten <mathieuv@matrix.org> Co-authored-by: Sean Quah <8349537+squahtx@users.noreply.github.com>
* Change default room version to 10. Implements MSC3904 (#14111)Catalan Lover2023-01-181-3/+9
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Change Documentation to have v10 as default room version * Change Default Room version to 10 * Add changelog entry for default room version swap * Add changelog entry for v10 default room version in docs * Clarify doc changelog entry Co-authored-by: David Robertson <david.m.robertson1@gmail.com> * Improve Documentation changes. Co-authored-by: David Robertson <david.m.robertson1@gmail.com> * Update Changelog entry to have correct format Co-authored-by: David Robertson <david.m.robertson1@gmail.com> * Update Spec Version to 1.5 * Only need 1 changelog. * Fix test. * Update "Changed in" line Co-authored-by: David Robertson <david.m.robertson1@gmail.com> Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com> Co-authored-by: Patrick Cloke <patrickc@matrix.org>
* Implement MSC3925: changes to bundling of edits (#14811)Richard van der Hoff2023-01-101-55/+130
| | | | | | | | Two parts to this: * Bundle the whole of the replacement with any edited events. This is backwards-compatible so I haven't put it behind a flag. * Optionally, inhibit server-side replacement of edited events. This has scope to break things, so it is currently disabled by default.
* Unescape HTML entities in oEmbed titles. (#14781)Jeyachandran Rathnam2023-01-091-0/+10
| | | | | | | It doesn't seem valid that HTML entities should appear in the title field of oEmbed responses, but a popular WordPress plug-in seems to do it. There should not be harm in unescaping these.
* Disable sending confirmation email when 3pid is disabled #14682 (#14725)Jeyachandran Rathnam2023-01-091-25/+5
| | | | | | | | | | | * Fixes #12277 :Disable sending confirmation email when 3pid is disabled * Fix test_add_email_if_disabled test case to reflect changes to enable_3pid_changes flag * Add changelog file * Rename newsfragment. Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
* Respond with proper error responses on unknown paths. (#14621)Patrick Cloke2022-12-083-4/+4
| | | | Returns a proper 404 with an errcode of M_RECOGNIZED for unknown endpoints per MSC3743.
* Reject receipt requests with invalid room or event IDs. (#14632)Nick Mills-Barrett2022-12-071-0/+76
| | | | If the room or event IDs are empty or of an invalid form they should be rejected.
* Stop using deprecated `keyIds` param on /key/v2/server (#14525)Richard van der Hoff2022-11-301-4/+1
| | | | Fixes #14523.
* Move MSC3030 `/timestamp_to_event` endpoint to stable v1 location (#14471)Eric Eastwood2022-11-281-6/+1
| | | | | | | | Fix https://github.com/matrix-org/synapse/issues/14390 - Client API: `/_matrix/client/unstable/org.matrix.msc3030/rooms/<roomID>/timestamp_to_event?ts=<timestamp>&dir=<direction>` -> `/_matrix/client/v1/rooms/<roomID>/timestamp_to_event?ts=<timestamp>&dir=<direction>` - Federation API: `/_matrix/federation/unstable/org.matrix.msc3030/timestamp_to_event/<roomID>?ts=<timestamp>&dir=<direction>` -> `/_matrix/federation/v1/timestamp_to_event/<roomID>?ts=<timestamp>&dir=<direction>` Complement test changes: https://github.com/matrix-org/complement/pull/559
* Implement message forward pagination from start when no from is given, fixes ↵Benjamin Kampmann2022-11-241-0/+40
| | | | | #12383 (#14149) Fixes https://github.com/matrix-org/synapse/issues/12383
* Add a type hint for `get_device_handler()` and fix incorrect types. (#14055)Patrick Cloke2022-11-221-1/+4
| | | | | This was the last untyped handler from the HomeServer object. Since it was being treated as Any (and thus unchecked) it was being used incorrectly in a few places.
* Batch fetch bundled references (#14508)Patrick Cloke2022-11-221-2/+2
| | | | | | | | | Avoid an n+1 query problem and fetch the bundled aggregations for m.reference relations in a single query instead of a query per event. This applies similar logic for as was previously done for edits in 8b309adb436c162510ed1402f33b8741d71fc058 (#11660; threads in b65acead428653b988351ae8d7b22127a22039cd (#11752); and annotations in 1799a54a545618782840a60950ef4b64da9ee24d (#14491).
* Batch fetch bundled annotations (#14491)Patrick Cloke2022-11-221-2/+2
| | | | | | | | Avoid an n+1 query problem and fetch the bundled aggregations for m.annotation relations in a single query instead of a query per event. This applies similar logic for as was previously done for edits in 8b309adb436c162510ed1402f33b8741d71fc058 (#11660) and threads in b65acead428653b988351ae8d7b22127a22039cd (#11752).
* Add an Admin API endpoint for looking up users based on 3PID (#14405)Ashish Kumar2022-11-111-13/+94
|
* Correctly create power level event during initial room creation (#14361)Shay2022-11-071-2/+2
|
* Fix /refresh endpoint version (#14364)Tulir Asokan2022-11-041-8/+8
|
* Implement MSC3912: Relation-based redactions (#14260)Brendan Abolivier2022-11-032-4/+306
| | | Co-authored-by: Sean Quah <8349537+squahtx@users.noreply.github.com>
* Fix dehydrated device REST checks (#14336)David Robertson2022-10-311-0/+34
|
* Support OIDC backchannel logouts (#11414)Quentin Gliech2022-10-312-28/+417
| | | | | | | If configured an OIDC IdP can log a user's session out of Synapse when they log out of the identity provider. The IdP sends a request directly to Synapse (and must be configured with an endpoint) when a user logs out.
* Merge branch 'master' into developOlivier Wilkinson (reivilibre)2022-10-281-3/+51
|\
| * Fix room creation being rate limited too aggressively since Synapse v1.69.0. ↵reivilibre2022-10-281-3/+51
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | (#14314) * Introduce a test for the old behaviour which we want to restore * Reintroduce the old behaviour in a simpler way * Newsfile Signed-off-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org> * Use 1 credit instead of 2 for creating a room: be more lenient than before Notably, the UI in Element Web was still broken after restoring to prior behaviour. After discussion, we agreed that it would be sensible to increase the limit. Signed-off-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org>
* | Merge branch 'release-v1.70' into developOlivier Wilkinson (reivilibre)2022-10-251-33/+77
|\|
| * Properly update the threads table when thread events are redacted. (#14248)Patrick Cloke2022-10-211-33/+77
| | | | | | | | | | | | | | | | | | When the last event in a thread is redacted we need to update the threads table: * Find the new latest event in the thread and store it into the table; or * Remove the thread from the table if it is no longer a thread (i.e. all events in the thread were redacted).
* | Refactor OIDC tests to better mimic an actual OIDC provider. (#13910)Quentin Gliech2022-10-253-88/+120
| | | | | | | | | | | | | | | | | | This implements a fake OIDC server, which intercepts calls to the HTTP client. Improves accuracy of tests by covering more internal methods. One particular example was the ID token validation, which previously mocked. This uncovered an incorrect dependency: Synapse actually requires at least authlib 0.15.1, not 0.14.0.
* | Show erasure status when listing users in the Admin API (#14205)Tadeusz Sośnierz2022-10-211-1/+34
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Show erasure status when listing users in the Admin API * Use USING when joining erased_users * Add changelog entry * Revert "Use USING when joining erased_users" This reverts commit 30bd2bf106415caadcfdbdd1b234ef2b106cc394. * Make the erased check work on postgres * Add a testcase for showing erased user status * Appease the style linter * Explicitly convert `erased` to bool to make SQLite consistent with Postgres This also adds us an easy way in to fix the other accidentally integered columns. * Move erasure status test to UsersListTestCase * Include user erased status when fetching user info via the admin API * Document the erase status in user_admin_api * Appease the linter and mypy * Signpost comments in tests Co-authored-by: Tadeusz Sośnierz <tadeusz@sosnierz.com> Co-authored-by: David Robertson <david.m.robertson1@gmail.com>
* | Use servlets for /key/ endpoints. (#14229)Patrick Cloke2022-10-201-2/+2
|/ | | | | To fix the response for unknown endpoints under that prefix. See MSC3743.
* Fix MSC3030 `/timestamp_to_event` returning `outliers` that it has no idea ↵Eric Eastwood2022-10-181-0/+65
| | | | | | | | | | | | | | | | | | | | | | | | | | | whether are near a gap or not (#14215) Fix MSC3030 `/timestamp_to_event` endpoint returning `outliers` that it has no idea whether are near a gap or not (and therefore unable to determine whether it's actually the closest event). The reason Synapse doesn't know whether an `outlier` is next to a gap is because our gap checks rely on entries in the `event_edges`, `event_forward_extremeties`, and `event_backward_extremities` tables which is [not the case for `outliers`](https://github.com/matrix-org/synapse/blob/2c63cdcc3f1aa4625e947de3c23e0a8133c61286/docs/development/room-dag-concepts.md#outliers). Also fixes MSC3030 Complement `can_paginate_after_getting_remote_event_from_timestamp_to_event_endpoint` test flake. Although this acted flakey in Complement, if `sync_partial_state` raced and beat us before `/timestamp_to_event`, then even if we retried the failing `/context` request it wouldn't work until we made this Synapse change. With this PR, Synapse will never return an `outlier` event so that test will always go and ask over federation. Fix https://github.com/matrix-org/synapse/issues/13944 ### Why did this fail before? Why was it flakey? Sleuthing the server logs on the [CI failure](https://github.com/matrix-org/synapse/actions/runs/3149623842/jobs/5121449357#step:5:5805), it looks like `hs2:/timestamp_to_event` found `$NP6-oU7mIFVyhtKfGvfrEQX949hQX-T-gvuauG6eurU` as an `outlier` event locally. Then when we went and asked for it via `/context`, since it's an `outlier`, it was filtered out of the results -> `You don't have permission to access that event.` This is reproducible when `sync_partial_state` races and persists `$NP6-oU7mIFVyhtKfGvfrEQX949hQX-T-gvuauG6eurU` as an `outlier` before we evaluate `get_event_for_timestamp(...)`. To consistently reproduce locally, just add a delay at the [start of `get_event_for_timestamp(...)`](https://github.com/matrix-org/synapse/blob/cb20b885cb4bd1648581dd043a184d86fc8c7a00/synapse/handlers/room.py#L1470-L1496) so it always runs after `sync_partial_state` completes. ```py from twisted.internet import task as twisted_task d = twisted_task.deferLater(self.hs.get_reactor(), 3.5) await d ``` In a run where it passes, on `hs2`, `get_event_for_timestamp(...)` finds a different event locally which is next to a gap and we request from a closer one from `hs1` which gets backfilled. And since the backfilled event is not an `outlier`, it's returned as expected during `/context`. With this PR, Synapse will never return an `outlier` event so that test will always go and ask over federation.
* Implementation of HTTP 307 response for MSC3886 POST endpoint (#14018)Hugh Nimmo-Smith2022-10-181-0/+45
| | | | Co-authored-by: reivilibre <olivier@librepush.net> Co-authored-by: Andrew Morgan <andrewm@element.io>
* Support filtering the /messages API by relation type (MSC3874). (#14148)Patrick Cloke2022-10-172-138/+8
| | | Gated behind an experimental configuration flag.
* Do not allow a None-limit on PaginationConfig. (#14146)Patrick Cloke2022-10-141-1/+2
| | | | | | | The callers either set a default limit or manually handle a None-limit later on (by setting a default value). Update the callers to always instantiate PaginationConfig with a default limit and then assume the limit is non-None.
* Stabilize the threads API. (#14175)Patrick Cloke2022-10-141-18/+29
| | | | | | | Stabilize the threads API (MSC3856) by supporting (only) the v1 path for the endpoint. This also marks the API as safe for workers since it is a read-only API.
* Add an API for listing threads in a room. (#13394)Patrick Cloke2022-10-131-0/+151
| | | | | | | | | Implement the /threads endpoint from MSC3856. This is currently unstable and behind an experimental configuration flag. It includes a background update to backfill data, results from the /threads endpoint will be partial until that finishes.
* Be more lenient in the oEmbed response parsing. (#14089)Patrick Cloke2022-10-071-1/+102
| | | | | | Attempt to parse any valid information from an oEmbed response (instead of bailing at the first unexpected data). This should allow for more partial oEmbed data to be returned, resulting in better / more URL previews, even if those URL previews are only partial.
* Fix handling of public rooms filter with a network tuple. (#14053)Patrick Cloke2022-10-051-10/+31
| | | | | | | | Fixes two related bugs: * The handling of `[null]` for a `room_types` filter was incorrect. * The ordering of arguments when providing both a network tuple and room type field was incorrect.
* Do not return unspecced original_event field when using the stable ↵Patrick Cloke2022-10-031-5/+8
| | | | | | | | | | /relations endpoint. (#14025) Keep the old behavior (of including the original_event field) for any requests to the /unstable version of the endpoint, but do not include the field when the /v1 version is used. This should avoid new clients from depending on this field, but will not help with current dependencies.
* Add query parameter `ts` to allow appservices set the `origin_server_ts` for ↵lukasdenk2022-10-031-2/+117
| | | | | | | state events. (#11866) MSC3316 declares that both /rooms/{roomId}/send and /rooms/{roomId}/state should accept a ts parameter for appservices. This change expands support to /state and adds tests.
* Skip filtering during push if there are no push actions (#13992)Erik Johnston2022-09-301-2/+2
|
* Allow admins to require a manual approval process before new accounts can be ↵Brendan Abolivier2022-09-295-7/+297
| | | | used (using MSC3866) (#13556)
* Expose MSC3882 only be under an unstable endpoint. (#13868)Hugh Nimmo-Smith2022-09-291-7/+9
|
* Prepatory work for batching events to send (#13487)Shay2022-09-281-2/+2
| | | This PR begins work on batching up events during the creation of a room. The PR splits out the creation and sending/persisting of the events. The first three events in the creation of the room-creating the room, joining the creator to the room, and the power levels event are sent sequentially, while the subsequent events are created and collected to be sent at the end of the function. This is currently done by appending them to a list and then iterating over the list to send, the next step (after this PR) would be to send and persist the collected events as a batch.
* Support the stable dir parameter for /relations. (#13920)Patrick Cloke2022-09-271-4/+2
| | | | | | | Since MSC3715 has passed FCP, the stable parameter can be used. This currently falls back to the unstable parameter if the stable parameter is not provided (and MSC3715 support is enabled in the configuration).
* Properly paginate forward in the /relations API. (#13840)Patrick Cloke2022-09-221-1/+28
| | | | | This fixes a bug where the `/relations` API with `dir=f` would skip the first item of each page (except the first page), causing incomplete data to be returned to the client.
* Implementation of MSC3882 login token request (#13722)Hugh Nimmo-Smith2022-09-211-0/+132
|
* Support enabling/disabling pushers (from MSC3881) (#13799)Brendan Abolivier2022-09-211-1/+1
| | | Partial implementation of MSC3881
* Add an admin API endpoint to find a user based on its external ID in an auth ↵Quentin Gliech2022-09-161-0/+87
| | | | provider. (#13810)
* A third batch of Pydantic validation for rest/client/account.py (#13736)David Robertson2022-09-151-3/+26
|
* Add a `MXCUri` class to make working with mxc uri's easier. (#13162)Andrew Morgan2022-09-151-64/+38
|
* Add timestamp to user's consent (#13741)Dirk Klimpel2022-09-081-0/+1
| | | Co-authored-by: reivilibre <olivier@librepush.net>
* Cancel the processing of key query requests when they time out. (#13680)reivilibre2022-09-071-0/+29
|
* Add Admin API to Fetch Messages Within a Particular Window (#13672)Connor Davis2022-09-071-1/+157
| | | This adds two new admin APIs that allow us to fetch messages from a room within a particular time.
* Remove support for unstable private read receipts (#13653)Šimon Brandner2022-09-011-26/+11
| | | Signed-off-by: Šimon Brandner <simon.bra.ag@gmail.com>
* Drop support for calling `/_matrix/client/v3/rooms/{roomId}/invite` without ↵Jacek Kuśnierz2022-08-313-3/+25
| | | | | | | an `id_access_token` (#13241) Fixes #13206 Signed-off-by: Jacek Kusnierz jacek.kusnierz@tum.de
* Fix admin List Room API return type on sqlite (#13509)David Robertson2022-08-311-5/+14
|
* Directly lookup local membership instead of getting all members in a room ↵Eric Eastwood2022-08-241-6/+6
| | | | | first (`get_users_in_room` mis-use) (#13608) See https://github.com/matrix-org/synapse/pull/13575#discussion_r953023755
* Drop support for delegating email validation, round 2 (#13596)David Robertson2022-08-231-1/+1
|
* Fix that sending server notices fail if avatar is `None` (#13566)Dirk Klimpel2022-08-231-0/+56
| | | Indroduced in #11846.
* `synapse.api.auth.Auth` cleanup: make permission-related methods use ↵Quentin Gliech2022-08-222-5/+5
| | | | | | | | | `Requester` instead of the `UserID` (#13024) Part of #13019 This changes all the permission-related methods to rely on the Requester instead of the UserID. This is a first step towards enabling scoped access tokens at some point, since I expect the Requester to have scope-related informations in it. It also changes methods which figure out the user/device/appservice out of the access token to return a Requester instead of something else. This avoids having store-related objects in the methods signatures.
* Implement MSC3852: Expose `last_seen_user_agent` to users for their own ↵Andrew Morgan2022-08-191-1/+91
| | | | devices; also expose to Admin API (#13549)
* Fix a bug in the `/event_reports` Admin API which meant that the total count ↵reivilibre2022-08-171-0/+27
| | | | | could be larger than the number of results you can actually query for. (#13525) Co-authored-by: Brendan Abolivier <babolivier@matrix.org>
* Add forgotten status to Room Details API (#13503)Dirk Klimpel2022-08-171-0/+1
|
* Use Pydantic to systematically validate a first batch of endpoints in ↵David Robertson2022-08-152-5/+58
| | | | `synapse.rest.client.account`. (#13188)
* Use literals in place of `HTTPStatus` constants in tests (#13488)Dirk Klimpel2022-08-1011-447/+176
| | | | | | | | | * Use literals in place of `HTTPStatus` constants in tests * newsfile * code style * code style
* Use literals in place of `HTTPStatus` constants in tests (#13479)Dirk Klimpel2022-08-0911-141/+140
| | | | | | | | | | Replace - `HTTPStatus.NOT_FOUND` - `HTTPStatus.FORBIDDEN` - `HTTPStatus.UNAUTHORIZED` - `HTTPStatus.CONFLICT` - `HTTPStatus.CREATED` Signed-off-by: Dirk Klimpel <dirk@klimpel.org>
* Use literals in place of `HTTPStatus` constants in tests (#13469)Dirk Klimpel2022-08-0812-331/+328
|
* Support stable identifiers for MSC2285: private read receipts. (#13273)Šimon Brandner2022-08-051-19/+39
| | | | | This adds support for the stable identifiers of MSC2285 while continuing to support the unstable identifiers behind the configuration flag. These will be removed in a future version.
* Use literals in place of `HTTPStatus` constants in tests (#13463)Dirk Klimpel2022-08-059-162/+147
|
* Fix rooms not being properly excluded from incremental sync (#13408)Brendan Abolivier2022-08-041-0/+21
|
* Return 404 or member list when getting joined_members after leaving (#13374)andrew do2022-08-031-0/+15
| | | | | | Signed-off-by: Andrew Doh <andrewddo@gmail.com> Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com> Co-authored-by: Andrew Morgan <andrewm@element.io> Co-authored-by: Brendan Abolivier <babolivier@matrix.org>
* Merge tag 'v1.64.0rc2' into developRichard van der Hoff2022-07-291-1/+1
|\ | | | | | | | | | | | | Synapse 1.64.0rc2 (2022-07-29) ============================== This RC reintroduces support for `account_threepid_delegates.email`, which was removed in 1.64.0rc1. It remains deprecated and will be removed altogether in a future release. ([\#13406](https://github.com/matrix-org/synapse/issues/13406))
| * Revert "Drop support for delegating email validation (#13192)" (#13406)3nprob2022-07-291-1/+1
| | | | | | | | | | Reverts commit fa71bb18b527d1a3e2629b48640ea67fff2f8c59, and tweaks documentation. Signed-off-by: 3nprob <git@3n.anonaddy.com>
* | Use stable prefixes for MSC3827: filtering of `/publicRooms` by room type ↵Šimon Brandner2022-07-271-3/+2
| | | | | | | | | | | | (#13370) Signed-off-by: Šimon Brandner <simon.bra.ag@gmail.com>
* | Add missing type hints for tests.unittest. (#13397)Patrick Cloke2022-07-272-3/+5
| |
* | Implement MSC3848: Introduce errcodes for specific event sending failures ↵Will Hunt2022-07-271-2/+3
| | | | | | | | | | (#13343) Implements MSC3848
* | Make minor clarifications to the error messages given when we fail to join a ↵reivilibre2022-07-271-1/+4
|/ | | | room via any server. (#13160)
* Rate limit joins per-room (#13276)David Robertson2022-07-191-2/+2
|
* Update expected DB query count when creating a room (#13307)Andrew Morgan2022-07-181-2/+2
|
* Remove unnecessary `json.dumps` from tests (#13303)Dirk Klimpel2022-07-1710-159/+113
|
* Use HTTPStatus constants in place of literals in tests. (#13297)Dirk Klimpel2022-07-154-209/+274
|
* Provide more info why we don't have any thumbnails to serve (#13038)Eric Eastwood2022-07-151-8/+62
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Fix https://github.com/matrix-org/synapse/issues/13016 ## New error code and status ### Before Previously, we returned a `404` for `/thumbnail` which isn't even in the spec. ```json { "errcode": "M_NOT_FOUND", "error": "Not found [b'hs1', b'tefQeZhmVxoiBfuFQUKRzJxc']" } ``` ### After What does the spec say? > 400: The request does not make sense to the server, or the server cannot thumbnail the content. For example, the client requested non-integer dimensions or asked for negatively-sized images. > > *-- https://spec.matrix.org/v1.1/client-server-api/#get_matrixmediav3thumbnailservernamemediaid* Now with this PR, we respond with a `400` when we don't have thumbnails to serve and we explain why we might not have any thumbnails. ```json { "errcode": "M_UNKNOWN", "error": "Cannot find any thumbnails for the requested media ([b'example.com', b'12345']). This might mean the media is not a supported_media_format=(image/jpeg, image/jpg, image/webp, image/gif, image/png) or that thumbnailing failed for some other reason. (Dynamic thumbnails are disabled on this server.)", } ``` > Cannot find any thumbnails for the requested media ([b'example.com', b'12345']). This might mean the media is not a supported_media_format=(image/jpeg, image/jpg, image/webp, image/gif, image/png) or that thumbnailing failed for some other reason. (Dynamic thumbnails are disabled on this server.) --- We still respond with a 404 in many other places. But we can iterate on those later and maybe keep some in some specific places after spec updates/clarification: https://github.com/matrix-org/matrix-spec/issues/1122 We can also iterate on the bugs where Synapse doesn't thumbnail when it should in other issues/PRs.
* Don't pull out the full state when storing state (#13274)Erik Johnston2022-07-151-2/+2
|
* Rip out auth-event reconciliation code (#12943)Richard van der Hoff2022-07-141-9/+2
| | | | | | | There is a corner in `_check_event_auth` (long known as "the weird corner") where, if we get an event with auth_events which don't match those we were expecting, we attempt to resolve the diffence between our state and the remote's with a state resolution. This isn't specced, and there's general agreement we shouldn't be doing it. However, it turns out that the faster-joins code was relying on it, so we need to introduce something similar (but rather simpler) for that.
* Optimise room creation event lookups part 2 (#13224)Nick Mills-Barrett2022-07-131-4/+4
|
* Fix "add user" admin api error when request contains a "msisdn" threepid ↵Thomas Weston2022-07-131-0/+35
| | | | | | (#13263) Co-authored-by: Thomas Weston <thomas.weston@clearspancloud.com> Co-authored-by: David Robertson <david.m.robertson1@gmail.com>
* Drop support for delegating email validation (#13192)Richard van der Hoff2022-07-121-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Drop support for delegating email validation Delegating email validation to an IS is insecure (since it allows the owner of the IS to do a password reset on your HS), and has long been deprecated. It will now cause a config error at startup. * Update unit test which checks for email verification Give it an `email` config instead of a threepid delegate * Remove unused method `requestEmailToken` * Simplify config handling for email verification Rather than an enum and a boolean, all we need here is a single bool, which says whether we are or are not doing email verification. * update docs * changelog * upgrade.md: fix typo * update version number this will be in 1.64, not 1.63 * update version number this one too
* expose whether a room is a space in the Admin API (#13208)andrew do2022-07-121-3/+10
|
* Don't pull out the full state when calculating push actions (#13078)Erik Johnston2022-07-111-2/+2
|
* Reduce event lookups during room creation by passing known event IDs (#13210)Nick Mills-Barrett2022-07-111-0/+15
| | | | | | | | Inspired by the room batch handler, this uses previous event inserts to pre-populate prev events during room creation, reducing the number of queries required to create a room. Signed off by Nick @ Beeper (@Fizzadar)
* Uniformize spam-checker API, part 5: expand other spam-checker callbacks to ↵David Teller2022-07-113-10/+249
| | | | | | return `Tuple[Codes, dict]` (#13044) Signed-off-by: David Teller <davidt@element.io> Co-authored-by: Brendan Abolivier <babolivier@matrix.org>
* annotate tests.server.FakeChannel (#13136)David Robertson2022-07-045-9/+14
|
* Implement MSC3827: Filtering of `/publicRooms` by room type (#13031)Šimon Brandner2022-06-291-3/+89
| | | | Signed-off-by: Šimon Brandner <simon.bra.ag@gmail.com>
* Merge branch 'master' into developAndrew Morgan2022-06-281-0/+17
|\
| * Merge pull request from GHSA-22p3-qrh9-cx32reivilibre2022-06-281-0/+17
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Make _iterate_over_text easier to read by using simple data structures * Prefer a set of tags to ignore In my tests, it's 4x faster to check for containment in a set of this size * Add a stack size limit to _iterate_over_text * Continue accepting the case where there is no body element * Use an early return instead for None Co-authored-by: Richard van der Hoff <richard@matrix.org>
* | Add Cross-Origin-Resource-Policy header to thumbnail and download media ↵Robert Long2022-06-271-0/+20
| | | | | | | | endpoints (#12944)
* | validate room alias before interacting with the room directory (#13106)santhoshivan232022-06-221-0/+13
| |
* | Simplify the alias deletion logic as an application service. (#13093)Quentin Gliech2022-06-171-0/+34
| |
* | Improve URL previews for sites with only Twitter card information. (#13056)Patrick Cloke2022-06-161-0/+41
| | | | | | | | | | | | Pull out `twitter:` meta tags when generating a preview and use it to augment any `og:` meta tags. Prefers Open Graph information over Twitter card information.
* | Add custom well-known (#13035)Jacek Kuśnierz2022-06-161-0/+22
| | | | | | Co-authored-by: David Robertson <david.m.robertson1@gmail.com>
* | Add more tests for room upgrades (#13074)Sean Quah2022-06-151-5/+78
| | | | | | | | Signed-off-by: Sean Quah <seanq@element.io>
* | Replace pyjwt with authlib in `org.matrix.login.jwt` (#13011)Hannes Lerchl2022-06-151-21/+23
| |
* | Fix `destination_is` errors seen in sentry. (#13041)David Robertson2022-06-141-0/+8
| | | | | | | | | | | | * Rename test_fedclient to match its source file * Require at least one destination to be truthy * Explicitly validate user ID in profile endpoint GETs Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
* | Uniformize spam-checker API, part 4: port other spam-checker callbacks to ↵David Teller2022-06-131-5/+170
| | | | | | | | | | return `Union[Allow, Codes]`. (#12857) Co-authored-by: Brendan Abolivier <babolivier@matrix.org>
* | Move the (unstable) `dir` parameter for /relations behind an experimental ↵Patrick Cloke2022-06-081-0/+1
| | | | | | | | | | | | flag. (#12984) MSC3715 defines this parameter, but the unstable version of it should be behind an experimental flag.
* | Test cancellation at every `await` during request handling (#12674)Sean Quah2022-06-071-0/+100
|/ | | | | | * Add tests for `/rooms/<room_id>/members` cancellation. * Add tests for `/rooms/<room_id>/state` cancellation. Signed-off-by: Sean Quah <seanq@element.io>
* Prevent local quarantined media from being claimed by media retention (#12972)Andrew Morgan2022-06-071-13/+96
|
* Remove remaining pieces of groups code. (#12966)Patrick Cloke2022-06-061-2/+0
| | | | | * Remove an unused stream ID generator. * Remove the now unused remote profile cache.