summary refs log tree commit diff
path: root/tests (follow)
Commit message (Collapse)AuthorAgeFilesLines
* Task scheduler: add replication notify for new task to launch ASAP (#16184)Mathieu Velten2023-08-281-18/+40
|
* Prepare unit tests for Python 3.12 (#16099)V024602023-08-259-71/+80
|
* Replace simple_async_mock with AsyncMock (#16180)Patrick Cloke2023-08-2514-160/+139
| | | | Python 3.8 has a native AsyncMock, use it instead of a custom implementation.
* Replace make_awaitable with AsyncMock (#16179)Patrick Cloke2023-08-2435-603/+506
| | | | Python 3.8 provides a native AsyncMock, we can replace the homegrown version we have.
* Fix typechecking with twisted trunk (#16121)David Robertson2023-08-241-8/+6
|
* Add `Retry-After` to M_LIMIT_EXCEEDED error responses (#16136)Will Hunt2023-08-242-6/+54
| | | Implements MSC4041 behind an experimental configuration flag.
* Stabilize support for MSC3958 (suppress notifications from edits). (#16113)Patrick Cloke2023-08-231-1/+0
|
* Filter out unwanted user_agents from udv. (#16124)Neil Johnson2023-08-231-0/+65
|
* Add tests for restoring the presence state after a restart. (#16151)Patrick Cloke2023-08-231-0/+116
|
* Properly update retry_last_ts when hitting the maximum retry interval (#16156)DeepBlueV7.X2023-08-231-0/+51
| | | | | | | | | | | | | | | | | | | | | * Properly update retry_last_ts when hitting the maximum retry interval This was broken in 1.87 when the maximum retry interval got changed from almost infinite to a week (and made configurable). fixes #16101 Signed-off-by: Nicolas Werner <nicolas.werner@hotmail.de> * Add changelog * Change fix + add test * Add comment --------- Signed-off-by: Nicolas Werner <nicolas.werner@hotmail.de> Co-authored-by: Mathieu Velten <mathieuv@matrix.org>
* Clean-up presence tests (#16158)Patrick Cloke2023-08-221-92/+37
| | | Reduce duplicated code & remove unused variables.
* Disable `m.3pid_changes` capability when MSC3861 is enabled. (#16134)Mathieu Velten2023-08-221-0/+5
|
* Add an admin endpoint to allow authorizing server to signal token ↵Shay2023-08-222-1/+95
| | | | revocations (#16125)
* Fix user directory test for deactivated support user. (#16157)Patrick Cloke2023-08-221-8/+11
| | | | Support users should not be added to the user directory after being deactivated.
* Properly call setup_background_tasks in unit tests. (#16150)Patrick Cloke2023-08-221-2/+0
| | | | | This should only be called on HomeServer objects which are configured to run background tasks, which is automatically (and properly) done via the call to setup().
* Disallow user_consent where experimental MSC3861 is enabled (#16127)Hugh Nimmo-Smith2023-08-221-0/+16
|
* Implements a task scheduler for resumable potentially long running tasks ↵Mathieu Velten2023-08-211-0/+186
| | | | (#15891)
* Speed up state res in rare case we don't have all events (#16116)Erik Johnston2023-08-181-48/+193
| | | | | | | If we don't have all the auth events in a room then not all state events will have a chain cover index. Even so, we can still use the chain cover index on the events that do have it, rather than bailing and using the slower functions. This situation should not arise for newly persisted rooms, as we check we have the full auth chain for each event, but can happen for existing rooms. c.f. #15245
* MSC3861: allow impersonation by an admin using a query param (#16132)Mathieu Velten2023-08-181-0/+35
|
* Allow filtering for admins in the list accounts admin API (#16114)Alexander Fechler2023-08-181-0/+38
|
* Add cache to `get_server_keys_json_for_remote` (#16123)Erik Johnston2023-08-181-38/+23
|
* Cache token introspection response from OIDC provider (#16117)Shay2023-08-171-0/+62
|
* Fix database performance of read/write worker locks (#16061)Erik Johnston2023-08-171-3/+4
| | | | | | | We were seeing serialization errors when taking out multiple read locks. The transactions were retried, so isn't causing any failures. Introduced in #15782.
* Run pyupgrade for python 3.7 & 3.8. (#16110)Patrick Cloke2023-08-1513-42/+43
|
* Implements admin API to lock an user (MSC3939) (#15870)Mathieu Velten2023-08-103-1/+114
|
* Support MSC3814: Dehydrated Devices Part 2 (#16010)Shay2023-08-082-7/+79
|
* Fix deletion for Dehydrated Devices (#16046)Shay2023-08-041-1/+138
|
* Move support for application service query parameter authorization behind a ↵Shay2023-08-031-4/+81
| | | | configuration option (#16017)
* Allow config of the backoff algorithm for the federation client. (#15754)Mathieu Velten2023-08-032-13/+18
| | | | | | | | | | | Adds three new configuration variables: * destination_min_retry_interval is identical to before (10mn). * destination_retry_multiplier is now 2 instead of 5, the maximum value will be reached slower. * destination_max_retry_interval is one day instead of (essentially) infinity. Capping this will cause destinations to continue to be retried sometimes instead of being lost forever. The previous value was 2 ^ 62 milliseconds.
* Add forward-compatibility for the redacts property (MSC2174). (#16013)Patrick Cloke2023-08-021-17/+50
| | | | | | | | The location of the redacts field changes in room version 11. Ensure it is copied to the *new* location for *old* room versions for forwards-compatibility with clients. Note that copying it to the *old* location for the *new* room version was previously handled.
* Update MSC3958 support to interact with intentional mentions. (#15992)Patrick Cloke2023-08-021-2/+19
| | | | | | | * Updates the rule ID. * Use `event_property_is` instead of `event_match`. This updates the implementation of MSC3958 to match the latest text from the MSC.
* Trim whitespace when setting display names (#16031)Mohit Rathee2023-08-011-0/+12
|
* Add ability to wait for locks and add locks to purge history / room deletion ↵Erik Johnston2023-07-313-2/+128
| | | | | (#15791) c.f. #13476
* Fix 404 on /profile when the display name is empty but not the avatar (#16012)Mathieu Velten2023-07-271-0/+10
|
* Remove support for legacy application service paths (#15964)Shay2023-07-261-53/+0
|
* Support MSC3814: Dehydrated Devices (#15929)Shay2023-07-242-3/+246
| | | | | | Signed-off-by: Nicolas Werner <n.werner@famedly.com> Co-authored-by: Nicolas Werner <n.werner@famedly.com> Co-authored-by: Nicolas Werner <89468146+nico-famedly@users.noreply.github.com> Co-authored-by: Hubert Chathi <hubert@uhoreg.ca>
* Properly handle redactions of creation events (#15973)Shay2023-07-231-2/+7
|
* Actually stop reading from column `user_id` of tables `profiles` (#15955)Shay2023-07-231-1/+1
|
* Merge branch 'master' into developOlivier Wilkinson (reivilibre)2023-07-182-0/+157
|\
| * Revert "Stop writing to column `user_id` of tables `profiles` and ↵Shay2023-07-182-0/+157
| | | | | | | | | | | | | | | | | | `user_filters`. (#15953) * Revert "Stop writing to column `user_id` of tables `profiles` and `user_filters` (#15787)" This reverts commit f25b0f88081bb436bef914983cff7087b54eba5f. * newsfragement
* | Support room version 11 (#15912)Patrick Cloke2023-07-182-18/+33
| | | | | | | | | | And fix a bug in the implementation of the updated redaction format (MSC2174) where the top-level redacts field was not properly added for backwards-compatibility.
* | Re-introduce the outbound federation proxy (#15913)Eric Eastwood2023-07-1817-39/+396
| | | | | | | | | | Allow configuring the set of workers to proxy outbound federation traffic through (`outbound_federation_restricted_to`). This is useful when you have a worker setup with `federation_sender` instances responsible for sending outbound federation requests and want to make sure *all* outbound federation traffic goes through those instances. Before this change, the generic workers would still contact federation themselves for things like profile lookups, backfill, etc. This PR allows you to set more strict access controls/firewall for all workers and only allow the `federation_sender`'s to contact the outside world.
* | Mark get_user_in_directory private since only used in tests (#15884)Mathieu Velten2023-07-122-12/+12
| |
* | Unix Sockets for HTTP Replication (#15708)Jason Little2023-07-112-2/+37
| | | | | | | | | | | | | | | | | | Unix socket support for `federation` and `client` Listeners has existed now for a little while(since [1.81.0](https://github.com/matrix-org/synapse/pull/15353)), but there was one last hold out before it could be complete: HTTP Replication communication. This should finish it up. The Listeners would have always worked, but would have had no way to be talked to/at. --------- Co-authored-by: Eric Eastwood <madlittlemods@gmail.com> Co-authored-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org> Co-authored-by: Eric Eastwood <erice@element.io>
* | Add + as an allowed character for Matrix IDs (MSC4009) (#15911)Patrick Cloke2023-07-111-6/+5
|/
* Add `Server` to Access-Control-Expose-Headers header (#15908)Michael Telatynski2023-07-111-1/+1
|
* Revert "Federation outbound proxy" (#15910)Eric Eastwood2023-07-1016-299/+37
| | | | | | Revert "Federation outbound proxy (#15773)" This reverts commit b07b14b494ae1dd564b4c44f844c9a9545b3d08a.
* Fix downgrading to previous version of Synapse (#15907)Erik Johnston2023-07-101-2/+6
| | | We do this by marking the constraint as deferrable.
* Stop writing to column `user_id` of tables `profiles` and `user_filters` ↵Shay2023-07-072-157/+0
| | | | (#15787)
* Remove `worker_replication_*` deprecated settings, with helpful errors on ↵Jason Little2023-07-072-29/+4
| | | | | | startup (#15860) Co-authored-by: reivilibre <oliverw@matrix.org>
* Federation outbound proxy (#15773)Eric Eastwood2023-07-0516-37/+299
| | | | | | | Allow configuring the set of workers to proxy outbound federation traffic through (`outbound_federation_restricted_to`). This is useful when you have a worker setup with `federation_sender` instances responsible for sending outbound federation requests and want to make sure *all* outbound federation traffic goes through those instances. Before this change, the generic workers would still contact federation themselves for things like profile lookups, backfill, etc. This PR allows you to set more strict access controls/firewall for all workers and only allow the `federation_sender`'s to contact the outside world. The original code is from @erikjohnston's branches which I've gotten in-shape to merge.
* Remove support for Python 3.7 (#15851)Eric Eastwood2023-07-051-8/+2
| | | Fix https://github.com/matrix-org/synapse/issues/15836
* Add basic read/write lock (#15782)Erik Johnston2023-07-051-1/+282
|
* Add foreign key constraint to `event_forward_extremities`. (#15751)Erik Johnston2023-07-052-4/+258
|
* Add not_user_type param to the list accounts admin API (#15844)Michael Weimann2023-07-041-0/+78
| | | Signed-off-by: Michael Weimann <michaelw@element.io>
* Add login spam checker API (#15838)Erik Johnston2023-06-261-2/+106
|
* Fix forgotten rooms missing in initial sync (#15815)Nicolas Werner2023-06-211-0/+21
| | | | | | | | If you leave a room and forget it, then rejoin it, the room would be missing from the next initial sync. fixes #13262 Signed-off-by: Nicolas Werner <n.werner@famedly.com>
* Allow for the configuration of max request retries and min/max retry delays ↵Mathieu Velten2023-06-211-1/+19
| | | | in the matrix federation client (#15783)
* Switch from `matrix://` to `matrix-federation://` scheme for internal ↵Eric Eastwood2023-06-202-19/+23
| | | | | | | Synapse routing of outbound federation traffic (#15806) `matrix://` is a registered specced scheme nowadays and doesn't make sense for our internal to Synapse use case anymore. ([discussion] (https://github.com/matrix-org/synapse/pull/15773#discussion_r1227598679))
* Remove experimental MSC2716 implementation to incrementally import history ↵Eric Eastwood2023-06-162-513/+0
| | | | | | | | | | | | into existing rooms (#15748) Context for why we're removing the implementation: - https://github.com/matrix-org/matrix-spec-proposals/pull/2716#issuecomment-1487441010 - https://github.com/matrix-org/matrix-spec-proposals/pull/2716#issuecomment-1504262734 Anyone wanting to continue MSC2716, should also address these leftover tasks: https://github.com/matrix-org/synapse/issues/10737 Closes https://github.com/matrix-org/synapse/issues/10737 in the fact that it is not longer necessary to track those things.
* push rules: fix internal conversion from _type to value (#15781)Mathieu Velten2023-06-161-0/+67
| | | Also fix wrong rule names for `is_user_mention` and `is_room_mention`.
* Regularly try to wake up dests instead of waiting for next PDU/EDU (#15743)Mathieu Velten2023-06-161-13/+9
|
* Fix unsafe hotserving behaviour for non-multimedia uploads. (#15680)Josh Qou2023-06-152-16/+16
| | | | | | | | | | | | | | | | | | | | | | | | | | | * Fix unsafe hotserving behaviour for non-multimedia uploads. * invert disposition assert * test_media_storage.py: run lint * test_base.py: /inline/attachment/s * Only return attachment for disposition type, update tests * Update synapse/media/_base.py Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com> * Update changelog.d/15680.bugfix Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com> * add attribution * Update changelog. --------- Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
* Merge branch 'release-v1.86' into developMathieu Velten2023-06-141-19/+1
|\
| * Revert "Allow for the configuration of max request retries and min/max retry ↵Mathieu Velten2023-06-141-19/+1
| | | | | | | | | | | | delays in the matrix federation client (#12504)" This reverts commit d84e66144dc12dacf71c987a2ba802dd59c0b68e.
* | Prefill events after invalidate not before when persisting events (#15758)Jason Little2023-06-141-0/+49
| | | | | | | | | | Fixes #15757
* | Replace `EventContext` fields `prev_group` and `delta_ids` with field ↵Shay2023-06-133-6/+13
|/ | | | `state_group_deltas` (#15233)
* Allow for the configuration of max request retries and min/max retry delays ↵Shay2023-06-091-1/+19
| | | | | | | in the matrix federation client (#12504) Co-authored-by: Mathieu Velten <mathieuv@matrix.org> Co-authored-by: Erik Johnston <erik@matrix.org>
* Clear event caches when we purge history (#15609)Erik Johnston2023-06-083-8/+5
| | | | | | | This should help a little with #13476 --------- Co-authored-by: Patrick Cloke <patrickc@matrix.org>
* Check required power levels earlier in createRoom handler. (#15695)Grant McLean2023-06-071-0/+37
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Check required power levels earlier in createRoom handler. - If a server was configured to reject the creation of rooms with E2EE enabled (by specifying an unattainably high power level for "m.room.encryption" in default_power_level_content_override), the 403 error was not being triggered until after the room was created and before the "m.room.power_levels" was sent. This allowed a user to access the partially-configured room and complete the setup of E2EE and power levels manually. - This change causes the power level overrides to be checked earlier and the request to be rejected before the user gains access to the room. - A new `_validate_room_config` method is added to contain checks that should be run before a room is created. - The new test case confirms that a user request is rejected by the new validation method. Signed-off-by: Grant McLean <grant@catalyst.net.nz> * Add a changelog file. * Formatting fix for black. * Remove unneeded line from test. --------- Signed-off-by: Grant McLean <grant@catalyst.net.nz>
* Trace functions which return `Awaitable` (#15650)Eric Eastwood2023-06-061-11/+32
|
* Stabilize support for MSC3952: Intentional mentions. (#15520)Patrick Cloke2023-06-061-21/+13
|
* `N + 3`: Read from column `full_user_id` rather than `user_id` of tables ↵Shay2023-06-025-54/+26
| | | | `profiles` and `user_filters` (#15649)
* Add a catch-all * to the supported relation types when redacting (#15705)Mathieu Velten2023-06-021-3/+101
| | | This is an update to MSC3912 implementation
* Log when events are (unexpectedly) filtered out of responses in tests (#14213)Eric Eastwood2023-06-011-0/+12
| | | | | See https://github.com/matrix-org/synapse/pull/14095#discussion_r990335492 This is useful because when see that a relevant event is an `outlier` or `soft-failed`, then that's a good unexpected indicator explaining why it's not showing up. `filter_events_for_client` is used in `/sync`, `/messages`, `/context` which are all common end-to-end assertion touch points (also notifications, relations).
* Implement stable support for MSC3882 to allow an existing device/session to ↵Hugh Nimmo-Smith2023-06-014-20/+106
| | | | | | | | generate a login token for use on a new device/session (#15388) Implements stable support for MSC3882; this involves updating Synapse's support to match the MSC / the spec says. Continue to support the unstable version to allow clients to transition.
* Add stubs package for lxml. (#15697)Patrick Cloke2023-05-314-4/+20
| | | | | | | The stubs have some issues so this has some generous cast and ignores in it, but it is better than not having stubs. Note that confusing that Element is a function which creates _Element instances (and similarly for Comment).
* Add stricter mypy options (#15694)Patrick Cloke2023-05-311-1/+1
| | | | Enable warn_unused_configs, strict_concatenate, disallow_subclassing_any, and disallow_incomplete_defs.
* Enforce that an admin token also has the basic Matrix API scopeQuentin Gliech2023-05-301-1/+25
|
* Make the config tests spawn the homeserver only when neededQuentin Gliech2023-05-301-194/+154
|
* Reject tokens with multiple device scopesQuentin Gliech2023-05-301-1/+28
|
* Handle errors when introspecting tokensQuentin Gliech2023-05-302-2/+37
| | | | | This returns a proper 503 when the introspection endpoint is not working for some reason, which should avoid logging out clients in those cases.
* Disable incompatible Admin API endpointsQuentin Gliech2023-05-301-0/+19
|
* Make AS tokens work & allow ASes to /registerQuentin Gliech2023-05-301-2/+2
|
* Tests for JWKS endpointHugh Nimmo-Smith2023-05-302-11/+212
|
* Refactor config to be an experimental featureHugh Nimmo-Smith2023-05-303-13/+221
| | | | Also enforce you can't combine it with incompatible config options
* Test MSC2965 implementation: well-known discovery documentHugh Nimmo-Smith2023-05-301-0/+38
|
* Disable account related endpoints when using OAuth delegationQuentin Gliech2023-05-301-3/+177
|
* Actually enforce guest + return www-authenticate headerHugh Nimmo-Smith2023-05-301-3/+40
|
* Initial tests for OAuth delegationHugh Nimmo-Smith2023-05-301-0/+345
|
* Save the scopes in the requesterQuentin Gliech2023-05-301-0/+2
|
* Make the api.auth.Auth a ProtocolQuentin Gliech2023-05-303-6/+6
|
* Merge branch 'master' into developOlivier Wilkinson (reivilibre)2023-05-265-4/+44
|\
| * Fix a bug introduced in Synapse v1.84.0 where workers do not start up when ↵reivilibre2023-05-265-4/+44
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | no `instance_map` was provided. (#15672) * Fix #15669: always populate instance map even if it was empty * Fix some tests * Fix more tests * Newsfile Signed-off-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org> * CI fix: don't forget to update apt repository sources before installing olddeps deps * Add test testing the backwards compatibility --------- Signed-off-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org>
* | Process previously failed backfill events in the background (#15585)Eric Eastwood2023-05-242-0/+132
| | | | | | | | | | | | | | | | | | Process previously failed backfill events in the background because they are bound to fail again and we don't need to waste time holding up the request for something that is bound to fail again. Fix https://github.com/matrix-org/synapse/issues/13623 Follow-up to https://github.com/matrix-org/synapse/issues/13621 and https://github.com/matrix-org/synapse/issues/13622 Part of making `/messages` faster: https://github.com/matrix-org/synapse/issues/13356
* | Add requesting user id parameter to key claim methods in ↵Shay2023-05-241-1/+15
| | | | | | | | `TransportLayerClient` (#15663)
* | Add type hints to test_descriptors. (#15659)Patrick Cloke2023-05-241-93/+104
| | | | | | Require type hints in test_descriptors and add missing ones.
* | Improve type hints for cached decorator. (#15658)Patrick Cloke2023-05-242-59/+34
| | | | | | | | | | The cached decorators always return a Deferred, which was not properly propagated. It was close enough when wrapping coroutines, but failed if a bare function was wrapped.
* | Consolidate logic to check for deactivated users. (#15634)Patrick Cloke2023-05-231-38/+21
| | | | | | | | | | | | | | This moves the deactivated user check to the method which all login types call. Additionally updates the application service tests to be more realistic by removing invalid tests and fixing server names.
* | Remove duplicate timestamp from test logs (`_trial_temp/test.log`) (#15636)Eric Eastwood2023-05-221-4/+3
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Fix https://github.com/matrix-org/synapse/issues/15618 ### Before ``` 2023-05-17 22:51:36-0500 [-] 2023-05-17 22:51:36,889 - synapse.server - 338 - INFO - sentinel - Finished setting up. ``` ### After ``` 2023-05-19 18:16:20-0500 [-] synapse.server - 338 - INFO - sentinel - Finished setting up. ``` ### Dev notes The `Twisted.Logger` controls the `2023-05-19 18:16:20-0500 [-]` prefix, see : [`twisted/twisted` -> `src/twisted/logger/_format.py#L362-L374`](https://github.com/twisted/twisted/blob/34b161e66bc7c9f9efbb95e82c770a863933e498/src/twisted/logger/_format.py#L362-L374) And we delegate our logs to the Twisted Logger for the tests which puts it in `_trial_temp/test.log`
* | Properly parse event_fields in filters (#15607)Patrick Cloke2023-05-222-6/+39
| | | | | | | | | | | | | | | | The event_fields property in filters should use the proper escape rules, namely backslashes can be escaped with an additional backslash. This adds tests (adapted from matrix-js-sdk) and implements the logic to properly split the event_fields strings.
* | Limit the size of the `HomeServerConfig` cache in trial test runs (#15646)Sean Quah2023-05-221-16/+7
| | | | | | | | | | | | | | | | | | | | | | | | ...to try to control memory usage. `HomeServerConfig`s hold on to many Jinja2 objects, which come out to over 0.5 MiB per config. Over the course of a full test run, the cache grows to ~360 entries. Limit it to 8 entries. Part of #15622. Signed-off-by: Sean Quah <seanq@matrix.org>
* | Remove old R30 because R30v2 supercedes it (#10428)reivilibre2023-05-191-154/+0
| | | | | | | | | | R30v2 has been out since 2021-07-19 (https://github.com/matrix-org/synapse/pull/10332) and we started collecting stats on 2021-08-16. Since it's been over a year now (almost 2 years), this is enough grace period for us to now rip it out.
* | Rename blacklist/whitelist internally. (#15620)Patrick Cloke2023-05-1910-76/+70
| | | | | | | | Avoid renaming configuration settings for now and rename internal code to use blocklist and allowlist instead.
* | Do not allow deactivated users to login with JWT. (#15624)Patrick Cloke2023-05-191-2/+18
| | | | | | | | | | To improve the organization of this code it moves the JWT login checks to a separate handler and then fixes the bug (and a deprecation warning).
* | Remove experimental configuration flags & unstable values for faster joins ↵Patrick Cloke2023-05-191-32/+3
| | | | | | | | | | | | | | (#15625) Synapse will no longer send (or respond to) the unstable flags for faster joins. These were only available behind a configuration flag and handled in parallel with the stable flags.
* | Fix `HomeServer`s leaking during `trial` test runs (#15630)Sean Quah2023-05-191-2/+9
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This change fixes two memory leaks during `trial` test runs. Garbage collection is disabled during each test case and a gen-0 GC is run at the end of each test. However, when the gen-0 GC is run, the `TestCase` object usually still holds references to the `HomeServer` used during the test. As a result, the `HomeServer` gets promoted to gen-1 and then never garbage collected. Fix this by periodically running full GCs. Additionally, fix `HomeServer`s leaking after tests that touch inbound federation due to `FederationRateLimiter`s adding themselves to a global set, by turning the set into a `WeakSet`. Resolves #15622. Signed-off-by: Sean Quah <seanq@matrix.org>
* | Handle missing previous read marker event. (#15464)Nick Mills-Barrett2023-05-181-0/+147
| | | | | | | | | | If the previous read marker is pointing to an event that no longer exists (e.g. due to retention) then assume that the newly given read marker is newer.
* | Update Mutual Rooms (MSC2666) implementation (#15621)Jonathan de Jong2023-05-181-2/+4
| | | | | | | | | | | | | | | | To track changes in MSC2666: - The change from `/mutual_rooms/{user_id}` to `/mutual_rooms?user_id={user_id}`. - The addition of `next_batch_token` (and logic). - Unstable flag now being `uk.half-shot.msc2666.query_mutual_rooms`. - The error code when your own user is requested.
* | Apply url_preview_url_blacklist to oEmbed and pre-cached images (#15601)Patrick Cloke2023-05-162-4/+303
| | | | | | | | | | | | | | | | There are two situations which were previously not properly checked: 1. If the requested URL was replaced with an oEmbed URL, then the oEmbed URL was not checked against url_preview_url_blacklist. 2. Follow-up URLs (either via autodiscovery of oEmbed or to pre-cache images) were not checked against url_preview_url_blacklist.
* | Update code to refer to "workers". (#15606)Patrick Cloke2023-05-165-28/+15
| | | | | | | | A bunch of comments and variables are out of date and use obsolete terms.
* | Add not null constraint to column `full_user_id` of tables `profiles` and ↵Shay2023-05-162-0/+157
| | | | | | | | `user_filters` (#15537)
* | Fix subscriptable type usage in Python <3.9 (#15604)Eric Eastwood2023-05-161-1/+1
|/ | | | | | | | | | | | Fix the following `mypy` errors when running `mypy` with Python 3.7: ``` synapse/storage/controllers/stats.py:58: error: "Counter" is not subscriptable, use "typing.Counter" instead [misc] tests/test_state.py:267: error: "dict" is not subscriptable, use "typing.Dict" instead [misc] ``` Part of https://github.com/matrix-org/synapse/issues/15603 In Python 3.9, `typing` is deprecated and the types are subscriptable (generics) by default, https://peps.python.org/pep-0585/#implementation
* Revert "Fix subscriptable dict type"Eric Eastwood2023-05-151-1/+1
| | | | This reverts commit 55b08534a412f462251753f67308405ca4d02ebe.
* Fix subscriptable dict typeEric Eastwood2023-05-151-1/+1
| | | | | | | | | | Fix: ``` tests/test_state.py:267: error: "dict" is not subscriptable, use "typing.Dict" instead [misc] ``` In Python 3.9, `typing` is deprecated and the types are subscriptable (generics) by default, https://peps.python.org/pep-0585/#implementation
* Implement MSC3821 to update redaction rules (`third_party_invite.signed`) ↵Patrick Cloke2023-05-151-1/+74
| | | | | | (#15563) Updates the redaction rules to protect enough information that the event can still be properly verified.
* Implement MSC3389 to protect relations from redaction. (#15565)Patrick Cloke2023-05-151-0/+90
| | | | | | MSC3389 proposes protecting the relation type & parent event ID from redaction. This keeps the relation information intact after redaction which helps with some UX flaws (e.g. deleting an event causes it to no longer be in a thread, which is confusing).
* Remove `worker_replication_*` settings (#15491)Jason Little2023-05-115-10/+5
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Add master to the instance_map as part of Complement, have ReplicationEndpoint look at instance_map for master. * Fix typo in drive by. * Remove unnecessary worker_replication_* bits from unit tests and add master to instance_map(hopefully in the right place) * Several updates: 1. Switch from master to main for naming the main process in the instance_map. Add useful constants for easier adjustment of names in the future. 2. Add backwards compatibility for worker_replication_* to allow time to transition to new style. Make sure to prioritize declaring main directly on the instance_map. 3. Clean up old comments/commented out code. 4. Adjust unit tests to match with new code. 5. Adjust Complement setup infrastructure to only add main to the instance_map if workers are used and remove now unused options from the worker.yaml template. * Initial Docs upload * Changelog * Missed some commented out code that can go now * Remove TODO comment that no longer holds true. * Fix links in docs * More docs * Remove debug logging * Apply suggestions from code review Co-authored-by: reivilibre <olivier@librepush.net> * Apply suggestions from code review Co-authored-by: reivilibre <olivier@librepush.net> * Update version to latest, include completeish before/after examples in upgrade notes. * Fix up and docs too --------- Co-authored-by: reivilibre <olivier@librepush.net>
* HTTP Replication Client (#15470)Jason Little2023-05-091-0/+1
| | | | | | Separate out a HTTP client for replication in preparation for also supporting using UNIX sockets. The major difference from the base class is that this does not use treq to handle HTTP requests.
* Add config option to prevent media downloads from listed domains. (#15197)Travis Ralston2023-05-091-0/+139
| | | | | | | This stops media (and thumbnails) from being accessed from the listed domains. It does not delete any already locally cached media, but will prevent accessing it. Note that admin APIs are unaffected by this change.
* Check appservices for devices during a /user/devices query. (#15539)Patrick Cloke2023-05-051-1/+134
| | | | | | | | MSC3984 proxies /keys/query requests to appservices, but servers will can also requests devices / keys from the /user/devices endpoint. The formats are close enough that we can "proxy" that /user/devices to appservices (by calling /keys/query) and then change the format of the returned data before returning it over federation.
* Implement MSC4009 to widen the allowed Matrix ID grammar (#15536)Patrick Cloke2023-05-051-0/+13
| | | | | Behind a configuration flag this adds + to the list of allowed characters in Matrix IDs. The main feature this enables is using full E.164 phone numbers as Matrix IDs.
* Factor out an `is_mine_server_name` method (#15542)Sean Quah2023-05-051-2/+14
| | | | | | | | | | | | Add an `is_mine_server_name` method, similar to `is_mine_id`. Ideally we would use this consistently, instead of sometimes comparing against `hs.hostname` and other times reaching into `hs.config.server.server_name`. Also fix a bug in the tests where `hs.hostname` would sometimes differ from `hs.config.server.server_name`. Signed-off-by: Sean Quah <seanq@matrix.org>
* Move ThirdPartyEventRules into module_api/callbacks (#15535)Andrew Morgan2023-05-042-22/+38
|
* Merge branch 'release-v1.83' into developSean Quah2023-05-032-4/+4
|\
| * Revert "Reduce the size of the HTTP connection pool for non-pushers" (#15530) v1.83.0rc1Sean Quah2023-05-032-4/+4
| | | | | | | | | | | | | | | | | | #15514 introduced a regression where Synapse would encounter `PartialDownloadError`s when fetching OpenID metadata for certain providers on startup. Due to #8088, this prevents Synapse from starting entirely. Revert the change while we decide what to do about the regression.
* | Add config option to forget rooms automatically when users leave them (#15224)Sean Quah2023-05-031-0/+11
| | | | | | | | | | This is largely based off the stats and user directory updater code. Signed-off-by: Sean Quah <seanq@matrix.org>
* | Remove references to supporting per-user flag for msc2654 (#15522)Shay2023-05-021-6/+2
| |
* | Allow adding random delay to push (#15516)Erik Johnston2023-05-021-0/+37
|/ | | This is to discourage timing based profiling on the push gateways.
* Reduce the size of the HTTP connection pool for non-pushers. (#15514)Patrick Cloke2023-05-022-4/+4
| | | | | | | | | | | | | | Pushers tend to make many connections to the same HTTP host (e.g. a new event comes in, causes events to be pushed, and then the homeserver connects to the same host many times). Due to this the per-host HTTP connection pool size was increased, but this does not make sense for other SimpleHttpClients. Add a parameter for the connection pool and override it for pushers (making a separate SimpleHttpClient for pushers with the increased configuration). This returns the HTTP connection pool settings to the default Twisted ones for non-pusher HTTP clients.
* Initial implementation of MSC3981: recursive relations API (#15315)Patrick Cloke2023-05-021-0/+120
| | | | | | | | | | | Adds an optional keyword argument to the /relations API which will recurse a limited number of event relationships. This will cause the API to return not just the events related to the parent event, but also events related to those related to the parent event, etc. This is disabled by default behind an experimental configuration flag and is currently implemented using prefixed parameters.
* Add an admin API endpoint to support per-user feature flags (#15344)Shay2023-04-281-0/+127
|
* Add support for claiming multiple OTKs at once. (#15468)Patrick Cloke2023-04-272-24/+19
| | | | | | | MSC3983 provides a way to request multiple OTKs at once from appservices, this extends this concept to the Client-Server API. Note that this will likely be spit out into a separate MSC, but is currently part of MSC3983.
* Add column `full_user_id` to tables `profiles` and `user_filters`. (#15458)Shay2023-04-266-44/+56
|
* Add unstable /keys/claim endpoint which always returns fallback keys. (#15462)Patrick Cloke2023-04-251-6/+235
| | | | | | | | | | | | | It can be useful to always return the fallback key when attempting to claim keys. This adds an unstable endpoint for `/keys/claim` which always returns fallback keys in addition to one-time-keys. The fallback key(s) are not marked as "used" unless there are no corresponding OTKs. This is currently defined in MSC3983 (although likely to be split out to a separate MSC). The endpoint shape may change or be requested differently (i.e. a keyword parameter on the current endpoint), but the core logic should be reasonable.
* Finish type hints for federation client HTTP code. (#15465)Patrick Cloke2023-04-242-8/+8
|
* Remove legacy code of single user device resync api (#15418)Alok Kumar Singh2023-04-211-1/+3
| | | | | * Removed single-user resync usage and updated it to use multi-user counterpart Signed-off-by: Alok Kumar Singh alokaks601@gmail.com
* Modify StoreKeyFetcher to read from server_keys_json. (#15417)Patrick Cloke2023-04-204-48/+58
| | | | | | | | | | | | | | | | | | | | | | | | | | | Before this change: * `PerspectivesKeyFetcher` and `ServerKeyFetcher` write to `server_keys_json`. * `PerspectivesKeyFetcher` also writes to `server_signature_keys`. * `StoreKeyFetcher` reads from `server_signature_keys`. After this change: * `PerspectivesKeyFetcher` and `ServerKeyFetcher` write to `server_keys_json`. * `PerspectivesKeyFetcher` also writes to `server_signature_keys`. * `StoreKeyFetcher` reads from `server_keys_json`. This results in `StoreKeyFetcher` now using the results from `ServerKeyFetcher` in addition to those from `PerspectivesKeyFetcher`, i.e. keys which are directly fetched from a server will now be pulled from the database instead of refetched. An additional minor change is included to avoid creating a `PerspectivesKeyFetcher` (and checking it) if no `trusted_key_servers` are configured. The overall impact of this should be better usage of cached results: * If a server has no trusted key servers configured then it should reduce how often keys are fetched. * if a server's trusted key server does not have a requested server's keys cached then it should reduce how often keys are directly fetched.
* Speedup tests by caching HomeServerConfig instances (#15284)Val Lorentz2023-04-181-2/+60
| | | | | | | | | | | | | | | | | | | These two lines: ``` config_obj = HomeServerConfig() config_obj.parse_config_dict(config, "", "") ``` are called many times with the exact same value for `config`. As the test suite is CPU-bound and non-negligeably time is spent in `parse_config_dict`, this saves ~5% on the overall runtime of the Trial test suite (tested with both `-j2` and `-j12` on a 12t CPU). This is sadly rather limited, as the cache cannot be shared between processes (it contains at least jinja2.Template and RLock objects which aren't pickleable), and Trial tends to run close tests in different processes.
* Move Spam Checker callbacks to a dedicated file (#15453)Andrew Morgan2023-04-184-10/+22
|
* Delete pushers after calling on_logged_out module hook on device delete (#15410)Mathieu Velten2023-04-141-1/+50
|
* Convert async to normal tests in `TestSSOHandler` (#15433)Dirk Klimpel2023-04-141-4/+4
| | | | | * Convert async to normal tests in `TestSSOHandler` * newsfile
* Improve robustness when handling a perspective key response by deduplicating ↵reivilibre2023-04-133-21/+17
| | | | | | | | | | | | | | | received server keys. (#15423) * Change `store_server_verify_keys` to take a `Mapping[(str, str), FKR]` This is because we already can't handle duplicate keys — leads to cardinality violation * Newsfile Signed-off-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org> --------- Signed-off-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org>
* Implement MSC2174: move redacts to a content property. (#15395)Patrick Cloke2023-04-132-4/+47
| | | | | | | This moves `redacts` from being a top-level property to a `content` property in a new room version. MSC2176 (which was previously implemented) states to not `redact` this property.
* Throw if the appservice config list is the wrong type (#15425)Will Hunt2023-04-121-0/+40
| | | | | | | | | | | * raise a ConfigError on an invalid app_service_config_files * changelog * Move config check to read_config * Add test * Ensure list also contains strings
* Implement MSC3989 to redact the origin field. (#15393)Patrick Cloke2023-04-051-0/+7
| | | | This will be done in a future room version, for now an unstable room version is added which redacts the origin field.
* Delete server-side backup keys when deactivating an account. (#15181)Shay2023-04-041-0/+157
|
* Call appservices on modern paths, falling back to legacy paths. (#15317)Patrick Cloke2023-04-031-2/+55
| | | | | This uses the specced /_matrix/app/v1/... paths instead of the "legacy" paths. If the homeserver receives an error it will retry using the legacy path.
* Revert pruning of old devices (#15360)Erik Johnston2023-03-314-52/+3
| | | | | | | | | * Revert "Fix registering a device on an account with lots of devices (#15348)" This reverts commit f0d8f66eaaacfa75bed65bc5d0c602fbc5339c85. * Revert "Delete stale non-e2e devices for users, take 3 (#15183)" This reverts commit 78cdb72cd6b0e007c314d9fed9f629dfc5b937a6.
* to_device updates could be dropped when consuming the replication stream ↵Mathieu Velten2023-03-303-5/+93
| | | | | (#15349) Co-authored-by: reivilibre <oliverw@matrix.org>
* Speed up SQLite unit test CI (#15334)Erik Johnston2023-03-302-3/+36
| | | Tests now take 40% of the time.
* Implement MSC3984 to proxy /keys/query requests to appservices. (#15321)Patrick Cloke2023-03-301-1/+120
| | | | | If enabled, for users which are exclusively owned by an application service then the appservice will be queried for devices in addition to any information stored in the Synapse database.
* Fix spinloop during partial state sync when a prev event is in backoff (#15351)Sean Quah2023-03-301-4/+9
| | | | | | | | Previously, we would spin in a tight loop until `update_state_for_partial_state_event` stopped raising `FederationPullAttemptBackoffError`s. Replace the spinloop with a wait until the backoff period has expired. Signed-off-by: Sean Quah <seanq@matrix.org>
* Add the ability to enable/disable registrations when in the OIDC flow (#14978)Warren Bailey2023-03-301-1/+16
| | | Signed-off-by: Warren Bailey <warren@warrenbailey.net>
* Fix registering a device on an account with lots of devices (#15348)Erik Johnston2023-03-291-0/+47
| | | Fixes up #15183
* Delete stale non-e2e devices for users, take 3 (#15183)Erik Johnston2023-03-293-3/+5
| | | | | This should help reduce the number of devices e.g. simple bots the repeatedly login rack up. We only delete non-e2e devices as they should be safe to delete, whereas if we delete e2e devices for a user we may accidentally break their ability to receive e2e keys for a message.
* Implement MSC3983 to proxy /keys/claim queries to appservices. (#15314)Patrick Cloke2023-03-282-1/+134
| | | | | | Experimental support for MSC3983 is behind a configuration flag. If enabled, for users which are exclusively owned by an application service then the appservice will be queried for one-time keys *if* there are none uploaded to Synapse.
* Bump ruff from 0.0.252 to 0.0.259 (#15328)dependabot[bot]2023-03-282-3/+9
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Bump ruff from 0.0.252 to 0.0.259 Bumps [ruff](https://github.com/charliermarsh/ruff) from 0.0.252 to 0.0.259. - [Release notes](https://github.com/charliermarsh/ruff/releases) - [Changelog](https://github.com/charliermarsh/ruff/blob/main/BREAKING_CHANGES.md) - [Commits](https://github.com/charliermarsh/ruff/compare/v0.0.252...v0.0.259) --- updated-dependencies: - dependency-name: ruff dependency-type: direct:development update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> * Fix new warnings * Mypy * Newsfile --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Erik Johnston <erik@matrix.org>
* Make cleaning up pushers depend on the device_id instead of the token_id ↵Quentin Gliech2023-03-244-32/+28
| | | | | | | | | | | | | | (#15280) This makes it so that we rely on the `device_id` to delete pushers on logout, instead of relying on the `access_token_id`. This ensures we're not removing pushers on token refresh, and prepares for a world without access token IDs (also known as the OIDC). This actually runs the `set_device_id_for_pushers` background update, which was forgotten in #13831. Note that for backwards compatibility it still deletes pushers based on the `access_token` until the background update finishes.
* Reject mentions on the C-S API which are invalid. (#15311)Patrick Cloke2023-03-241-38/+56
| | | | | | Invalid mentions data received over the Client-Server API should be rejected with a 400 error. This will hopefully stop clients from sending invalid data, although does not help with data received over federation.
* Use immutabledict instead of frozendict (#15113)David Robertson2023-03-225-41/+43
| | | | | | | | | | Additionally: * Consistently use `freeze()` in test --------- Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com> Co-authored-by: 6543 <6543@obermui.de>
* Make `POST /_matrix/client/v3/rooms/{roomId}/report/{eventId}` endpoint ↵Andrew Morgan2023-03-211-0/+37
| | | | return 404 if event exists, but the user lacks access (#15300)
* Make `EventHandler.get_event` return `None` when the requested event is not ↵Andrew Morgan2023-03-211-0/+5
| | | | found (#15298)
* Fix error when sending message into deleted room. (#15235)Erik Johnston2023-03-211-0/+15
| | | | | | | | When a room is deleted in Synapse we remove the event forward extremities in the room, so if (say a bot) tries to send a message into the room we error out due to not being able to calculate prev events for the new event *before* we check if the sender is in the room. Fixes #8094
* Separate HTTP preview code and URL previewer. (#15269)Patrick Cloke2023-03-201-18/+16
| | | Separates REST layer code from the actual URL previewing.
* Add Synapse-Trace-Id to access-control-expose-headers header (#14974)Shay2023-03-201-0/+4
|
* Remove no-op send_command for Redis replication. (#15274)Patrick Cloke2023-03-161-63/+0
| | | | | With Redis commands do not need to be re-issued by the main process (they fan-out to all processes at once) and thus it is no longer necessary to worry about them reflecting recursively forever.
* Refresh remote profiles that have been marked as stale, in order to fill the ↵reivilibre2023-03-161-2/+185
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | user directory. [rei:userdirpriv] (#14756) * Scaffolding for background process to refresh profiles * Add scaffolding for background process to refresh profiles for a given server * Implement the code to select servers to refresh from * Ensure we don't build up multiple looping calls * Make `get_profile` able to respect backoffs * Add logic for refreshing users * When backing off, schedule a refresh when the backoff is over * Wake up the background processes when we receive an interesting state event * Add tests * Newsfile Signed-off-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org> * Add comment about 1<<62 --------- Signed-off-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org>
* Move Account Validity callbacks to a dedicated file (#15237)Andrew Morgan2023-03-161-3/+2
|
* Merge branch 'release-v1.79' into developDavid Robertson2023-03-134-15/+162
|\
| * Ensure fed-sender catchup does not block for full state (#15248)David Robertson2023-03-132-1/+117
| | | | | | | | * Reproduce bad scenario in test * Avoid catchup optimisation for partial state rooms
| * Refactor `filter_events_for_server` (#15240)David Robertson2023-03-101-5/+35
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Tweak docstring and type hint * Flip logic and provide better name * Separate decision from action * Track a set of strings, not EventBases * Require explicit boolean options from callers * Add explicit option for partial state rooms * Changelog * Rename param
| * Fix missing conditional for registering ↵Andrew Morgan2023-03-101-9/+10
| | | | | | | | `on_remove_user_third_party_identifier` module api callbacks (#15227
* | Pull in netaddr type hints. (#15231)Patrick Cloke2023-03-091-1/+1
| | | | | | And fix any issues from having those type hints.
* | Stabilize support for MSC3873: disambuguated event push keys. (#15190)Patrick Cloke2023-03-071-7/+3
| | | | | | | | | | | | | | This removes the experimental configuration option and always escapes the push rule condition keys. Also escapes any (experimental) push rule condition keys in the base rules which contain dot in a field name.
* | Pass the Requester down to the HttpTransactionCache. (#15200)Quentin Gliech2023-03-071-13/+42
| |
* | Stabilize support for MSC3966: event_property_contains push condition. (#15187)Patrick Cloke2023-03-072-18/+3
|/ | | | This removes the configuration flag & updates the identifiers to use the stable version.
* Stop applying edits to event contents (MSC3925). (#15193)Patrick Cloke2023-03-061-49/+10
| | | | | | | | | | | Enables MSC3925 support by default, which: * Includes the full edit event in the bundled aggregations of an edited event. * Stops modifying the original event's content to return the new content from the edit event. This is a backwards-incompatible change that is considered to be "correct" by the spec.
* Stabilize support for MSC3758: event_property_is push condition (#15185)Patrick Cloke2023-03-062-20/+5
| | | | This removes the configuration flag & updates the identifiers to use the stable version.
* Fix ICU tests on alpine / macOS. (#15177)65432023-03-031-0/+2
| | | | The word boundary behaviour is slightly different, consider it acceptable for the tests.
* Update intentional mentions (MSC3952) to depend on ↵Patrick Cloke2023-03-022-31/+4
| | | | | | | `exact_event_property_contains` (MSC3966). (#15051) This replaces the specific `is_user_mention` push rule condition used in MSC3952 with the generic `exact_event_property_contains` push rule condition from MSC3966.
* Implementation of MSC3967: Don't require UIA for initial upload of cross ↵Hugh Nimmo-Smith2023-03-021-0/+141
| | | | signing keys (#15077)
* Remove support for aggregating reactions (#15172)Richard van der Hoff2023-02-281-150/+28
| | | | | | | | | | It turns out that no clients rely on server-side aggregation of `m.annotation` relationships: it's just not very useful as currently implemented. It's also non-trivial to calculate. I want to remove it from MSC2677, so to keep the implementation in line, let's remove it here.
* Merge branch 'master' into developH. Shay2023-02-281-0/+61
|\
| * Fix bug where 5s delays would occasionally happen. (#15150)Erik Johnston2023-02-241-0/+61
| | | | | | This only affects deployments using workers.
* | Fix a long-standing bug where an initial sync would not respond to changes ↵reivilibre2023-02-281-0/+22
| | | | | | | | to the list of ignored users if there was an initial sync cached. (#15163)
* | Do not accept pattern_type from user input in push rules. (#15088)Patrick Cloke2023-02-281-0/+27
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Internally the push rules module uses a `pattern_type` property for `event_match` conditions (and `related_event_match`) to mark the condition as matching the current user's Matrix ID or localpart. This is leaky to the Client-Server API where a user can successfully set a condition which provides `pattern_type` instead of `pattern` (note that there's no benefit to doing this -- the user can just use their own Matrix ID or localpart instead). When serializing back to the client the `pattern_type` property is converted into a proper `pattern`. The following changes are made to avoid this: * Separate the `KnownCondition::EventMatch` enum value into `EventMatch` and `EventMatchType`, each with their own expected properties. (Note that a similar change is made for `RelatedEventMatch`.) * Make it such that the `pattern_type` variants serialize to the same condition kind, but cannot be deserialized (since they're only provided by base rules). * As a final tweak, convert `user_id` vs. `user_localpart` values into an enum.
* | Admin API endpoint to delete a reported event (#15116)Dirk Klimpel2023-02-281-2/+141
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Admin api to delete event report * lint + tests * newsfile * Apply suggestions from code review Co-authored-by: David Robertson <david.m.robertson1@gmail.com> * revert changes - move to WorkerStore * update unit test * Note that timestamp is in millseconds --------- Co-authored-by: David Robertson <david.m.robertson1@gmail.com>
* | Add module API callbacks for adding and deleting local 3PID associations (#15044Andrew Morgan2023-02-272-4/+123
| |
* | Refactor media modules. (#15146)Patrick Cloke2023-02-279-15/+15
| | | | | | | | | | | | | | * Removes the `v1` directory from `test.rest.media.v1`. * Moves the non-REST code from `synapse.rest.media.v1` to `synapse.media`. * Flatten the `v1` directory from `synapse.rest.media`, but leave compatiblity with 3rd party media repositories and spam checkers.
* | Batch up storing state groups when creating new room (#14918)Shay2023-02-247-20/+161
| |
* | Improve handling of non-ASCII characters in user directory search (#15143)Sean Quah2023-02-241-0/+133
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Fix a long-standing bug where non-ASCII characters in search terms, including accented letters, would not match characters in a different case. * Fix a long-standing bug where search terms using combining accents would not match display names using precomposed accents and vice versa. To fully take effect, the user directory must be rebuilt after this change. Fixes #14630. Signed-off-by: Sean Quah <seanq@matrix.org>
* | Do not fail completely if oEmbed autodiscovery fails. (#15092)Patrick Cloke2023-02-231-3/+41
| | | | | | | | | | | | Previously if an autodiscovered oEmbed request failed (e.g. the oEmbed endpoint is down or does not exist) then the entire URL preview would fail. Instead we now return everything we can, even if this additional request fails.
* | Fix a typo in MSC3873 config option. (#15138)Patrick Cloke2023-02-231-1/+1
| | | | | | | | Previously the experimental configuration option referred to the wrong MSC number.
* | Add information on uploaded media to user export command. (#15107)Dirk Klimpel2023-02-231-0/+29
| |
* | Bump black from 22.12.0 to 23.1.0 (#15103)dependabot[bot]2023-02-2245-101/+24
| |
* | Remove unused `room_alias` field from `/createRoom` response (#15093)David Robertson2023-02-224-11/+12
|/ | | | | | | | | | | | | | * Change `create_room` return type * Don't return room alias from /createRoom * Update other callsites * Fix up mypy complaints It looks like new_room_user_id is None iff new_room_id is None. It's a shame we haven't expressed this in a way that mypy can understand. * Changelog
* Fix a bug introduced in Synapse v1.74.0 where searching with colons when ↵reivilibre2023-02-202-1/+69
| | | | | | using ICU for search term tokenisation would fail with an error. (#15079) Co-authored-by: David Robertson <davidr@element.io>
* Fix-up type hints in tests/server.py. (#15084)Patrick Cloke2023-02-177-127/+225
| | | | | This file was being ignored by mypy, we remove that and add the missing type hints & deal with any fallout.
* Add account data to export command (#14969)Dirk Klimpel2023-02-171-0/+27
| | | | | | | | | | | * Add account data to to export command * newsfile * remove not needed function * update newsfile * adopt #14973
* Use mypy 1.0 (#15052)David Robertson2023-02-167-38/+65
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Update mypy and mypy-zope * Remove unused ignores These used to suppress ``` synapse/storage/engines/__init__.py:28: error: "__new__" must return a class instance (got "NoReturn") [misc] ``` and ``` synapse/http/matrixfederationclient.py:1270: error: "BaseException" has no attribute "reasons" [attr-defined] ``` (note that we check `hasattr(e, "reasons")` above) * Avoid empty body warnings, sometimes by marking methods as abstract E.g. ``` tests/handlers/test_register.py:58: error: Missing return statement [empty-body] tests/handlers/test_register.py:108: error: Missing return statement [empty-body] ``` * Suppress false positive about `JaegerConfig` Complaint was ``` synapse/logging/opentracing.py:450: error: Function "Type[Config]" could always be true in boolean context [truthy-function] ``` * Fix not calling `is_state()` Oops! ``` tests/rest/client/test_third_party_rules.py:428: error: Function "Callable[[], bool]" could always be true in boolean context [truthy-function] ``` * Suppress false positives from ParamSpecs ```` synapse/logging/opentracing.py:971: error: Argument 2 to "_custom_sync_async_decorator" has incompatible type "Callable[[Arg(Callable[P, R], 'func'), **P], _GeneratorContextManager[None]]"; expected "Callable[[Callable[P, R], **P], _GeneratorContextManager[None]]" [arg-type] synapse/logging/opentracing.py:1017: error: Argument 2 to "_custom_sync_async_decorator" has incompatible type "Callable[[Arg(Callable[P, R], 'func'), **P], _GeneratorContextManager[None]]"; expected "Callable[[Callable[P, R], **P], _GeneratorContextManager[None]]" [arg-type] ```` * Drive-by improvement to `wrapping_logic` annotation * Workaround false "unreachable" positives See https://github.com/Shoobx/mypy-zope/issues/91 ``` tests/http/test_proxyagent.py:626: error: Statement is unreachable [unreachable] tests/http/test_proxyagent.py:762: error: Statement is unreachable [unreachable] tests/http/test_proxyagent.py:826: error: Statement is unreachable [unreachable] tests/http/test_proxyagent.py:838: error: Statement is unreachable [unreachable] tests/http/test_proxyagent.py:845: error: Statement is unreachable [unreachable] tests/http/federation/test_matrix_federation_agent.py:151: error: Statement is unreachable [unreachable] tests/http/federation/test_matrix_federation_agent.py:452: error: Statement is unreachable [unreachable] tests/logging/test_remote_handler.py:60: error: Statement is unreachable [unreachable] tests/logging/test_remote_handler.py:93: error: Statement is unreachable [unreachable] tests/logging/test_remote_handler.py:127: error: Statement is unreachable [unreachable] tests/logging/test_remote_handler.py:152: error: Statement is unreachable [unreachable] ``` * Changelog * Tweak DBAPI2 Protocol to be accepted by mypy 1.0 Some extra context in: - https://github.com/matrix-org/python-canonicaljson/pull/57 - https://github.com/python/mypy/issues/6002 - https://mypy.readthedocs.io/en/latest/common_issues.html#covariant-subtyping-of-mutable-protocol-members-is-rejected * Pull in updated canonicaljson lib so the protocol check just works * Improve comments in opentracing I tried to workaround the ignores but found it too much trouble. I think the corresponding issue is https://github.com/python/mypy/issues/12909. The mypy repo has a PR claiming to fix this (https://github.com/python/mypy/pull/14677) which might mean this gets resolved soon? * Better annotation for INTERACTIVE_AUTH_CHECKERS * Drive-by AUTH_TYPE annotation, to remove an ignore
* Update intentional mentions (MSC3952) to depend on `exact_event_match` ↵Patrick Cloke2023-02-162-25/+16
| | | | | | | | | | (MSC3758). (#15037) This replaces the specific `is_room_mention` push rule condition used in MSC3952 with the generic `exact_event_match` push rule condition from MSC3758. No functionality changes due to this.
* Add final type hint to tests.unittest. (#15072)Patrick Cloke2023-02-1453-317/+432
| | | | Adds a return type to HomeServerTestCase.make_homeserver and deal with any variables which are no longer Any.
* Implement MSC3966: Add a push rule condition to search for a value in an ↵Patrick Cloke2023-02-141-3/+50
| | | | | | array. (#15045) The `exact_event_property_contains` condition can be used to search for a value inside of an array.
* Prevent clients from reporting nonexistent events. (#13779)reivilibre2023-02-141-0/+12
|
* Faster joins: don't stall when a user joins during a fast join (#14606)Mathieu Velten2023-02-101-20/+20
| | | | | | | | | | | | | | | | Fixes #12801. Complement tests are at https://github.com/matrix-org/complement/pull/567. Avoid blocking on full state when handling a subsequent join into a partial state room. Also always perform a remote join into partial state rooms, since we do not know whether the joining user has been banned and want to avoid leaking history to banned users. Signed-off-by: Mathieu Velten <mathieuv@matrix.org> Co-authored-by: Sean Quah <seanq@matrix.org> Co-authored-by: David Robertson <davidr@element.io>
* Return read-only collections from `@cached` methods (#13755)Sean Quah2023-02-101-2/+2
| | | | | | | | | | | | | It's important that collections returned from `@cached` methods are not modified, otherwise future retrievals from the cache will return the modified collection. This applies to the return values from `@cached` methods and the values inside the dictionaries returned by `@cachedList` methods. It's not necessary for the dictionaries returned by `@cachedList` methods themselves to be read-only. Signed-off-by: Sean Quah <seanq@matrix.org> Co-authored-by: David Robertson <davidr@element.io>
* Support for MSC3758: exact_event_match push condition (#14964)Patrick Cloke2023-02-101-4/+143
| | | | | This specifies to search for an exact value match, instead of string globbing. It only works across non-compound JSON values (null, boolean, integer, and strings).
* Add a class UnpersistedEventContext to allow for the batching up of storing ↵Shay2023-02-097-14/+43
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | state groups (#14675) * add class UnpersistedEventContext * modify create new client event to create unpersistedeventcontexts * persist event contexts after creation * fix tests to persist unpersisted event contexts * cleanup * misc lints + cleanup * changelog + fix comments * lints * fix batch insertion? * reduce redundant calculation * add unpersisted event classes * rework compute_event_context, split into function that returns unpersisted event context and then persists it * use calculate_context_info to create unpersisted event contexts * update typing * $%#^&* * black * fix comments and consolidate classes, use attr.s for class * requested changes * lint * requested changes * requested changes * refactor to be stupidly explicit * clearer renaming and flow * make partial state non-optional * update docstrings --------- Co-authored-by: Erik Johnston <erik@matrix.org>
* Respond correctly to unknown methods on known endpoints (#14605)Patrick Cloke2023-02-091-3/+6
| | | | Respond with a 405 error if a request is received on a known endpoint, but to an unknown method, per MSC3743.
* Proper types for `tests.module_api` (#15031)David Robertson2023-02-092-53/+79
| | | | | | | | | | | | | * -> None for test methods * A first batch of type fixes * Introduce common parent test case * Fixup that big test method * tests.module_api passes mypy * Changelog
* Add more missing type hints to tests. (#15028)Patrick Cloke2023-02-0812-93/+123
|
* Add missing type hints to tests. (#15027)Patrick Cloke2023-02-086-58/+69
|
* MSC3873: Escape keys when flattening dicts. (#15004)Patrick Cloke2023-02-081-0/+8
| | | | | | This disambiguates keys which attempt to match fields with a dot in them (e.g. m.relates_to). Disabled by default behind an experimental configuration flag.
* Typecheck tests.rest.media.v1.test_media_storage (#15008)David Robertson2023-02-071-18/+31
| | | | | | | | | | | * Fix MediaStorage type hint * Typecheck tests.rest.media.v1.test_media_storage * Changelog * Remove assert and make the comment succinct * Fix syntax for olddeps
* Proper types for tests.test_terms_auth (#15007)David Robertson2023-02-071-6/+13
| | | | | * Proper types for tests.test_terms_auth * Changelog
* Add tests for using _flatten_dict with an event. (#15002)Patrick Cloke2023-02-071-1/+62
|
* Properly typecheck types.http (#14988)David Robertson2023-02-0713-185/+289
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Tweak http types in Synapse AFACIS these are correct, and they make mypy happier on tests.http. * Type hints for test_proxyagent * type hints for test_srv_resolver * test_matrix_federation_agent * tests.http.server._base * tests.http.__init__ * tests.http.test_additional_resource * tests.http.test_client * tests.http.test_endpoint * tests.http.test_matrixfederationclient * tests.http.test_servlet * tests.http.test_simple_client * tests.http.test_site * One fixup in tests.server * Untyped defs * Changelog * Fixup syntax for Python 3.7 * Fix olddeps syntax * Use a twisted IPv4 addr for dummy_address * Fix typo, thanks Sean Co-authored-by: Sean Quah <8349537+squahtx@users.noreply.github.com> * Remove redundant `Optional` --------- Co-authored-by: Sean Quah <8349537+squahtx@users.noreply.github.com>
* Add missing types to test_state. (#14985)Patrick Cloke2023-02-061-61/+137
|
* Type hints for tests.federation (#14991)David Robertson2023-02-068-91/+125
| | | | | | | | | | | | | * Make tests.federation pass mypy * Untyped defs in tests.federation.transport * test methods return None * Remaining type hints in tests.federation * Changelog * Avoid an uncessary type-ignore
* Add missing type hints to tests.replication. (#14987)Patrick Cloke2023-02-0619-149/+189
|
* Type hints for tests.appservice (#14990)David Robertson2023-02-063-51/+100
| | | | | | | | | | | | | | | | | | | | * Accept a Sequence of events in synapse.appservice This avoids some casts/ignores in the tests I'm about to fixup. It seems that `List[Mock]` is not a subtype of `List[EventBase]`, but `Sequence[Mock]` is a subtype of `Sequence[EventBase]`. So presumably `Mock` is considered a subtype of anything, much like `Any`. * make tests.appservice.test_scheduler pass mypy * Extra hints in tests.appservice.test_scheduler * Extra hints in tests.appservice.test_api * Extra hints in tests.appservice.test_appservice * Disallow untyped defs * Changelog
* Properly typecheck tests.api (#14983)David Robertson2023-02-034-107/+134
|
* Implement MSC3958: suppress notifications from edits (#14960)Patrick Cloke2023-02-031-1/+41
| | | | | | | | Co-authored-by: Brad Murray <brad@beeper.com> Co-authored-by: Nick Barrett <nick@beeper.com> Copy the suppress_edits push rule from Beeper to implement MSC3958. https://github.com/beeper/synapse/blame/9415a1284b1bfb558bd66f28c24ca1611e6c6fa2/rust/src/push/base_rules.rs#L98-L114
* Properly typecheck tests.app (#14984David Robertson2023-02-033-14/+25
|
* Add a docstring & tests for _flatten_dict. (#14981)Patrick Cloke2023-02-031-1/+25
|
* Support the backwards compatibility features in MSC3952. (#14958)Patrick Cloke2023-02-032-58/+151
| | | | If the feature is enabled and the event has a `m.mentions` property, skip processing of the legacy mentions rules.
* Faster joins: Refactor handling of servers in room (#14954)Sean Quah2023-02-032-2/+2
| | | | | | | | | | | | | | Ensure that the list of servers in a partial state room always contains the server we joined off. Also refactor `get_partial_state_servers_at_join` to return `None` when the given room is no longer partial stated, to explicitly indicate when the room has partial state. Otherwise it's not clear whether an empty list means that the room has full state, or the room is partial stated, but the server we joined off told us that there are no servers in the room. Signed-off-by: Sean Quah <seanq@matrix.org>
* Reload the pyo3-log config when the Python logging config changes. (#14976)Patrick Cloke2023-02-031-0/+3
| | | | | | | | | | | Since pyo3-log is initialized very early in the Python start-up it caches the state of the loggers before they're fully initialized (and thus are essentially disabled). Whenever we reload the logging configuration we now also tell pyo3-log to discard any cached logging configuration it has; it will refetch the current logging configuration from Python at the next point it logs. This fixes Rust log lines not appearing in the homeserver logs.
* Add helper to parse an enum from query args & use it. (#14956)Patrick Cloke2023-02-011-1/+4
| | | | | | | | The `parse_enum` helper pulls an enum value from the query string (by delegating down to the parse_string helper with values generated from the enum). This is used to pull out "f" and "b" in most places and then we thread the resulting Direction enum throughout more code.
* Attempt to delete more duplicate rows in receipts_linearized table. (#14915)Patrick Cloke2023-02-011-1/+3
| | | | | | | | | The previous assumption was that the stream_id column was unique (for a room ID, receipt type, user ID tuple), but this turned out to be incorrect. Now find the max stream ID, then map this back to a database-specific row identifier and delete other rows which match the (room ID, receipt type, user ID) tuple, but *not* the row ID.
* Add more user information to export-data command. (#14894)Dirk Klimpel2023-02-011-0/+60
| | | | | | * The user's profile information. * The user's devices. * The user's connections / IP address information.
* Make sqlite database migrations transactional again, part two (#14926)Sean Quah2023-01-311-0/+96
| | | | | | | | | | | | | | | | #14910 fixed the regression introduced by #13873 where sqlite database migrations would no longer run inside a transaction. However, it committed the transaction before Synapse updated its bookkeeping of which migrations have been run, which means that migrations may be run again after they have completed successfully. Leave the transaction open at the end of `executescript`, to restore the old, correct behaviour. Also make the PostgreSQL behaviour consistent with SQLite. Fixes #14909. Signed-off-by: Sean Quah <seanq@matrix.org>
* Handle malformed values of `notification.room` in power level events (#14942)David Robertson2023-01-302-14/+111
| | | | | | | | | | | | | | | | | | | | | | | * Better test for bad values in power levels events The previous test only checked that Synapse didn't raise an exception, but didn't check that we had correctly interpreted the value of the dodgy power level. It also conflated two things: bad room notification levels, and bad user levels. There _is_ logic for converting the latter to integers, but we should test it separately. * Check we ignore types that don't convert to int * Handle `None` values in `notifications.room` * Changelog * Also test that bad values are rejected by event auth * Docstring * linter scripttttttttt
* Implement MSC3952: Intentional mentions (#14823)Patrick Cloke2023-01-272-7/+147
| | | | | | | | MSC3952 defines push rules which searches for mentions in a list of Matrix IDs in the event body, instead of searching the entire event body for display name / local part. This is implemented behind an experimental configuration flag and does not yet implement the backwards compatibility pieces of the MSC.
* Add missing type hints in tests (#14879)Patrick Cloke2023-01-265-38/+69
| | | | * FIx-up type hints in tests.logging. * Add missing type hints to test_transactions.
* Fix type hints in typing edu unit tests (#14886)Andrew Morgan2023-01-263-42/+65
|
* Add missing type hints for tests.events. (#14904)Patrick Cloke2023-01-253-61/+85
|
* Fix type hints in knocking tests. (#14887)Andrew Morgan2023-01-252-6/+4
|
* Faster joins: omit partial rooms from eager syncs until the resync completes ↵David Robertson2023-01-233-8/+10
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | (#14870) * Allow `AbstractSet` in `StrCollection` Or else frozensets are excluded. This will be useful in an upcoming commit where I plan to change a function that accepts `List[str]` to accept `StrCollection` instead. * `rooms_to_exclude` -> `rooms_to_exclude_globally` I am about to make use of this exclusion mechanism to exclude rooms for a specific user and a specific sync. This rename helps to clarify the distinction between the global config and the rooms to exclude for a specific sync. * Better function names for internal sync methods * Track a list of excluded rooms on SyncResultBuilder I plan to feed a list of partially stated rooms for this sync to ignore * Exclude partial state rooms during eager sync using the mechanism established in the previous commit * Track un-partial-state stream in sync tokens So that we can work out which rooms have become fully-stated during a given sync period. * Fix mutation of `@cached` return value This was fouling up a complement test added alongside this PR. Excluding a room would mean the set of forgotten rooms in the cache would be extended. This means that room could be erroneously considered forgotten in the future. Introduced in #12310, Synapse 1.57.0. I don't think this had any user-visible side effects (until now). * SyncResultBuilder: track rooms to force as newly joined Similar plan as before. We've omitted rooms from certain sync responses; now we establish the mechanism to reintroduce them into future syncs. * Read new field, to present rooms as newly joined * Force un-partial-stated rooms to be newly-joined for eager incremental syncs only, provided they're still fully stated * Notify user stream listeners to wake up long polling syncs * Changelog * Typo fix Co-authored-by: Sean Quah <8349537+squahtx@users.noreply.github.com> * Unnecessary list cast Co-authored-by: Sean Quah <8349537+squahtx@users.noreply.github.com> * Rephrase comment Co-authored-by: Sean Quah <8349537+squahtx@users.noreply.github.com> * Another comment Co-authored-by: Sean Quah <8349537+squahtx@users.noreply.github.com> * Fixup merge(?) * Poke notifier when receiving un-partial-stated msg over replication * Fixup merge whoops Thanks MV :) Co-authored-by: Mathieu Velen <mathieuv@matrix.org> Co-authored-by: Mathieu Velten <mathieuv@matrix.org> Co-authored-by: Sean Quah <8349537+squahtx@users.noreply.github.com>
* Skip processing stats for broken rooms. (#14873)Patrick Cloke2023-01-231-34/+54
| | | | | | | * Skip processing stats for broken rooms. * Newsfragment * Use a custom exception.
* Fix type hints for Monthly Active Users tests (#14889)Andrew Morgan2023-01-221-6/+7
|
* Add type hints to `TestRatelimiter` (#14885)Andrew Morgan2023-01-211-15/+51
|
* Always notify replication when a stream advances (#14877)Erik Johnston2023-01-203-16/+14
| | | This ensures that all other workers are told about stream updates in a timely manner, without having to remember to manually poke replication.
* Faster joins: Avoid starting duplicate partial state syncs (#14844)Sean Quah2023-01-201-1/+111
| | | | | | | | | | | | | | | | | | Currently, we will try to start a new partial state sync every time we perform a remote join, which is undesirable if there is already one running for a given room. We intend to perform remote joins whenever additional local users wish to join a partial state room, so let's ensure that we do not start more than one concurrent partial state sync for any given room. ------------------------------------------------------------------------ There is a race condition where the homeserver leaves a room and later rejoins while the partial state sync from the previous membership is still running. There is no guarantee that the previous partial state sync will process the latest join, so we restart it if needed. Signed-off-by: Sean Quah <seanq@matrix.org>
* Wait for streams to catch up when processing HTTP replication. (#14820)Erik Johnston2023-01-182-15/+14
| | | | This should hopefully mitigate a class of races where data gets out of sync due a HTTP replication request racing with the replication streams.
* Change default room version to 10. Implements MSC3904 (#14111)Catalan Lover2023-01-181-3/+9
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Change Documentation to have v10 as default room version * Change Default Room version to 10 * Add changelog entry for default room version swap * Add changelog entry for v10 default room version in docs * Clarify doc changelog entry Co-authored-by: David Robertson <david.m.robertson1@gmail.com> * Improve Documentation changes. Co-authored-by: David Robertson <david.m.robertson1@gmail.com> * Update Changelog entry to have correct format Co-authored-by: David Robertson <david.m.robertson1@gmail.com> * Update Spec Version to 1.5 * Only need 1 changelog. * Fix test. * Update "Changed in" line Co-authored-by: David Robertson <david.m.robertson1@gmail.com> Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com> Co-authored-by: Patrick Cloke <patrickc@matrix.org>
* Stabilise serving partial join responses (#14839)David Robertson2023-01-171-2/+1
| | | | | Serving partial join responses is no longer experimental. They will only be served under the stable identifier if the the undocumented config flag experimental.msc3706_enabled is set to true. Synapse continues to request a partial join only if the undocumented config flag experimental.faster_joins is set to true; this setting remains present and unaffected.
* Fix bug in `wait_for_stream_position` (#14856)Erik Johnston2023-01-171-0/+78
| | | | | We were incorrectly checking if the *local* token had been advanced, rather than the token for the remote instance. In practice, I don't think this has caused any bugs due to where we use `wait_for_stream_position`, as critically we don't use it on instances that also write to the given streams (and so the local token will lag behind all remote tokens).
* Remove unnecessary reactor reference from `_PerHostRatelimiter` (#14842)Sean Quah2023-01-161-4/+4
| | | | | Fix up #14812 to avoid introducing a reference to the reactor. Signed-off-by: Sean Quah <seanq@matrix.org>
generated by cgit-pink 1.4.1 (git 2.36.1) at 2024-11-08 09:42:07 +0000