summary refs log tree commit diff
path: root/tests/handlers (follow)
Commit message (Collapse)AuthorAgeFilesLines
* Fix bug where typing replication breaks (#17252)Erik Johnston2024-05-311-1/+52
| | | | This can happen on restarts of the service, due to old rooms being pruned.
* Fix deduplicating of membership events to not create unused state groups. ↵Erik Johnston2024-05-301-0/+21
| | | | | | | | | | | | | (#17164) We try and deduplicate in two places: 1) really early on, and 2) just before we persist the event. The first case was broken due to it occuring before the profile information was added, and so it thought the event contents were different. The second case did catch it and handle it correctly, however doing so creates a redundant state group leading to bloat. Fixes #3791
* Add a short sleep if the request is rate-limited (#17210)Erik Johnston2024-05-182-0/+5
| | | This helps prevent clients from "tight-looping" retrying their request.
* Removed `request_key` from the `SyncConfig` (moved outside as its own ↵Eric Eastwood2024-05-161-8/+39
| | | | | | | | | function parameter) (#17201) Removed `request_key` from the `SyncConfig` (moved outside as its own function parameter) so it doesn't have to flow into `_generate_sync_entry_for_xxx` methods. This way we can separate the concerns of caching from generating the response and reuse the `_generate_sync_entry_for_xxx` functions as we see fit. Plus caching doesn't really have anything to do with the config of sync. Split from https://github.com/element-hq/synapse/pull/17167 Spawning from https://github.com/element-hq/synapse/pull/17167#discussion_r1601497279
* Fix bug where push rules would be empty in `/sync` (#17142)Erik Johnston2024-05-161-1/+28
| | | | | | Fixes #16987 Some old accounts seem to have an entry in global account data table for push rules, which we should ignore
* Refactor Sync handler to be able to return different sync responses ↵Eric Eastwood2024-05-161-16/+65
| | | | | | | | | | | | | | (`SyncVersion`) (#17200) Refactor Sync handler to be able to be able to return different sync responses (`SyncVersion`). Preparation to be able support sync v2 and a new Sliding Sync `/sync/e2ee` endpoint which returns a subset of sync v2. Split upon request: https://github.com/element-hq/synapse/pull/17167#discussion_r1601497279 Split from https://github.com/element-hq/synapse/pull/17167 where we will add `SyncVersion.E2EE_SYNC` and a new type of sync response.
* Redact membership events if the user requested erasure upon deactivating ↵Till2024-04-251-0/+37
| | | | | (#17076) Fixes #15355 by redacting all membership events before leaving rooms.
* bugfix: make msc3967 idempotent (#16943)Kegan Dougal2024-04-151-0/+50
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | MSC3967 was updated recently to make it more robust to network failures: > there is an existing cross-signing master key and it exactly matches the cross-signing master key provided in the request body. If there are any additional keys provided in the request (self signing key, user signing key) they MUST also match the existing keys stored on the server. In other words, the request contains no new keys. If there are new keys, UIA MUST be performed. https://github.com/matrix-org/matrix-spec-proposals/blob/hughns/device-signing-upload-uia/proposals/3967-device-signing-upload-uia.md#proposal This covers the case where the 200 OK is lost in transit so the client retries the upload, only to then get UIA'd. Complement tests: https://github.com/matrix-org/complement/pull/713 - passing example https://github.com/element-hq/synapse/actions/runs/7976948122/job/21778795094?pr=16943#step:7:8820 ### Pull Request Checklist <!-- Please read https://element-hq.github.io/synapse/latest/development/contributing_guide.html before submitting your pull request --> * [x] Pull request is based on the develop branch * [x] Pull request includes a [changelog file](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#changelog). The entry should: - Be a short description of your change which makes sense to users. "Fixed a bug that prevented receiving messages from other servers." instead of "Moved X method from `EventStore` to `EventWorkerStore`.". - Use markdown where necessary, mostly for `code blocks`. - End with either a period (.) or an exclamation mark (!). - Start with a capital letter. - Feel free to credit yourself, by adding a sentence "Contributed by @github_username." or "Contributed by [Your Name]." to the end of the entry. * [x] [Code style](https://element-hq.github.io/synapse/latest/code_style.html) is correct (run the [linters](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#run-the-linters)) --------- Co-authored-by: reivilibre <oliverw@matrix.org>
* Also check if first event matches the last in prev batch (#17066)Erik Johnston2024-04-091-0/+95
| | | | | Refinement of #17064 cc @richvdh
* Fix bug in calculating state for non-gappy syncs (#16942)Richard van der Hoff2024-04-041-0/+105
| | | | | | | | | Unfortunately, the optimisation we applied here for non-gappy syncs is not actually valid. Fixes https://github.com/element-hq/synapse/issues/16941. ~~Based on https://github.com/element-hq/synapse/pull/16930.~~ Requires https://github.com/matrix-org/sytest/pull/1374.
* `/sync`: fix bug in calculating `state` response (#16930)Richard van der Hoff2024-04-041-0/+80
| | | | | | | Fix a long-standing issue which could cause state to be omitted from the sync response if the last event was filtered out. Fixes: https://github.com/element-hq/synapse/issues/16928
* Fix bug in `/sync` response for archived rooms (#16932)Richard van der Hoff2024-04-041-13/+195
| | | | | | | | | | | | This PR fixes a very, very niche edge-case, but I've got some more work coming which will otherwise make the problem worse. The bug happens when the syncing user leaves a room, and has a sync filter which includes "left" rooms, but sets the timeline limit to 0. In that case, the state returned in the `state` section is calculated incorrectly. The fix is to pass a token corresponding to the point that the user leaves the room through to `compute_state_delta`.
* Fix reject knocks on deactivating account (#17010)Hanadi2024-03-211-2/+85
|
* Improve event validation (#16908)Shay2024-03-192-2/+153
| | | As the title states.
* Improve lock performance when a lot of locks are waiting (#16840)Mathieu Velten2024-03-141-0/+23
| | | | | | | | | | | | When a lot of locks are waiting for a single lock, notifying all locks independently with `call_later` on each release is really costly and incurs some kind of async contention, where the CPU is spinning a lot for not much. The included test is taking around 30s before the change, and 0.5s after. It was found following failing tests with https://github.com/element-hq/synapse/pull/16827.
* Correctly mention previous copyright (#16820)Erik Johnston2024-01-2324-0/+28
| | | | | During the migration the automated script to update the copyright headers accidentally got rid of some of the existing copyright lines. Reinstate them.
* Allow room creation but not publishing to continue if room publication rules ↵Shay2024-01-221-28/+23
| | | | | | | | | | | | | are violated when creating a new room. (#16811) Prior to this PR, if a request to create a public (public as in published to the rooms directory) room violated the room list publication rules set in the [config](https://matrix-org.github.io/synapse/latest/usage/configuration/config_documentation.html#room_list_publication_rules), the request to create the room was denied and the room was not created. This PR changes the behavior such that when a request to create a room published to the directory violates room list publication rules, the room is still created but the room is not published to the directory.
* Filter out rooms from the room directory being served to other homeservers ↵reivilibre2024-01-081-0/+88
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | when those rooms block that homeserver by their Access Control Lists. (#16759) The idea here being that the directory server shouldn't advertise rooms to a requesting server is the requesting server would not be allowed to join or participate in the room. <!-- Fixes: # <!-- --> <!-- Supersedes: # <!-- --> <!-- Follows: # <!-- --> <!-- Part of: # <!-- --> Base: `develop` <!-- git-stack-base-branch:develop --> <!-- This pull request is commit-by-commit review friendly. <!-- --> <!-- This pull request is intended for commit-by-commit review. <!-- --> Original commit schedule, with full messages: <ol> <li> Pass `from_federation_origin` down into room list retrieval code </li> <li> Don't cache /publicRooms response for inbound federated requests </li> <li> fixup! Don't cache /publicRooms response for inbound federated requests </li> <li> Cap the number of /publicRooms entries to 100 </li> <li> Simplify code now that you can't request unlimited rooms </li> <li> Filter out rooms from federated requests that don't have the correct ACL </li> <li> Request a handful more when filtering ACLs so that we can try to avoid shortchanging the requester </li> </ol> --------- Signed-off-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org>
* Merge remote-tracking branch 'gitlab/clokep/license-license' into new_developErik Johnston2023-12-1328-286/+449
|\
| * Update license headersPatrick Cloke2023-11-2128-286/+449
| |
* | Keep track of `user_ips` and `monthly_active_users` when delegating auth ↵David Robertson2023-11-231-14/+115
|/ | | | | | | | | | | | | (#16672) * Describe `insert_client_ip` * Pull out client_ips and MAU tracking to BaseAuth * Define HAS_AUTHLIB once in tests sick of copypasting * Track ips and token usage when delegating auth * Test that we track MAU and user_ips * Don't track `__oidc_admin`
* Add an Admin API to temporarily grant the ability to update an existing ↵David Robertson2023-11-151-0/+47
| | | | cross-signing key without UIA (#16634)
* Use full GitHub links instead of bare issue numbers. (#16637)Patrick Cloke2023-11-151-3/+3
|
* Convert simple_select_one_txn and simple_select_one to return tuples. (#16612)Patrick Cloke2023-11-092-3/+5
|
* Remove remaining usage of cursor_to_dict. (#16564)Patrick Cloke2023-10-311-11/+11
|
* Claim local one-time-keys in bulk (#16565)David Robertson2023-10-301-0/+158
| | | | Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
* Claim fallback keys in bulk (#16570)David Robertson2023-10-301-0/+77
|
* Add a new module API to update user presence state. (#16544)Patrick Cloke2023-10-261-9/+102
| | | | | | | | | | This adds a module API which allows a module to update a user's presence state/status message. This is useful for controlling presence from an external system. To fully control presence from the module the presence.enabled config parameter gains a new state of "untracked" which disables internal tracking of presence changes via user actions, etc. Only updates from the module will be persisted and sent down sync properly).
* Convert simple_select_list and simple_select_list_txn to return lists of ↵Patrick Cloke2023-10-261-5/+9
| | | | | tuples (#16505) This should use fewer allocations and improves type hints.
* Allow multiple workers to write to receipts stream. (#16432)Erik Johnston2023-10-251-4/+13
| | | Fixes #16417
* Fix type hint errors from Twisted trunk (#16526)Patrick Cloke2023-10-231-0/+1
|
* Convert state delta processing from a dict to attrs. (#16469)Patrick Cloke2023-10-161-1/+1
| | | For improved type checking & memory usage.
* Factor out `MultiWriter` token from `RoomStreamToken` (#16427)Erik Johnston2023-10-051-4/+4
|
* Some refactors around receipts stream (#16426)Erik Johnston2023-10-042-11/+23
|
* Return an immutable value from get_latest_event_ids_in_room. (#16326)Patrick Cloke2023-09-181-1/+1
|
* Fix using dehydrated devices (MSC2697) & refresh tokens (#16288)Hanadi2023-09-131-1/+9
| | | | | Refresh tokens were not correctly moved to the rehydrated device (similar to how the access token is currently handled). This resulted in invalid refresh tokens after rehydration.
* Update ruff config (#16283)Patrick Cloke2023-09-082-2/+2
| | | Enable additional checks & clean-up unneeded configuration.
* Handle "registration_enabled" parameter for CAS (#16262)Aurélien Grimpard2023-09-061-0/+17
| | | | Similar to OIDC, CAS providers can now disable registration such that only existing users are able to login via SSO.
* Merge remote-tracking branch 'origin/release-v1.92' into developPatrick Cloke2023-09-061-129/+23
|\
| * Merge remote-tracking branch 'origin/release-v1.91' into release-v1.92Patrick Cloke2023-09-061-129/+23
| |\
| | * Revert MSC3861 introspection cache, admin impersonation and account lock ↵Quentin Gliech2023-09-061-130/+24
| | | | | | | | | | | | (#16258)
* | | Delete device messages asynchronously and in staged batches (#16240)Mathieu Velten2023-09-061-0/+47
| | |
* | | Fix appservices being unable to handle to_device messages for multiple users ↵Will Hunt2023-09-051-0/+125
| | | | | | | | | | | | (#16251)
* | | Time out busy presence status & test multi-device busy (#16174)Patrick Cloke2023-09-051-3/+101
| | | | | | | | | | | | | | | | | | | | | Add a (long) timeout to when a "busy" device is considered not online. This does *not* match MSC3026, but is a reasonable thing for an implementation to do. Expands tests for the (unstable) busy presence with multiple devices.
* | | Track presence state per-device and combine to a user state. (#16066)Patrick Cloke2023-09-051-10/+490
|/ / | | | | | | | | | | | | | | | | | | | | Tracks presence on an individual per-device basis and combine the per-device state into a per-user state. This should help in situations where a user has multiple devices with conflicting status (e.g. one is syncing with unavailable and one is syncing with online). The tie-breaking is done by priority: BUSY > ONLINE > UNAVAILABLE > OFFLINE
* | Don't wake up destination transaction queue if they're not due for retry. ↵Erik Johnston2023-09-042-14/+48
| | | | | | | | (#16223)
* | Don't reset retry timers on "valid" error codes (#16221)Erik Johnston2023-09-041-2/+2
| |
* | Stop writing to the event_txn_id table (#16175)Patrick Cloke2023-08-301-11/+4
| |
* | Bump mypy-zope & mypy. (#16188)Patrick Cloke2023-08-299-39/+39
| |
* | Support IPv6-only SMTP servers (#16155)Evilham2023-08-291-10/+59
| | | | | | | | Use Twisted HostnameEndpoint to connect to SMTP servers (instead of connectTCP/connectSSL) which properly supports IPv6-only servers.
* | Pass the device ID around in the presence handler (#16171)Patrick Cloke2023-08-281-11/+27
| | | | | | | | | | | | Refactoring to pass the device ID (in addition to the user ID) through the presence handler (specifically the `user_syncing`, `set_state`, and `bump_presence_active_time` methods and their replication versions).
* | Combine logic about not overriding BUSY presence. (#16170)Patrick Cloke2023-08-281-7/+30
| | | | | | | | | | | | | | | | | | | | | | | | | | Simplify some of the presence code by reducing duplicated code between worker & non-worker modes. The main change is to push some of the logic from `user_syncing` into `set_state`. This is done by passing whether the user is setting the presence via a `/sync` with a new `is_sync` flag to `set_state`. If this is `true` some additional logic is performed: * Don't override `busy` presence. * Update the `last_user_sync_ts`. * Never update the status message.
* | Prepare unit tests for Python 3.12 (#16099)V024602023-08-251-12/+12
| |
* | Replace simple_async_mock with AsyncMock (#16180)Patrick Cloke2023-08-255-41/+39
| | | | | | | | Python 3.8 has a native AsyncMock, use it instead of a custom implementation.
* | Replace make_awaitable with AsyncMock (#16179)Patrick Cloke2023-08-2414-344/+300
|/ | | | Python 3.8 provides a native AsyncMock, we can replace the homegrown version we have.
* Add tests for restoring the presence state after a restart. (#16151)Patrick Cloke2023-08-231-0/+116
|
* Clean-up presence tests (#16158)Patrick Cloke2023-08-221-92/+37
| | | Reduce duplicated code & remove unused variables.
* Add an admin endpoint to allow authorizing server to signal token ↵Shay2023-08-221-1/+33
| | | | revocations (#16125)
* Fix user directory test for deactivated support user. (#16157)Patrick Cloke2023-08-221-8/+11
| | | | Support users should not be added to the user directory after being deactivated.
* MSC3861: allow impersonation by an admin using a query param (#16132)Mathieu Velten2023-08-181-0/+35
|
* Cache token introspection response from OIDC provider (#16117)Shay2023-08-171-0/+62
|
* Support MSC3814: Dehydrated Devices Part 2 (#16010)Shay2023-08-081-4/+5
|
* Add ability to wait for locks and add locks to purge history / room deletion ↵Erik Johnston2023-07-311-0/+74
| | | | | (#15791) c.f. #13476
* Fix 404 on /profile when the display name is empty but not the avatar (#16012)Mathieu Velten2023-07-271-0/+10
|
* Support MSC3814: Dehydrated Devices (#15929)Shay2023-07-241-1/+98
| | | | | | Signed-off-by: Nicolas Werner <n.werner@famedly.com> Co-authored-by: Nicolas Werner <n.werner@famedly.com> Co-authored-by: Nicolas Werner <89468146+nico-famedly@users.noreply.github.com> Co-authored-by: Hubert Chathi <hubert@uhoreg.ca>
* Re-introduce the outbound federation proxy (#15913)Eric Eastwood2023-07-184-4/+12
| | | | | Allow configuring the set of workers to proxy outbound federation traffic through (`outbound_federation_restricted_to`). This is useful when you have a worker setup with `federation_sender` instances responsible for sending outbound federation requests and want to make sure *all* outbound federation traffic goes through those instances. Before this change, the generic workers would still contact federation themselves for things like profile lookups, backfill, etc. This PR allows you to set more strict access controls/firewall for all workers and only allow the `federation_sender`'s to contact the outside world.
* Mark get_user_in_directory private since only used in tests (#15884)Mathieu Velten2023-07-121-9/+9
|
* Add + as an allowed character for Matrix IDs (MSC4009) (#15911)Patrick Cloke2023-07-111-6/+5
|
* Revert "Federation outbound proxy" (#15910)Eric Eastwood2023-07-104-12/+4
| | | | | | Revert "Federation outbound proxy (#15773)" This reverts commit b07b14b494ae1dd564b4c44f844c9a9545b3d08a.
* Federation outbound proxy (#15773)Eric Eastwood2023-07-054-4/+12
| | | | | | | Allow configuring the set of workers to proxy outbound federation traffic through (`outbound_federation_restricted_to`). This is useful when you have a worker setup with `federation_sender` instances responsible for sending outbound federation requests and want to make sure *all* outbound federation traffic goes through those instances. Before this change, the generic workers would still contact federation themselves for things like profile lookups, backfill, etc. This PR allows you to set more strict access controls/firewall for all workers and only allow the `federation_sender`'s to contact the outside world. The original code is from @erikjohnston's branches which I've gotten in-shape to merge.
* Fix forgotten rooms missing in initial sync (#15815)Nicolas Werner2023-06-211-0/+21
| | | | | | | | If you leave a room and forget it, then rejoin it, the room would be missing from the next initial sync. fixes #13262 Signed-off-by: Nicolas Werner <n.werner@famedly.com>
* Clear event caches when we purge history (#15609)Erik Johnston2023-06-081-1/+1
| | | | | | | This should help a little with #13476 --------- Co-authored-by: Patrick Cloke <patrickc@matrix.org>
* `N + 3`: Read from column `full_user_id` rather than `user_id` of tables ↵Shay2023-06-021-20/+8
| | | | `profiles` and `user_filters` (#15649)
* Enforce that an admin token also has the basic Matrix API scopeQuentin Gliech2023-05-301-1/+25
|
* Reject tokens with multiple device scopesQuentin Gliech2023-05-301-1/+28
|
* Handle errors when introspecting tokensQuentin Gliech2023-05-301-0/+35
| | | | | This returns a proper 503 when the introspection endpoint is not working for some reason, which should avoid logging out clients in those cases.
* Disable incompatible Admin API endpointsQuentin Gliech2023-05-301-0/+19
|
* Make AS tokens work & allow ASes to /registerQuentin Gliech2023-05-301-2/+2
|
* Refactor config to be an experimental featureHugh Nimmo-Smith2023-05-301-6/+9
| | | | Also enforce you can't combine it with incompatible config options
* Disable account related endpoints when using OAuth delegationQuentin Gliech2023-05-301-3/+177
|
* Actually enforce guest + return www-authenticate headerHugh Nimmo-Smith2023-05-301-3/+40
|
* Initial tests for OAuth delegationHugh Nimmo-Smith2023-05-301-0/+345
|
* Make the api.auth.Auth a ProtocolQuentin Gliech2023-05-301-2/+2
|
* Process previously failed backfill events in the background (#15585)Eric Eastwood2023-05-241-0/+95
| | | | | | | | | Process previously failed backfill events in the background because they are bound to fail again and we don't need to waste time holding up the request for something that is bound to fail again. Fix https://github.com/matrix-org/synapse/issues/13623 Follow-up to https://github.com/matrix-org/synapse/issues/13621 and https://github.com/matrix-org/synapse/issues/13622 Part of making `/messages` faster: https://github.com/matrix-org/synapse/issues/13356
* Add requesting user id parameter to key claim methods in ↵Shay2023-05-241-1/+15
| | | | `TransportLayerClient` (#15663)
* Consolidate logic to check for deactivated users. (#15634)Patrick Cloke2023-05-231-38/+21
| | | | | | | This moves the deactivated user check to the method which all login types call. Additionally updates the application service tests to be more realistic by removing invalid tests and fixing server names.
* Rename blacklist/whitelist internally. (#15620)Patrick Cloke2023-05-191-1/+1
| | | | Avoid renaming configuration settings for now and rename internal code to use blocklist and allowlist instead.
* Check appservices for devices during a /user/devices query. (#15539)Patrick Cloke2023-05-051-1/+134
| | | | | | | | MSC3984 proxies /keys/query requests to appservices, but servers will can also requests devices / keys from the /user/devices endpoint. The formats are close enough that we can "proxy" that /user/devices to appservices (by calling /keys/query) and then change the format of the returned data before returning it over federation.
* Implement MSC4009 to widen the allowed Matrix ID grammar (#15536)Patrick Cloke2023-05-051-0/+13
| | | | | Behind a configuration flag this adds + to the list of allowed characters in Matrix IDs. The main feature this enables is using full E.164 phone numbers as Matrix IDs.
* Add config option to forget rooms automatically when users leave them (#15224)Sean Quah2023-05-031-0/+11
| | | | | This is largely based off the stats and user directory updater code. Signed-off-by: Sean Quah <seanq@matrix.org>
* Add support for claiming multiple OTKs at once. (#15468)Patrick Cloke2023-04-271-18/+14
| | | | | | | MSC3983 provides a way to request multiple OTKs at once from appservices, this extends this concept to the Client-Server API. Note that this will likely be spit out into a separate MSC, but is currently part of MSC3983.
* Add column `full_user_id` to tables `profiles` and `user_filters`. (#15458)Shay2023-04-261-14/+12
|
* Add unstable /keys/claim endpoint which always returns fallback keys. (#15462)Patrick Cloke2023-04-251-6/+235
| | | | | | | | | | | | | It can be useful to always return the fallback key when attempting to claim keys. This adds an unstable endpoint for `/keys/claim` which always returns fallback keys in addition to one-time-keys. The fallback key(s) are not marked as "used" unless there are no corresponding OTKs. This is currently defined in MSC3983 (although likely to be split out to a separate MSC). The endpoint shape may change or be requested differently (i.e. a keyword parameter on the current endpoint), but the core logic should be reasonable.
* Move Spam Checker callbacks to a dedicated file (#15453)Andrew Morgan2023-04-181-1/+1
|
* Convert async to normal tests in `TestSSOHandler` (#15433)Dirk Klimpel2023-04-141-4/+4
| | | | | * Convert async to normal tests in `TestSSOHandler` * newsfile
* Revert pruning of old devices (#15360)Erik Johnston2023-03-312-2/+2
| | | | | | | | | * Revert "Fix registering a device on an account with lots of devices (#15348)" This reverts commit f0d8f66eaaacfa75bed65bc5d0c602fbc5339c85. * Revert "Delete stale non-e2e devices for users, take 3 (#15183)" This reverts commit 78cdb72cd6b0e007c314d9fed9f629dfc5b937a6.
* Implement MSC3984 to proxy /keys/query requests to appservices. (#15321)Patrick Cloke2023-03-301-1/+120
| | | | | If enabled, for users which are exclusively owned by an application service then the appservice will be queried for devices in addition to any information stored in the Synapse database.
* Add the ability to enable/disable registrations when in the OIDC flow (#14978)Warren Bailey2023-03-301-1/+16
| | | Signed-off-by: Warren Bailey <warren@warrenbailey.net>
* Delete stale non-e2e devices for users, take 3 (#15183)Erik Johnston2023-03-292-2/+2
| | | | | This should help reduce the number of devices e.g. simple bots the repeatedly login rack up. We only delete non-e2e devices as they should be safe to delete, whereas if we delete e2e devices for a user we may accidentally break their ability to receive e2e keys for a message.
* Implement MSC3983 to proxy /keys/claim queries to appservices. (#15314)Patrick Cloke2023-03-281-1/+75
| | | | | | Experimental support for MSC3983 is behind a configuration flag. If enabled, for users which are exclusively owned by an application service then the appservice will be queried for one-time keys *if* there are none uploaded to Synapse.
* Refresh remote profiles that have been marked as stale, in order to fill the ↵reivilibre2023-03-161-2/+185
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | user directory. [rei:userdirpriv] (#14756) * Scaffolding for background process to refresh profiles * Add scaffolding for background process to refresh profiles for a given server * Implement the code to select servers to refresh from * Ensure we don't build up multiple looping calls * Make `get_profile` able to respect backoffs * Add logic for refreshing users * When backing off, schedule a refresh when the backoff is over * Wake up the background processes when we receive an interesting state event * Add tests * Newsfile Signed-off-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org> * Add comment about 1<<62 --------- Signed-off-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org>
* Batch up storing state groups when creating new room (#14918)Shay2023-02-242-9/+19
|
* Add information on uploaded media to user export command. (#15107)Dirk Klimpel2023-02-231-0/+29
|
* Bump black from 22.12.0 to 23.1.0 (#15103)dependabot[bot]2023-02-222-2/+0
|
* Fix a bug introduced in Synapse v1.74.0 where searching with colons when ↵reivilibre2023-02-201-0/+7
| | | | | | using ICU for search term tokenisation would fail with an error. (#15079) Co-authored-by: David Robertson <davidr@element.io>
* Add account data to export command (#14969)Dirk Klimpel2023-02-171-0/+27
| | | | | | | | | | | * Add account data to to export command * newsfile * remove not needed function * update newsfile * adopt #14973
* Use mypy 1.0 (#15052)David Robertson2023-02-161-2/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Update mypy and mypy-zope * Remove unused ignores These used to suppress ``` synapse/storage/engines/__init__.py:28: error: "__new__" must return a class instance (got "NoReturn") [misc] ``` and ``` synapse/http/matrixfederationclient.py:1270: error: "BaseException" has no attribute "reasons" [attr-defined] ``` (note that we check `hasattr(e, "reasons")` above) * Avoid empty body warnings, sometimes by marking methods as abstract E.g. ``` tests/handlers/test_register.py:58: error: Missing return statement [empty-body] tests/handlers/test_register.py:108: error: Missing return statement [empty-body] ``` * Suppress false positive about `JaegerConfig` Complaint was ``` synapse/logging/opentracing.py:450: error: Function "Type[Config]" could always be true in boolean context [truthy-function] ``` * Fix not calling `is_state()` Oops! ``` tests/rest/client/test_third_party_rules.py:428: error: Function "Callable[[], bool]" could always be true in boolean context [truthy-function] ``` * Suppress false positives from ParamSpecs ```` synapse/logging/opentracing.py:971: error: Argument 2 to "_custom_sync_async_decorator" has incompatible type "Callable[[Arg(Callable[P, R], 'func'), **P], _GeneratorContextManager[None]]"; expected "Callable[[Callable[P, R], **P], _GeneratorContextManager[None]]" [arg-type] synapse/logging/opentracing.py:1017: error: Argument 2 to "_custom_sync_async_decorator" has incompatible type "Callable[[Arg(Callable[P, R], 'func'), **P], _GeneratorContextManager[None]]"; expected "Callable[[Callable[P, R], **P], _GeneratorContextManager[None]]" [arg-type] ```` * Drive-by improvement to `wrapping_logic` annotation * Workaround false "unreachable" positives See https://github.com/Shoobx/mypy-zope/issues/91 ``` tests/http/test_proxyagent.py:626: error: Statement is unreachable [unreachable] tests/http/test_proxyagent.py:762: error: Statement is unreachable [unreachable] tests/http/test_proxyagent.py:826: error: Statement is unreachable [unreachable] tests/http/test_proxyagent.py:838: error: Statement is unreachable [unreachable] tests/http/test_proxyagent.py:845: error: Statement is unreachable [unreachable] tests/http/federation/test_matrix_federation_agent.py:151: error: Statement is unreachable [unreachable] tests/http/federation/test_matrix_federation_agent.py:452: error: Statement is unreachable [unreachable] tests/logging/test_remote_handler.py:60: error: Statement is unreachable [unreachable] tests/logging/test_remote_handler.py:93: error: Statement is unreachable [unreachable] tests/logging/test_remote_handler.py:127: error: Statement is unreachable [unreachable] tests/logging/test_remote_handler.py:152: error: Statement is unreachable [unreachable] ``` * Changelog * Tweak DBAPI2 Protocol to be accepted by mypy 1.0 Some extra context in: - https://github.com/matrix-org/python-canonicaljson/pull/57 - https://github.com/python/mypy/issues/6002 - https://mypy.readthedocs.io/en/latest/common_issues.html#covariant-subtyping-of-mutable-protocol-members-is-rejected * Pull in updated canonicaljson lib so the protocol check just works * Improve comments in opentracing I tried to workaround the ignores but found it too much trouble. I think the corresponding issue is https://github.com/python/mypy/issues/12909. The mypy repo has a PR claiming to fix this (https://github.com/python/mypy/pull/14677) which might mean this gets resolved soon? * Better annotation for INTERACTIVE_AUTH_CHECKERS * Drive-by AUTH_TYPE annotation, to remove an ignore
* Add final type hint to tests.unittest. (#15072)Patrick Cloke2023-02-1411-78/+93
| | | | Adds a return type to HomeServerTestCase.make_homeserver and deal with any variables which are no longer Any.
* Faster joins: don't stall when a user joins during a fast join (#14606)Mathieu Velten2023-02-101-20/+20
| | | | | | | | | | | | | | | | Fixes #12801. Complement tests are at https://github.com/matrix-org/complement/pull/567. Avoid blocking on full state when handling a subsequent join into a partial state room. Also always perform a remote join into partial state rooms, since we do not know whether the joining user has been banned and want to avoid leaking history to banned users. Signed-off-by: Mathieu Velten <mathieuv@matrix.org> Co-authored-by: Sean Quah <seanq@matrix.org> Co-authored-by: David Robertson <davidr@element.io>
* Add a class UnpersistedEventContext to allow for the batching up of storing ↵Shay2023-02-091-1/+3
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | state groups (#14675) * add class UnpersistedEventContext * modify create new client event to create unpersistedeventcontexts * persist event contexts after creation * fix tests to persist unpersisted event contexts * cleanup * misc lints + cleanup * changelog + fix comments * lints * fix batch insertion? * reduce redundant calculation * add unpersisted event classes * rework compute_event_context, split into function that returns unpersisted event context and then persists it * use calculate_context_info to create unpersisted event contexts * update typing * $%#^&* * black * fix comments and consolidate classes, use attr.s for class * requested changes * lint * requested changes * requested changes * refactor to be stupidly explicit * clearer renaming and flow * make partial state non-optional * update docstrings --------- Co-authored-by: Erik Johnston <erik@matrix.org>
* Add more missing type hints to tests. (#15028)Patrick Cloke2023-02-081-2/+2
|
* Faster joins: Refactor handling of servers in room (#14954)Sean Quah2023-02-032-2/+2
| | | | | | | | | | | | | | Ensure that the list of servers in a partial state room always contains the server we joined off. Also refactor `get_partial_state_servers_at_join` to return `None` when the given room is no longer partial stated, to explicitly indicate when the room has partial state. Otherwise it's not clear whether an empty list means that the room has full state, or the room is partial stated, but the server we joined off told us that there are no servers in the room. Signed-off-by: Sean Quah <seanq@matrix.org>
* Add more user information to export-data command. (#14894)Dirk Klimpel2023-02-011-0/+60
| | | | | | * The user's profile information. * The user's devices. * The user's connections / IP address information.
* Fix type hints in typing edu unit tests (#14886)Andrew Morgan2023-01-261-40/+59
|
* Faster joins: Avoid starting duplicate partial state syncs (#14844)Sean Quah2023-01-201-1/+111
| | | | | | | | | | | | | | | | | | Currently, we will try to start a new partial state sync every time we perform a remote join, which is undesirable if there is already one running for a given room. We intend to perform remote joins whenever additional local users wish to join a partial state room, so let's ensure that we do not start more than one concurrent partial state sync for any given room. ------------------------------------------------------------------------ There is a race condition where the homeserver leaves a room and later rejoins while the partial state sync from the previous membership is still running. There is no guarantee that the previous partial state sync will process the latest join, so we restart it if needed. Signed-off-by: Sean Quah <seanq@matrix.org>
* Support RFC7636 PKCE in the OAuth 2.0 flow. (#14750)Patrick Cloke2023-01-041-8/+144
| | | | | | | PKCE can protect against certain attacks and is enabled by default. Support can be controlled manually by setting the pkce_method of each oidc_providers entry to 'auto' (default), 'always', or 'never'. This is required by Twitter OAuth 2.0 support.
* Add missing type hints to tests.handlers. (#14680)Patrick Cloke2022-12-1619-373/+524
| | | And do not allow untyped defs in tests.handlers.
* Revert the deletion of stale devices due to performance issues. (#14662)reivilibre2022-12-121-32/+1
|
* Limit the number of devices we delete at once (#14649)Erik Johnston2022-12-091-0/+31
|
* Delete stale non-e2e devices for users, take 2 (#14595)Erik Johnston2022-12-091-1/+1
| | | | | This should help reduce the number of devices e.g. simple bots the repeatedly login rack up. We only delete non-e2e devices as they should be safe to delete, whereas if we delete e2e devices for a user we may accidentally break their ability to receive e2e keys for a message.
* Improve logging and opentracing for to-device message handling (#14598)Richard van der Hoff2022-12-061-1/+6
| | | | | | | A batch of changes intended to make it easier to trace to-device messages through the system. The intention here is that a client can set a property org.matrix.msgid in any to-device message it sends. That ID is then included in any tracing or logging related to the message. (Suggestions as to where this field should be documented welcome. I'm not enthusiastic about speccing it - it's very much an optional extra to help with debugging.) I've also generally improved the data we send to opentracing for these messages.
* Merge branch 'release-v1.73' into developDavid Robertson2022-12-011-1/+1
|\
| * Revert "POC delete stale non-e2e devices for users (#14038)" (#14582)David Robertson2022-11-291-1/+1
| |
* | Modernize unit tests configuration settings for workers. (#14568)realtyem2022-12-013-5/+11
|/ | | | Use the newer foo_instances configuration instead of the deprecated flags to enable specific features (e.g. start_pushers).
* POC delete stale non-e2e devices for users (#14038)Erik Johnston2022-11-291-1/+1
| | | | | | | | This should help reduce the number of devices e.g. simple bots the repeatedly login rack up. We only delete non-e2e devices as they should be safe to delete, whereas if we delete e2e devices for a user we may accidentally break their ability to receive e2e keys for a message. Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com> Co-authored-by: Sean Quah <8349537+squahtx@users.noreply.github.com>
* Use `device_one_time_keys_count` to match MSC3202 (#14565)Andrew Ferrazzutti2022-11-281-2/+2
| | | | | | | | | | | | | | | | | | | * Use `device_one_time_keys_count` to match MSC3202 Rename the `device_one_time_key_counts` key in responses to `device_one_time_keys_count` to match the name specified by MSC3202. Also change related variable/class names for consistency. Signed-off-by: Andrew Ferrazzutti <andrewf@element.io> * Update changelog.d/14565.misc * Revert name change for `one_time_key_counts` key as this is a different key altogether from `device_one_time_keys_count`, which is used for `/sync` instead of appservice transactions. Signed-off-by: Andrew Ferrazzutti <andrewf@element.io>
* Add support for handling avatar with SSO login (#13917)Ashish Kumar2022-11-251-0/+145
| | | | | | | | This commit adds support for handling a provided avatar picture URL when logging in via SSO. Signed-off-by: Ashish Kumar <ashfame@users.noreply.github.com> Fixes #9357.
* Add a type hint for `get_device_handler()` and fix incorrect types. (#14055)Patrick Cloke2022-11-221-6/+13
| | | | | This was the last untyped handler from the HomeServer object. Since it was being treated as Any (and thus unchecked) it was being used incorrectly in a few places.
* Fix an invalid comparison of `UserPresenceState` to `str` (#14393)Andrew Morgan2022-11-161-6/+35
|
* Check appservice user interest against the local users instead of all users ↵Eric Eastwood2022-10-271-5/+157
| | | | (`get_users_in_room` mis-use) (#13958)
* fix broken avatar checks when server_name contains a port (#13927)Ashish Kumar2022-10-261-0/+49
| | | | | | Fixes check_avatar_size_and_mime_type() to successfully update avatars on homeservers running on non-default ports which it would mistakenly treat as remote homeserver while validating the avatar's size and mime type. Signed-off-by: Ashish Kumar ashfame@users.noreply.github.com
* Save login tokens in database (#13844)Quentin Gliech2022-10-261-57/+78
| | | | | | | | | | | * Save login tokens in database Signed-off-by: Quentin Gliech <quenting@element.io> * Add upgrade notes * Track login token reuse in a Prometheus metric Signed-off-by: Quentin Gliech <quenting@element.io>
* Refactor OIDC tests to better mimic an actual OIDC provider. (#13910)Quentin Gliech2022-10-251-330/+250
| | | | | | | | | This implements a fake OIDC server, which intercepts calls to the HTTP client. Improves accuracy of tests by covering more internal methods. One particular example was the ID token validation, which previously mocked. This uncovered an incorrect dependency: Synapse actually requires at least authlib 0.15.1, not 0.14.0.
* Avoid checking the event cache when backfilling events (#14164)Andrew Morgan2022-10-181-1/+104
|
* Stop getting missing `prev_events` after we already know their signature is ↵Eric Eastwood2022-10-151-2/+199
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | invalid (#13816) While https://github.com/matrix-org/synapse/pull/13635 stops us from doing the slow thing after we've already done it once, this PR stops us from doing one of the slow things in the first place. Related to - https://github.com/matrix-org/synapse/issues/13622 - https://github.com/matrix-org/synapse/pull/13635 - https://github.com/matrix-org/synapse/issues/13676 Part of https://github.com/matrix-org/synapse/issues/13356 Follow-up to https://github.com/matrix-org/synapse/pull/13815 which tracks event signature failures. With this PR, we avoid the call to the costly `_get_state_ids_after_missing_prev_event` because the signature failure will count as an attempt before and we filter events based on the backoff before calling `_get_state_ids_after_missing_prev_event` now. For example, this will save us 156s out of the 185s total that this `matrix.org` `/messages` request. If you want to see the full Jaeger trace of this, you can drag and drop this `trace.json` into your own Jaeger, https://gist.github.com/MadLittleMods/4b12d0d0afe88c2f65ffcc907306b761 To explain this exact scenario around `/messages` -> backfill, we call `/backfill` and first check the signatures of the 100 events. We see bad signature for `$luA4l7QHhf_jadH3mI-AyFqho0U2Q-IXXUbGSMq6h6M` and `$zuOn2Rd2vsC7SUia3Hp3r6JSkSFKcc5j3QTTqW_0jDw` (both member events). Then we process the 98 events remaining that have valid signatures but one of the events references `$luA4l7QHhf_jadH3mI-AyFqho0U2Q-IXXUbGSMq6h6M` as a `prev_event`. So we have to do the whole `_get_state_ids_after_missing_prev_event` rigmarole which pulls in those same events which fail again because the signatures are still invalid. - `backfill` - `outgoing-federation-request` `/backfill` - `_check_sigs_and_hash_and_fetch` - `_check_sigs_and_hash_and_fetch_one` for each event received over backfill - ❗ `$luA4l7QHhf_jadH3mI-AyFqho0U2Q-IXXUbGSMq6h6M` fails with `Signature on retrieved event was invalid.`: `unable to verify signature for sender domain xxx: 401: Failed to find any key to satisfy: _FetchKeyRequest(...)` - ❗ `$zuOn2Rd2vsC7SUia3Hp3r6JSkSFKcc5j3QTTqW_0jDw` fails with `Signature on retrieved event was invalid.`: `unable to verify signature for sender domain xxx: 401: Failed to find any key to satisfy: _FetchKeyRequest(...)` - `_process_pulled_events` - `_process_pulled_event` for each validated event - ❗ Event `$Q0iMdqtz3IJYfZQU2Xk2WjB5NDF8Gg8cFSYYyKQgKJ0` references `$luA4l7QHhf_jadH3mI-AyFqho0U2Q-IXXUbGSMq6h6M` as a `prev_event` which is missing so we try to get it - `_get_state_ids_after_missing_prev_event` - `outgoing-federation-request` `/state_ids` - ❗ `get_pdu` for `$luA4l7QHhf_jadH3mI-AyFqho0U2Q-IXXUbGSMq6h6M` which fails the signature check again - ❗ `get_pdu` for `$zuOn2Rd2vsC7SUia3Hp3r6JSkSFKcc5j3QTTqW_0jDw` which fails the signature check
* Fix a bug where redactions were not being sent over federation if we did not ↵Shay2022-10-111-7/+11
| | | | have the original event. (#13813)
* Optimise get_rooms_for_user (drop with_stream_ordering) (#13787)Nick Mills-Barrett2022-09-291-0/+1
|
* Persist CreateRoom events to DB in a batch (#13800)Shay2022-09-282-3/+11
|
* typing: check origin server of typing event against room's servers (#13830)Mathieu Velten2022-09-261-0/+4
| | | | | | | This is also using the partial state approximation if needed so we do not block here during a fast join. Signed-off-by: Mathieu Velten <mathieuv@matrix.org> Co-authored-by: Sean Quah <8349537+squahtx@users.noreply.github.com>
* Accept & store thread IDs for receipts (implement MSC3771). (#13782)Patrick Cloke2022-09-231-0/+1
| | | | Updates the `/receipts` endpoint and receipt EDU handler to parse a `thread_id` from the body and insert it in the database.
* Faster Remote Room Joins: tell remote homeservers that we are unable to ↵reivilibre2022-09-231-1/+1
| | | | authorise them if they query a room which has partial state on our server. (#13823)
* Port the push rule classes to Rust. (#13768)Erik Johnston2022-09-201-15/+12
|
* Avoid putting rejected events in room state (#13723)Sean Quah2022-09-161-0/+399
| | | Signed-off-by: Sean Quah <seanq@matrix.org>
* Keep track when we try and fail to process a pulled event (#13589)Eric Eastwood2022-09-141-0/+222
| | | | | | | | | | | | | | We can follow-up this PR with: 1. Only try to backfill from an event if we haven't tried recently -> https://github.com/matrix-org/synapse/issues/13622 1. When we decide to backfill that event again, process it in the background so it doesn't block and make `/messages` slow when we know it will probably fail again -> https://github.com/matrix-org/synapse/issues/13623 1. Generally track failures everywhere we try and fail to pull an event over federation -> https://github.com/matrix-org/synapse/issues/13700 Fix https://github.com/matrix-org/synapse/issues/13621 Part of https://github.com/matrix-org/synapse/issues/13356 Mentioned in [internal doc](https://docs.google.com/document/d/1lvUoVfYUiy6UaHB6Rb4HicjaJAU40-APue9Q4vzuW3c/edit#bookmark=id.qv7cj51sv9i5)
* Fix bug in device list caching when remote users leave rooms (#13749)Sean Quah2022-09-141-1/+7
| | | | | | | | | | | | When a remote user leaves the last room shared with the homeserver, we have to mark their device list as unsubscribed, otherwise we would hold on to a stale device list in our cache. Crucially, the device list would remain cached even after the remote user rejoined the room, which could lead to E2EE failures until the next change to the remote user's device list. Fixes #13651. Signed-off-by: Sean Quah <seanq@matrix.org>
* Remove support for unstable private read receipts (#13653)Šimon Brandner2022-09-011-35/+13
| | | Signed-off-by: Šimon Brandner <simon.bra.ag@gmail.com>
* Fix that user cannot `/forget` rooms after the last member has left (#13546)Dirk Klimpel2022-08-301-2/+91
|
* `synapse.api.auth.Auth` cleanup: make permission-related methods use ↵Quentin Gliech2022-08-221-3/+5
| | | | | | | | | `Requester` instead of the `UserID` (#13024) Part of #13019 This changes all the permission-related methods to rely on the Requester instead of the UserID. This is a first step towards enabling scoped access tokens at some point, since I expect the Requester to have scope-related informations in it. It also changes methods which figure out the user/device/appservice out of the access token to return a Requester instead of something else. This avoids having store-related objects in the methods signatures.
* Register homeserver modules when creating test homeserver (#13558)Andrew Morgan2022-08-192-18/+0
|
* Reduce the number of tests using TCP replication. (#13543)Patrick Cloke2022-08-191-2/+2
| | | | Uses Redis replication in additional test cases (instead of TCP replication). A small step towards dropping TCP replication.
* Fix validation problem that occurs when a user tries to deactivate their ↵reivilibre2022-08-191-0/+15
| | | | account or change their password. (#13563)
* Make push rules use proper structures. (#13522)Erik Johnston2022-08-161-16/+17
| | | | | | | | | | | | This improves load times for push rules: | Version | Time per user | Time for 1k users | | -------------------- | ------------- | ----------------- | | Before | 138 µs | 138ms | | Now (with custom) | 2.11 µs | 2.11ms | | Now (without custom) | 49.7 ns | 0.05 ms | This therefore has a large impact on send times for rooms with large numbers of local users in the room.
* Support stable identifiers for MSC2285: private read receipts. (#13273)Šimon Brandner2022-08-051-16/+42
| | | | | This adds support for the stable identifiers of MSC2285 while continuing to support the unstable identifiers behind the configuration flag. These will be removed in a future version.
* Use literals in place of `HTTPStatus` constants in tests (#13463)Dirk Klimpel2022-08-053-5/+3
|
* Faster Room Joins: don't leave a stuck room partial state flag if the join ↵reivilibre2022-08-011-1/+121
| | | | fails. (#13403)
* Refactor `_resolve_state_at_missing_prevs` to return an `EventContext` (#13404)Sean Quah2022-08-011-4/+11
| | | | | | | | Previously, `_resolve_state_at_missing_prevs` returned the resolved state before an event and a partial state flag. These were unwieldy to carry around would only ever be used to build an event context. Build the event context directly instead. Signed-off-by: Sean Quah <seanq@matrix.org>
* Add missing type hints for tests.unittest. (#13397)Patrick Cloke2022-07-271-10/+2
|
* Faster room joins: avoid blocking when pulling events with missing prevs ↵Sean Quah2022-07-261-0/+1
| | | | | | | | | (#13355) Avoid blocking on full state in `_resolve_state_at_missing_prevs` and return a new flag indicating whether the resolved state is partial. Thread that flag around so that it makes it into the event context. Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
* Support Implicit TLS for sending emails (#13317)Jan Schär2022-07-251-1/+56
| | | | | | | | | | Previously, TLS could only be used with STARTTLS. Add a new option `force_tls`, where TLS is used from the start. Implicit TLS is recommended over STARTLS, see https://datatracker.ietf.org/doc/html/rfc8314 Fixes #8046. Signed-off-by: Jan Schär <jan@jschaer.ch>
* Rate limit joins per-room (#13276)David Robertson2022-07-191-0/+290
|
* Use HTTPStatus constants in place of literals in tests. (#13297)Dirk Klimpel2022-07-151-20/+21
|
* Async get event cache prep (#13242)Nick Mills-Barrett2022-07-151-1/+1
| | | | | Some experimental prep work to enable external event caching based on #9379 & #12955. Doesn't actually move the cache at all, just lays the groundwork for async implemented caches. Signed off by Nick @ Beeper (@Fizzadar)
* Federation Sender & Appservice Pusher Stream Optimisations (#13251)Nick Mills-Barrett2022-07-151-8/+8
| | | | | | | | | | | | | * Replace `get_new_events_for_appservice` with `get_all_new_events_stream` The functions were near identical and this brings the AS worker closer to the way federation senders work which can allow for multiple workers to handle AS traffic. * Pull received TS alongside events when processing the stream This avoids an extra query -per event- when both federation sender and appservice pusher process events.
* Rip out auth-event reconciliation code (#12943)Richard van der Hoff2022-07-141-139/+1
| | | | | | | There is a corner in `_check_event_auth` (long known as "the weird corner") where, if we get an event with auth_events which don't match those we were expecting, we attempt to resolve the diffence between our state and the remote's with a state resolution. This isn't specced, and there's general agreement we shouldn't be doing it. However, it turns out that the faster-joins code was relying on it, so we need to introduce something similar (but rather simpler) for that.
* Rename test case method to `add_hashes_and_signatures_from_other_server` ↵David Robertson2022-07-122-4/+4
| | | | (#13255)
* Fix inconsistencies in event validation (#13088)Richard van der Hoff2022-06-172-5/+10
|
* Fix typechecks against twisted trunk (#13061)David Robertson2022-06-151-4/+4
|
* Move the "email unsubscribe" resource, refactor the macaroon generator & ↵Quentin Gliech2022-06-141-4/+3
| | | | | | | | | | | | | | | | | | | | | simplify the access token verification logic. (#12986) This simplifies the access token verification logic by removing the `rights` parameter which was only ever used for the unsubscribe link in email notifications. The latter has been moved under the `/_synapse` namespace, since it is not a standard API. This also makes the email verification link more secure, by embedding the app_id and pushkey in the macaroon and verifying it. This prevents the user from tampering the query parameters of that unsubscribe link. Macaroon generation is refactored: - Centralised all macaroon generation and verification logic to the `MacaroonGenerator` - Moved to `synapse.utils` - Changed the constructor to require only a `Clock`, hostname, and a secret key (instead of a full `Homeserver`). - Added tests for all methods.
* Decouple `synapse.api.auth_blocking.AuthBlocking` from ↵Quentin Gliech2022-06-143-3/+3
| | | | `synapse.api.auth.Auth`. (#13021)
* Replace noop background updates with DELETE. (#12954)Patrick Cloke2022-06-131-28/+0
| | | | Removes the `register_noop_background_update` and deletes the background updates directly in a delta file.
* Stop depending on `room_id` to be returned for children state in the ↵Patrick Cloke2022-06-101-1/+1
| | | | | | | | | | | hierarchy response. (#12991) The `room_id` field was removed from MSC2946 before it was accepted. It was initially kept for backwards compatibility and should be removed now that the stable form of the API is used. This change only stops Synapse from validating that it is returned, a future PR will remove returning it as part of the response.
* Consolidate the logic of delete_device/delete_devices. (#12970)Patrick Cloke2022-06-071-2/+2
| | | | | | | | By always using delete_devices and sometimes passing a list with a single device ID. Previously these methods had gotten out of sync with each other and it seems there's little benefit to the single-device variant.
* Reduce state pulled from DB due to sending typing and receipts over ↵Erik Johnston2022-06-061-2/+4
| | | | | federation (#12964) Reducing the amount of state we pull from the DB is useful as fetching state is expensive in terms of DB, CPU and memory.
* Reduce the amount of state we pull from the DB (#12811)Erik Johnston2022-06-061-1/+2
|
* Wait for lazy join to complete when getting current state (#12872)Erik Johnston2022-06-013-4/+8
|
* Remove remaining bits of groups code. (#12936)Patrick Cloke2022-06-012-4/+0
| | | | | | * Update worker docs to remove group endpoints. * Removes an unused parameter to `ApplicationService`. * Break dependency between media repo and groups. * Avoid copying `m.room.related_groups` state events during room upgrades.
* Rename storage classes (#12913)Erik Johnston2022-05-314-12/+19
|
* Additional constants for EDU types. (#12884)Patrick Cloke2022-05-273-27/+41
| | | Instead of hard-coding strings in many places.
* Remove unstable APIs for /hierarchy. (#12851)Patrick Cloke2022-05-261-10/+10
| | | | Removes the unstable endpoint as well as a duplicated field which was modified during stabilization.
* Pull out less state when handling gaps mk2 (#12852)Erik Johnston2022-05-261-1/+5
|
* Fix up `state_store` naming (#12871)Erik Johnston2022-05-251-2/+2
|
* Add some type hints to `event_federation` datastore (#12753)Dirk Klimpel2022-05-181-0/+1
| | | Co-authored-by: David Robertson <david.m.robertson1@gmail.com>
* Discard null-containing strings before updating the user directory (#12762)David Robertson2022-05-181-0/+28
|
* Remove code which updates `application_services_state.last_txn` (#12680)Richard van der Hoff2022-05-171-10/+0
| | | | This column is unused as of #12209, so let's stop writing to it.
* Track in memory events using weakrefs (#10533)Erik Johnston2022-05-171-0/+1
|
* Avoid unnecessary copies when filtering private read receipts. (#12711)Šimon Brandner2022-05-161-38/+26
| | | | | | A minor optimization to avoid unnecessary copying/building identical dictionaries when filtering private read receipts. Also clarifies comments and cleans-up some tests.
* Refactor `EventContext` (#12689)Erik Johnston2022-05-101-1/+3
| | | | | | | | | | Refactor how the `EventContext` class works, with the intention of reducing the amount of state we fetch from the DB during event processing. The idea here is to get rid of the cached `current_state_ids` and `prev_state_ids` that live in the `EventContext`, and instead defer straight to the database (and its caching). One change that may have a noticeable effect is that we now no longer prefill the `get_current_state_ids` cache on a state change. However, that query is relatively light, since its just a case of reading a table from the DB (unlike fetching state at an event which is more heavyweight). For deployments with workers this cache isn't even used. Part of #12684
* Support MSC3266 room summaries over federation (#11507)DeepBlueV7.X2022-05-051-0/+26
| | | | Signed-off-by: Nicolas Werner <nicolas.werner@hotmail.de>
* Use `private` instead of `hidden` in MSC2285 related code. (#12635)Šimon Brandner2022-05-051-16/+16
|
* Use `getClientAddress` instead of `getClientIP`. (#12599)Patrick Cloke2022-05-043-4/+4
| | | | | getClientIP was deprecated in Twisted 18.4.0, which also added getClientAddress. The Synapse minimum version for Twisted is currently 18.9.0, so all supported versions have the new API.
* Implement changes to MSC2285 (hidden read receipts) (#12168)Šimon Brandner2022-05-041-53/+76
| | | | | * Changes hidden read receipts to be a separate receipt type (instead of a field on `m.read`). * Updates the `/receipts` endpoint to accept `m.fully_read`.
* Remove unstable/unspecced login types. (#12597)Patrick Cloke2022-05-041-3/+1
| | | | | | * `m.login.jwt`, which was never specced and has been deprecated since Synapse 1.16.0. (`org.matrix.login.jwt` can be used instead.) * `uk.half-shot.msc2778.login.application_service`, which was stabilized as part of the Matrix spec v1.2 release.
* Use constants for receipt types in tests. (#12582)Šimon Brandner2022-04-281-19/+19
|
* Fix logging of incorrect status codes for disconnected requests (#12580)Sean Quah2022-04-282-2/+26
| | | | | | | | | | | | | | The status code of requests must always be set, regardless of client disconnection, otherwise they will always be logged as 200!. Broken for `respond_with_json` in f48792eec43f893f4f893ffdcbf00f8958b6f6b5. Broken for `respond_with_json_bytes` in 3e58ce72b42f2ae473c1e76a967548cd6fa7e2e6. Broken for `respond_with_html_bytes` in ea26e9a98b0541fc886a1cb826a38352b7599dbe. Signed-off-by: Sean Quah <seanq@element.io>
* Prefer `make_awaitable` over `defer.succeed` in tests (#12505)Sean Quah2022-04-274-28/+25
| | | | | | | | | | | When configuring the return values of mocks, prefer awaitables from `make_awaitable` over `defer.succeed`. `Deferred`s are only awaitable once, so it is inappropriate for a mock to return the same `Deferred` multiple times. Also update `run_in_background` to support functions that return arbitrary awaitables. Signed-off-by: Sean Quah <seanq@element.io>
* Remove unused `# type: ignore`s (#12531)David Robertson2022-04-271-6/+3
| | | | | | | | | | | | | | | | | | | | | | Over time we've begun to use newer versions of mypy, typeshed, stub packages---and of course we've improved our own annotations. This makes some type ignore comments no longer necessary. I have removed them. There was one exception: a module that imports `select.epoll`. The ignore is redundant on Linux, but I've kept it ignored for those of us who work on the source tree using not-Linux. (#11771) I'm more interested in the config line which enforces this. I want unused ignores to be reported, because I think it's useful feedback when annotating to know when you've fixed a problem you had to previously ignore. * Installing extras before typechecking Lacking an easy way to install all extras generically, let's bite the bullet and make install the hand-maintained `all` extra before typechecking. Now that https://github.com/matrix-org/backend-meta/pull/6 is merged to the release/v1 branch.
* Bound ephemeral events by key (#12544)Nick Mills-Barrett2022-04-261-0/+82
| | | | Co-authored-by: Brad Murray <bradtgmurray@gmail.com> Co-authored-by: Andrew Morgan <andrewm@element.io>
* Prevent a sync request from removing a user's busy presence status (#12213)David Baker2022-04-131-0/+79
| | | | | | | | | | In trying to use the MSC3026 busy presence status, the user's status would be set back to 'online' next time they synced. This change makes it so that syncing does not affect a user's presence status if it is currently set to 'busy': it must be removed through the presence API. The MSC defers to implementations on the behaviour of busy presence, so this ought to remain compatible with the MSC.
* Add type hints for `tests/unittest.py`. (#12347)Richard van der Hoff2022-04-014-7/+13
| | | In particular, add type hints for get_success and friends, which are then helpful in a bunch of places.
* Remove redundant `get_success` calls in test code (#12346)Richard van der Hoff2022-04-013-17/+14
| | | There are a bunch of places we call get_success on an immediate value, which is unnecessary. Let's rip them out, and remove the redundant functionality in get_success and friends.
* Raise an exception when getting state at an outlier (#12191)Richard van der Hoff2022-04-011-18/+54
| | | | | | It seems like calling `_get_state_group_for_events` for an event where the state is unknown is an error. Accordingly, let's raise an exception rather than silently returning an empty result.
* Optimise `_get_state_after_missing_prev_event`: use `/state` (#12040)Richard van der Hoff2022-04-011-0/+225
| | | If we're missing most of the events in the room state, then we may as well call the /state endpoint, instead of individually requesting each and every event.
* Send device list updates to application services (MSC3202) - part 1 (#11881)Andrew Morgan2022-03-301-1/+120
| | | Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
* Bump `black` and `click` versions (#12320)David Robertson2022-03-291-3/+3
|
* Always allow the empty string as an avatar_url. (#12261)David Robertson2022-03-251-0/+6
| | | | | Hopefully this fixes #12257. Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
* Add type hints to tests files. (#12256)Dirk Klimpel2022-03-211-15/+20
|
* Add type hints to more tests files. (#12240)Dirk Klimpel2022-03-173-26/+42
|
* Add type hints to some tests/handlers files. (#12224)Dirk Klimpel2022-03-155-126/+155
|
* Add some type hints to the tests.handlers module. (#12207)Patrick Cloke2022-03-114-59/+71
|
* Remove unnecessary pass statements. (#12206)Patrick Cloke2022-03-111-1/+0
|
* Do not return allowed_room_ids from /hierarchy response. (#12175)Patrick Cloke2022-03-081-0/+3
| | | | | This field is only to be used in the Server-Server API, and not the Client-Server API, but was being leaked when a federation response was used in the /hierarchy API.
* Rename various ApplicationServices interested methods (#11915)Andrew Morgan2022-03-031-14/+42
|
* Make get_room_version use cached get_room_version_id. (#11808)lukasdenk2022-03-021-1/+4
|
* Remove the unstable `/spaces` endpoint. (#12073)Patrick Cloke2022-02-281-117/+2
| | | | | | | | ...and various code supporting it. The /spaces endpoint was from an old version of MSC2946 and included both a Client-Server and Server-Server API. Note that the unstable /hierarchy endpoint (from the final version of MSC2946) is not yet removed.
* Replace assertEquals and friends with non-deprecated versions. (#12092)Patrick Cloke2022-02-289-100/+100
|
* Add support for MSC3202: sending one-time key counts and fallback key usage ↵reivilibre2022-02-241-4/+190
| | | | | states to Application Services. (#11617) Co-authored-by: Erik Johnston <erik@matrix.org>
* Remove `HomeServer.get_datastore()` (#12031)Richard van der Hoff2022-02-2318-37/+39
| | | | | | | The presence of this method was confusing, and mostly present for backwards compatibility. Let's get rid of it. Part of #11733
* Allow modules to set a display name on registration (#12009)Brendan Abolivier2022-02-171-30/+93
| | | Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
* Adds misc missing type hints (#11953)Patrick Cloke2022-02-111-7/+2
|
* Add a callback to allow modules to deny 3PID (#11854)Brendan Abolivier2022-02-081-1/+75
| | | | | Part of the Tchap Synapse mainlining. This allows modules to implement extra logic to figure out whether a given 3PID can be added to the local homeserver. In the Tchap use case, this will allow a Synapse module to interface with the custom endpoint /internal_info.
* Add a background database update to purge account data for deactivated ↵reivilibre2022-02-021-0/+106
| | | | users. (#11655)
* Expose the registered device ID from the `register_appservice_user` test ↵reivilibre2022-02-021-2/+4
| | | | helper. (#11615)
* Send to-device messages to application services (#11215)Andrew Morgan2022-02-011-11/+270
| | | Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
* Configurable limits on avatars (#11846)Brendan Abolivier2022-01-281-2/+92
| | | | | | Only allow files which file size and content types match configured limits to be set as avatar. Most of the inspiration from the non-test code comes from matrix-org/synapse-dinsic#19
* Add a module callback to set username at registration (#11790)Brendan Abolivier2022-01-261-2/+77
| | | | | | This is in the context of mainlining the Tchap fork of Synapse. Currently in Tchap usernames are derived from the user's email address (extracted from the UIA results, more specifically the m.login.email.identity step). This change also exports the check_username method from the registration handler as part of the module API, so that a module can check if the username it's trying to generate is correct and doesn't conflict with an existing one, and fallback gracefully if not. Co-authored-by: David Robertson <davidr@element.io>
* Remove account data (including client config, push rules and ignored users) ↵reivilibre2022-01-241-0/+219
| | | | | upon user deactivation. (#11621) Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
* Correctly await on_logged_out callbacks (#11786)Brendan Abolivier2022-01-201-1/+27
|
* Fix a bug that corrupted the cache of federated space hierarchies (#11775)Sean Quah2022-01-201-2/+90
| | | | `FederationClient.get_room_hierarchy()` caches its return values, so refactor the code to avoid modifying the returned room summary.
* Support spaces with > 50 rooms in the /hierarchy endpoint. (#11695)Patrick Cloke2022-01-071-0/+32
| | | | | | | By returning all of the m.space.child state of the space, not just the first 50. The number of rooms returned is still capped at 50. For the federation API this implies that the requesting server will need to individually query for any other rooms it is not joined to.
* Test that bans win a join against a race when computing `/sync` response ↵David Robertson2022-01-071-3/+94
| | | | (#11701)
* Cache empty responses from `/user/devices` (#11587)David Robertson2022-01-051-0/+96
| | | If we've never made a request to a remote homeserver, we should cache the response---even if the response is "this user has no devices".
* Refactor the way we set `outlier` (#11634)Richard van der Hoff2022-01-051-3/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * `_auth_and_persist_outliers`: mark persisted events as outliers Mark any events that get persisted via `_auth_and_persist_outliers` as, well, outliers. Currently this will be a no-op as everything will already be flagged as an outlier, but I'm going to change that. * `process_remote_join`: stop flagging as outlier The events are now flagged as outliers later on, by `_auth_and_persist_outliers`. * `send_join`: remove `outlier=True` The events created here are returned in the result of `send_join` to `FederationHandler.do_invite_join`. From there they are passed into `FederationEventHandler.process_remote_join`, which passes them to `_auth_and_persist_outliers`... which sets the `outlier` flag. * `get_event_auth`: remove `outlier=True` stop flagging the events returned by `get_event_auth` as outliers. This method is only called by `_get_remote_auth_chain_for_event`, which passes the results into `_auth_and_persist_outliers`, which will flag them as outliers. * `_get_remote_auth_chain_for_event`: remove `outlier=True` we pass all the events into `_auth_and_persist_outliers`, which will now flag the events as outliers. * `_check_sigs_and_hash_and_fetch`: remove unused `outlier` parameter This param is now never set to True, so we can remove it. * `_check_sigs_and_hash_and_fetch_one`: remove unused `outlier` param This is no longer set anywhere, so we can remove it. * `get_pdu`: remove unused `outlier` parameter ... and chase it down into `get_pdu_from_destination_raw`. * `event_from_pdu_json`: remove redundant `outlier` param This is never set to `True`, so can be removed. * changelog * update docstring
* Allow events to be created with no `prev_events` (MSC2716) (#11243)Eric Eastwood2021-12-101-0/+103
| | | | | The event still needs to have `auth_events` defined to be valid. Split out from https://github.com/matrix-org/synapse/pull/11114
* Support unprefixed versions of fallback key property names. (#11541)Hubert Chathi2021-12-091-5/+25
|
* Save the OIDC session ID (sid) with the device on login (#11482)Quentin Gliech2021-12-064-33/+188
| | | As a step towards allowing back-channel logout for OIDC.
* Support the stable /hierarchy endpoint from MSC2946 (#11329)Patrick Cloke2021-11-291-29/+65
| | | | | | This also makes additional updates where the implementation had drifted from the approved MSC. Unstable endpoints will be removed at a later data.
* Fix checking whether a room can be published on creation. (#11392)Patrick Cloke2021-11-191-33/+62
| | | | | | | | If `room_list_publication_rules` was configured with a rule with a non-wildcard alias and a room was created with an alias then an internal server error would have been thrown. This fixes the error and properly applies the publication rules during room creation.
* Keep fallback key marked as used if it's re-uploaded (#11382)Hubert Chathi2021-11-191-1/+31
|
* Rename `get_access_token_for_user_id` method to ↵reivilibre2021-11-171-5/+5
| | | | `create_access_token_for_user_id` (#11369)
* Avoid sharing room hierarchy responses between users (#11355)Sean Quah2021-11-161-0/+55
| | | | | Different users may be allowed to see different rooms within a space, so sharing responses between users is inadvisable.
* Misc typing fixes for tests, part 2 of N (#11330)David Robertson2021-11-161-3/+6
|
* Add support for the stable version of MSC2778 (#11335)Tulir Asokan2021-11-151-1/+4
| | | | | | | | | * Add support for the stable version of MSC2778 Signed-off-by: Tulir Asokan <tulir@maunium.net> * Expect m.login.application_service in login and password provider tests Signed-off-by: Tulir Asokan <tulir@maunium.net>
* Clarifications and small fixes to to-device related code (#11247)Andrew Morgan2021-11-091-2/+6
| | | Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
* Support filtering by relations per MSC3440 (#11236)Patrick Cloke2021-11-091-2/+3
| | | | Adds experimental support for `relation_types` and `relation_senders` fields for filters.
* Include cross-signing signatures when syncing remote devices for the first ↵Erik Johnston2021-11-091-0/+151
| | | | | | | time (#11234) When fetching remote devices for the first time, we did not correctly include the cross signing keys in the returned results. c.f. #11159
* Add a linearizer on (appservice, stream) when handling ephemeral events. ↵Nick Barrett2021-11-031-0/+51
| | | | | (#11207) Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
* Add knock information in admin exported data (#11171)Rafael Gonçalves2021-10-281-2/+33
| | | Signed-off-by: Rafael Goncalves <rafaelgoncalves@riseup.net>
* Delete messages from `device_inbox` table when deleting device (#10969)Dirk Klimpel2021-10-271-0/+31
| | | Fixes: #9346
* Don't set new room alias before potential 403 (#10930)AndrewFerr2021-10-251-1/+101
| | | | | | Fixes: #10929 Signed-off-by: Andrew Ferrazzutti <fair@miscworks.net>
* Fix adding excluded users to the private room sharing tables when joining a ↵David Robertson2021-10-211-14/+53
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | room (#11143) * We only need to fetch users in private rooms * Filter out `user_id` at the top * Discard excluded users in the top loop We weren't doing this in the "First, if they're our user" branch so this is a bugfix. * The caller must check that `user_id` is included This is in the docstring. There are two call sites: - one in `_handle_room_publicity_change`, which explicitly checks before calling; - and another in `_handle_room_membership_event`, which returns early if the user is excluded. So this change is safe. * Test joining a private room with an excluded user * Tweak an existing test * Changelog * test docstring * lint
generated by cgit-pink 1.4.1 (git 2.36.1) at 2024-11-01 20:28:09 +0000