summary refs log tree commit diff
path: root/tests (follow)
Commit message (Collapse)AuthorAgeFilesLines
* Move media retention tests out of rest tests (#16684)David Robertson2023-11-271-0/+0
| | | | | | | | * Move media retention tests out of rest tests AFAICS this doesn't make any HTTP requests and so it ought not to belong in `tests.rest`. * Changelog
* Keep track of `user_ips` and `monthly_active_users` when delegating auth ↵David Robertson2023-11-236-44/+126
| | | | | | | | | | | | | (#16672) * Describe `insert_client_ip` * Pull out client_ips and MAU tracking to BaseAuth * Define HAS_AUTHLIB once in tests sick of copypasting * Track ips and token usage when delegating auth * Test that we track MAU and user_ips * Don't track `__oidc_admin`
* Revert "Add a Postgres `REPLICA IDENTITY` to tables that do not have an ↵Erik Johnston2023-11-161-84/+1
| | | | | implicit one. This should allow use of Postgres logical replication. (#16456)" (#16651) This reverts commit 69afe3f7a0d89f3422ddbd3aa16bc9bbc01056eb.
* Speed up persisting large number of outliers (#16649)Erik Johnston2023-11-161-1/+75
| | | Recalculating the roots tuple every iteration could be very expensive, so instead let's do a topological sort.
* Fix sending out of order `POSITION` over replication (#16639)Erik Johnston2023-11-161-0/+8
| | | | | If a worker reconnects to Redis we send out the current positions of all our streams. However, if we're also trying to send out a backlog of RDATA at the same time then we can end up sending a `POSITION` with the current token *before* we've sent all the RDATA before the current token. This doesn't cause actual bugs as the receiving servers see the POSITION, fetch the relevant rows from the DB, and then ignore the old RDATA as they come in. However, this is inefficient so it'd be better if we didn't send out-of-order positions
* Fix test not detecting tables with missing primary keys and missing replica ↵reivilibre2023-11-161-3/+4
| | | | | | | | | | | | | | | identities, then add more replica identities. (#16647) * Fix the CI query that did not detect all cases of missing primary keys * Add more missing REPLICA IDENTITY entries * Newsfile Signed-off-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org> --------- Signed-off-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org>
* Add an Admin API to temporarily grant the ability to update an existing ↵David Robertson2023-11-154-2/+410
| | | | cross-signing key without UIA (#16634)
* Asynchronous Uploads (#15503)Sumner Evans2023-11-151-1/+3
| | | Support asynchronous uploads as defined in MSC2246.
* Use full GitHub links instead of bare issue numbers. (#16637)Patrick Cloke2023-11-1515-29/+38
|
* Add a Postgres `REPLICA IDENTITY` to tables that do not have an implicit ↵reivilibre2023-11-131-1/+83
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | one. This should allow use of Postgres logical replication. (#16456) * Add Postgres replica identities to tables that don't have an implicit one Fixes #16224 * Newsfile Signed-off-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org> * Move the delta to version 83 as we missed the boat for 82 * Add a test that all tables have a REPLICA IDENTITY * Extend the test to include when indices are deleted * isort * black * Fully qualify `oid` as it is a 'hidden attribute' in Postgres 11 * Update tests/storage/test_database.py Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com> * Add missed tables --------- Signed-off-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org> Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
* Use attempt_to_set_autocommit everywhere. (#16615)Patrick Cloke2023-11-091-10/+5
| | | To avoid asserting the type of the database connection.
* Use dbname instead of database for Postgres config. (#16618)Patrick Cloke2023-11-092-4/+4
|
* Convert simple_select_one_txn and simple_select_one to return tuples. (#16612)Patrick Cloke2023-11-097-26/+21
|
* Return attrs for more media repo APIs. (#16611)Patrick Cloke2023-11-093-19/+19
|
* Bulk-invalidate e2e cached queries after claiming keys (#16613)David Robertson2023-11-091-0/+117
| | | | Co-authored-by: Patrick Cloke <patrickc@matrix.org>
* Avoid updating the same rows multiple times with simple_update_many_txn. ↵Patrick Cloke2023-11-071-2/+2
| | | | | | (#16609) simple_update_many_txn had a bug in it which would cause each update to be applied twice.
* Avoid executing no-op queries. (#16583)Patrick Cloke2023-11-071-20/+5
| | | | | | If simple_{insert,upsert,update}_many_txn is called without any data to modify then return instead of executing the query. This matches the behavior of simple_{select,delete}_many_txn.
* More tests for the simple_* methods. (#16596)Patrick Cloke2023-11-071-18/+628
| | | | Expand tests for the simple_* database methods, additionally test against both PostgreSQL and SQLite variants.
* Bump twisted from 23.8.0 to 23.10.0 (#16588)dependabot[bot]2023-11-012-2/+2
|
* Remove remaining usage of cursor_to_dict. (#16564)Patrick Cloke2023-10-313-21/+16
|
* Claim local one-time-keys in bulk (#16565)David Robertson2023-10-301-0/+158
| | | | Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
* Claim fallback keys in bulk (#16570)David Robertson2023-10-301-0/+77
|
* Ensure local invited & knocking users leave before purge. (#16559)Patrick Cloke2023-10-271-1/+52
| | | | | This is mostly useful for federated rooms where some users would get stuck in the invite or knock state when the room was purged from their homeserver.
* Add new module API for adding custom fields to events `unsigned` section ↵Erik Johnston2023-10-272-1/+60
| | | | (#16549)
* Remove more usages of cursor_to_dict. (#16551)Patrick Cloke2023-10-264-11/+10
| | | Mostly to improve type safety.
* Add a new module API to update user presence state. (#16544)Patrick Cloke2023-10-262-11/+119
| | | | | | | | | | This adds a module API which allows a module to update a user's presence state/status message. This is useful for controlling presence from an external system. To fully control presence from the module the presence.enabled config parameter gains a new state of "untracked" which disables internal tracking of presence changes via user actions, etc. Only updates from the module will be persisted and sent down sync properly).
* Convert simple_select_list and simple_select_list_txn to return lists of ↵Patrick Cloke2023-10-269-240/+262
| | | | | tuples (#16505) This should use fewer allocations and improves type hints.
* Allow multiple workers to write to receipts stream. (#16432)Erik Johnston2023-10-252-4/+256
| | | Fixes #16417
* Fix tests on Twisted trunk. (#16528)Patrick Cloke2023-10-255-111/+94
| | | | | | | | | | | | | Twisted trunk makes a change to the `TLSMemoryBIOFactory` where the underlying protocol is changed from `TLSMemoryBIOProtocol` to `BufferingTLSTransport` to improve performance of TLS code (see https://github.com/twisted/twisted/issues/11989). In order to properly hook this code up in tests we need to pass the test reactor's clock into `TLSMemoryBIOFactory` to avoid the global (trial) reactor being used by default. Twisted does something similar internally for tests: https://github.com/twisted/twisted/blob/157cd8e659705940e895d321339d467e76ae9d0a/src/twisted/web/test/test_agent.py#L871-L874
* Fix http/s proxy authentication with long username/passwords (#16504)Richard Brežák2023-10-241-0/+21
|
* Revert "Add test case to detect dodgy b64 encoding"David Robertson2023-10-241-14/+0
| | | | | | | | | This reverts commit 5fe76b9434e22bb752c252dd9c66c3c2bfb90dfc. I think I had this accidentally commited on my local develop branch, and so it accidentally got merged into upstream develop. This should re-land with corrections in #16504.
* Merge branch 'develop' of github.com:matrix-org/synapse into developDavid Robertson2023-10-245-15/+129
|\
| * Fix type hint errors from Twisted trunk (#16526)Patrick Cloke2023-10-234-3/+5
| |
| * Fix bug where a new writer advances their token too quickly (#16473)Erik Johnston2023-10-231-12/+124
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Fix bug where a new writer advances their token too quickly When starting a new writer (for e.g. persisting events), the `MultiWriterIdGenerator` doesn't have a minimum token for it as there are no rows matching that new writer in the DB. This results in the the first stream ID it acquired being announced as persisted *before* it actually finishes persisting, if another writer gets and persists a subsequent stream ID. This is due to the logic of setting the minimum persisted position to the minimum known position of across all writers, and the new writer starts off not being considered. * Fix sending out POSITIONs when our token advances without update Broke in #14820 * For replication HTTP requests, only wait for minimal position
* | Add test case to detect dodgy b64 encodingDavid Robertson2023-10-231-0/+14
|/
* Avoid sending massive replication updates when purging a room. (#16510)Patrick Cloke2023-10-181-29/+62
|
* Convert DeviceLastConnectionInfo to attrs. (#16507)Patrick Cloke2023-10-171-67/+70
| | | To improve type safety & memory usage.
* Fix a bug where servers could be marked as up when they were failing (#16506)Patrick Cloke2023-10-171-0/+75
| | | | After this change a server will only be reported as back online if they were previously having requests fail.
* Convert state delta processing from a dict to attrs. (#16469)Patrick Cloke2023-10-161-1/+1
| | | For improved type checking & memory usage.
* Convert user_get_threepids response to attrs. (#16468)Patrick Cloke2023-10-111-4/+4
| | | This improves type annotations by not having a dictionary of Any values.
* Convert simple_select_many_batch, simple_select_many_txn to tuples. (#16444)Patrick Cloke2023-10-111-26/+38
|
* Handle content types with parameters. (#16440)Patrick Cloke2023-10-111-1/+18
|
* Apply join rate limiter outside the lineariser (#16441)David Robertson2023-10-061-0/+24
|
* Return ThumbnailInfo in more places (#16438)Patrick Cloke2023-10-061-18/+18
| | | | Improves type hints by using concrete types instead of dictionaries.
* Stop sending incorrect knock_state_events. (#16403)Patrick Cloke2023-10-061-1/+1
| | | | | | | | | Synapse was incorrectly implemented with a knock_state_events property on some APIs (instead of knock_room_state). This was correct in Synapse 1.70.0, but *both* fields were sent to also be compatible with Synapse versions expecting the wrong field. Enough time has passed that only the correct field needs to be included/handled.
* Register media servlets via regex. (#16419)Patrick Cloke2023-10-0610-219/+193
| | | | | This converts the media servlet URLs in the same way as (most) of the rest of Synapse. This will give more flexibility in the versions each endpoint exists under.
* Factor out `MultiWriter` token from `RoomStreamToken` (#16427)Erik Johnston2023-10-051-4/+4
|
* Some refactors around receipts stream (#16426)Erik Johnston2023-10-042-11/+23
|
* Remove Python version from `/_synapse/admin/v1/server_version` (#16380)MomentQYC2023-10-021-3/+1
| | | | There's no reason to expose the full Python version over what is frequently a public API.
* Support rendering some media downloads as inline (#15988)Will Hunt2023-09-292-4/+65
| | | | | Use an `inline` Content-Disposition header when the media is "safe" to display inline (some known text, image, video, audio formats).
* Improve state types. (#16395)Patrick Cloke2023-09-281-5/+8
|
* Add a cache around server ACL checking (#16360)Patrick Cloke2023-09-261-13/+22
| | | | | * Pre-compiles the server ACLs onto an object per room and invalidates them when new events come in. * Converts the server ACL checking into Rust.
* Add support for pydantic v2 via pydantic.v1 compat module (#16332)Maxwell G2023-09-251-1/+7
| | | While maintaining support with pydantic v1.
* Convert more cached return values to immutable types (#16356)Patrick Cloke2023-09-201-16/+19
|
* Stop patching EventBase.__eq__ in tests. (#16349)Patrick Cloke2023-09-182-36/+30
| | | | It is clearer to directly test equality instead of doing indirect assertions via patching __eq__.
* Filter locked users in the admin API (#16328)Hanadi2023-09-181-0/+26
| | | | Co-authored-by: Hanadi Tamimi <hanadi.tamimi@sdui.de>
* Return an immutable value from get_latest_event_ids_in_room. (#16326)Patrick Cloke2023-09-186-25/+33
|
* Additional validation of receipts (#16327)Patrick Cloke2023-09-182-161/+214
| | | | Reject invalid receipts with a reasonable error message & expands tests for receipts.
* Add automatic purge after all users forget a room (#15488)Mathieu Velten2023-09-153-23/+162
| | | | | | Also add restore of purge/shutdown rooms after a synapse restart. Co-authored-by: Eric Eastwood <erice@matrix.org> Co-authored-by: Erik Johnston <erikj@matrix.org>
* Refactor `get_user_by_id` (#16316)Erik Johnston2023-09-142-27/+33
|
* Fix using dehydrated devices (MSC2697) & refresh tokens (#16288)Hanadi2023-09-131-1/+9
| | | | | Refresh tokens were not correctly moved to the rehydrated device (similar to how the access token is currently handled). This resulted in invalid refresh tokens after rehydration.
* Refactor storing of server keys (#16261)Erik Johnston2023-09-123-190/+26
|
* Add the List-Unsubscribe header for notification emails. (#16274)Patrick Cloke2023-09-111-0/+55
| | | | | | | | Adds both the List-Unsubscribe (RFC2369) and List-Unsubscribe-Post (RFC8058) headers to push notification emails, which together should: * Show an "Unsubscribe" link in the MUA UI when viewing Synapse notification emails. * Enable "one-click" unsubscribe (the user never leaves their MUA, which automatically makes a POST request to the specified endpoint).
* Update ruff config (#16283)Patrick Cloke2023-09-0816-38/+38
| | | Enable additional checks & clean-up unneeded configuration.
* Avoid temporary storage of sensitive information. (#16272)Patrick Cloke2023-09-081-0/+13
| | | | During the UI auth process, avoid storing sensitive information into the database.
* Send the opentracing span information to appservices (#16227)Marcel2023-09-061-6/+12
|
* Handle "registration_enabled" parameter for CAS (#16262)Aurélien Grimpard2023-09-061-0/+17
| | | | Similar to OIDC, CAS providers can now disable registration such that only existing users are able to login via SSO.
* Merge remote-tracking branch 'origin/release-v1.92' into developPatrick Cloke2023-09-062-191/+23
|\
| * Merge remote-tracking branch 'origin/release-v1.91' into release-v1.92Patrick Cloke2023-09-062-191/+23
| |\
| | * Revert MSC3861 introspection cache, admin impersonation and account lock ↵Quentin Gliech2023-09-062-192/+24
| | | | | | | | | | | | (#16258)
* | | Delete device messages asynchronously and in staged batches (#16240)Mathieu Velten2023-09-061-0/+47
| | |
* | | Fix appservices being unable to handle to_device messages for multiple users ↵Will Hunt2023-09-051-0/+125
| | | | | | | | | | | | (#16251)
* | | Add MSC4040 `matrix-fed` service lookups (#16137)Travis Ralston2023-09-051-29/+294
| | |
* | | Time out busy presence status & test multi-device busy (#16174)Patrick Cloke2023-09-051-3/+101
| | | | | | | | | | | | | | | | | | | | | Add a (long) timeout to when a "busy" device is considered not online. This does *not* match MSC3026, but is a reasonable thing for an implementation to do. Expands tests for the (unstable) busy presence with multiple devices.
* | | Track presence state per-device and combine to a user state. (#16066)Patrick Cloke2023-09-051-10/+490
|/ / | | | | | | | | | | | | | | | | | | | | Tracks presence on an individual per-device basis and combine the per-device state into a per-user state. This should help in situations where a user has multiple devices with conflicting status (e.g. one is syncing with unavailable and one is syncing with online). The tie-breaking is done by priority: BUSY > ONLINE > UNAVAILABLE > OFFLINE
* | Don't wake up destination transaction queue if they're not due for retry. ↵Erik Johnston2023-09-043-23/+66
| | | | | | | | (#16223)
* | Add last_seen_ts to the admin users API (#16218)Mathieu Velten2023-09-042-0/+61
| |
* | Don't reset retry timers on "valid" error codes (#16221)Erik Johnston2023-09-041-2/+2
| |
* | Log the details of background update failures (#16212)David Robertson2023-09-011-1/+23
| |
* | Fix rare bug that broke looping calls (#16210)Erik Johnston2023-08-301-0/+2
| | | | | | | | | | | | | | | | | | | | * Fix rare bug that broke looping calls We can't interact with the reactor from the main thread via looping call. Introduced in v1.90.0 / #15791. * Newsfile
* | Stop writing to the event_txn_id table (#16175)Patrick Cloke2023-08-301-11/+4
| |
* | Describe which rate limiter was hit in logs (#16135)David Robertson2023-08-303-29/+84
| |
* | Bump mypy-zope & mypy. (#16188)Patrick Cloke2023-08-2933-111/+111
| |
* | Support IPv6-only SMTP servers (#16155)Evilham2023-08-293-12/+113
| | | | | | | | Use Twisted HostnameEndpoint to connect to SMTP servers (instead of connectTCP/connectSSL) which properly supports IPv6-only servers.
* | Pass the device ID around in the presence handler (#16171)Patrick Cloke2023-08-281-11/+27
| | | | | | | | | | | | Refactoring to pass the device ID (in addition to the user ID) through the presence handler (specifically the `user_syncing`, `set_state`, and `bump_presence_active_time` methods and their replication versions).
* | Combine logic about not overriding BUSY presence. (#16170)Patrick Cloke2023-08-281-7/+30
| | | | | | | | | | | | | | | | | | | | | | | | | | Simplify some of the presence code by reducing duplicated code between worker & non-worker modes. The main change is to push some of the logic from `user_syncing` into `set_state`. This is done by passing whether the user is setting the presence via a `/sync` with a new `is_sync` flag to `set_state`. If this is `true` some additional logic is performed: * Don't override `busy` presence. * Update the `last_user_sync_ts`. * Never update the status message.
* | Task scheduler: add replication notify for new task to launch ASAP (#16184)Mathieu Velten2023-08-281-18/+40
| |
* | Prepare unit tests for Python 3.12 (#16099)V024602023-08-259-71/+80
| |
* | Replace simple_async_mock with AsyncMock (#16180)Patrick Cloke2023-08-2514-160/+139
| | | | | | | | Python 3.8 has a native AsyncMock, use it instead of a custom implementation.
* | Replace make_awaitable with AsyncMock (#16179)Patrick Cloke2023-08-2435-603/+506
| | | | | | | | Python 3.8 provides a native AsyncMock, we can replace the homegrown version we have.
* | Fix typechecking with twisted trunk (#16121)David Robertson2023-08-241-8/+6
| |
* | Add `Retry-After` to M_LIMIT_EXCEEDED error responses (#16136)Will Hunt2023-08-242-6/+54
| | | | | | Implements MSC4041 behind an experimental configuration flag.
* | Stabilize support for MSC3958 (suppress notifications from edits). (#16113)Patrick Cloke2023-08-231-1/+0
|/
* Filter out unwanted user_agents from udv. (#16124)Neil Johnson2023-08-231-0/+65
|
* Add tests for restoring the presence state after a restart. (#16151)Patrick Cloke2023-08-231-0/+116
|
* Properly update retry_last_ts when hitting the maximum retry interval (#16156)DeepBlueV7.X2023-08-231-0/+51
| | | | | | | | | | | | | | | | | | | | | * Properly update retry_last_ts when hitting the maximum retry interval This was broken in 1.87 when the maximum retry interval got changed from almost infinite to a week (and made configurable). fixes #16101 Signed-off-by: Nicolas Werner <nicolas.werner@hotmail.de> * Add changelog * Change fix + add test * Add comment --------- Signed-off-by: Nicolas Werner <nicolas.werner@hotmail.de> Co-authored-by: Mathieu Velten <mathieuv@matrix.org>
* Clean-up presence tests (#16158)Patrick Cloke2023-08-221-92/+37
| | | Reduce duplicated code & remove unused variables.
* Disable `m.3pid_changes` capability when MSC3861 is enabled. (#16134)Mathieu Velten2023-08-221-0/+5
|
* Add an admin endpoint to allow authorizing server to signal token ↵Shay2023-08-222-1/+95
| | | | revocations (#16125)
* Fix user directory test for deactivated support user. (#16157)Patrick Cloke2023-08-221-8/+11
| | | | Support users should not be added to the user directory after being deactivated.
* Properly call setup_background_tasks in unit tests. (#16150)Patrick Cloke2023-08-221-2/+0
| | | | | This should only be called on HomeServer objects which are configured to run background tasks, which is automatically (and properly) done via the call to setup().
* Disallow user_consent where experimental MSC3861 is enabled (#16127)Hugh Nimmo-Smith2023-08-221-0/+16
|
* Implements a task scheduler for resumable potentially long running tasks ↵Mathieu Velten2023-08-211-0/+186
| | | | (#15891)
* Speed up state res in rare case we don't have all events (#16116)Erik Johnston2023-08-181-48/+193
| | | | | | | If we don't have all the auth events in a room then not all state events will have a chain cover index. Even so, we can still use the chain cover index on the events that do have it, rather than bailing and using the slower functions. This situation should not arise for newly persisted rooms, as we check we have the full auth chain for each event, but can happen for existing rooms. c.f. #15245
* MSC3861: allow impersonation by an admin using a query param (#16132)Mathieu Velten2023-08-181-0/+35
|
* Allow filtering for admins in the list accounts admin API (#16114)Alexander Fechler2023-08-181-0/+38
|
* Add cache to `get_server_keys_json_for_remote` (#16123)Erik Johnston2023-08-181-38/+23
|
* Cache token introspection response from OIDC provider (#16117)Shay2023-08-171-0/+62
|
* Fix database performance of read/write worker locks (#16061)Erik Johnston2023-08-171-3/+4
| | | | | | | We were seeing serialization errors when taking out multiple read locks. The transactions were retried, so isn't causing any failures. Introduced in #15782.
* Run pyupgrade for python 3.7 & 3.8. (#16110)Patrick Cloke2023-08-1513-42/+43
|
* Implements admin API to lock an user (MSC3939) (#15870)Mathieu Velten2023-08-103-1/+114
|
* Support MSC3814: Dehydrated Devices Part 2 (#16010)Shay2023-08-082-7/+79
|
* Fix deletion for Dehydrated Devices (#16046)Shay2023-08-041-1/+138
|
* Move support for application service query parameter authorization behind a ↵Shay2023-08-031-4/+81
| | | | configuration option (#16017)
* Allow config of the backoff algorithm for the federation client. (#15754)Mathieu Velten2023-08-032-13/+18
| | | | | | | | | | | Adds three new configuration variables: * destination_min_retry_interval is identical to before (10mn). * destination_retry_multiplier is now 2 instead of 5, the maximum value will be reached slower. * destination_max_retry_interval is one day instead of (essentially) infinity. Capping this will cause destinations to continue to be retried sometimes instead of being lost forever. The previous value was 2 ^ 62 milliseconds.
* Add forward-compatibility for the redacts property (MSC2174). (#16013)Patrick Cloke2023-08-021-17/+50
| | | | | | | | The location of the redacts field changes in room version 11. Ensure it is copied to the *new* location for *old* room versions for forwards-compatibility with clients. Note that copying it to the *old* location for the *new* room version was previously handled.
* Update MSC3958 support to interact with intentional mentions. (#15992)Patrick Cloke2023-08-021-2/+19
| | | | | | | * Updates the rule ID. * Use `event_property_is` instead of `event_match`. This updates the implementation of MSC3958 to match the latest text from the MSC.
* Trim whitespace when setting display names (#16031)Mohit Rathee2023-08-011-0/+12
|
* Add ability to wait for locks and add locks to purge history / room deletion ↵Erik Johnston2023-07-313-2/+128
| | | | | (#15791) c.f. #13476
* Fix 404 on /profile when the display name is empty but not the avatar (#16012)Mathieu Velten2023-07-271-0/+10
|
* Remove support for legacy application service paths (#15964)Shay2023-07-261-53/+0
|
* Support MSC3814: Dehydrated Devices (#15929)Shay2023-07-242-3/+246
| | | | | | Signed-off-by: Nicolas Werner <n.werner@famedly.com> Co-authored-by: Nicolas Werner <n.werner@famedly.com> Co-authored-by: Nicolas Werner <89468146+nico-famedly@users.noreply.github.com> Co-authored-by: Hubert Chathi <hubert@uhoreg.ca>
* Properly handle redactions of creation events (#15973)Shay2023-07-231-2/+7
|
* Actually stop reading from column `user_id` of tables `profiles` (#15955)Shay2023-07-231-1/+1
|
* Merge branch 'master' into developOlivier Wilkinson (reivilibre)2023-07-182-0/+157
|\
| * Revert "Stop writing to column `user_id` of tables `profiles` and ↵Shay2023-07-182-0/+157
| | | | | | | | | | | | | | | | | | `user_filters`. (#15953) * Revert "Stop writing to column `user_id` of tables `profiles` and `user_filters` (#15787)" This reverts commit f25b0f88081bb436bef914983cff7087b54eba5f. * newsfragement
* | Support room version 11 (#15912)Patrick Cloke2023-07-182-18/+33
| | | | | | | | | | And fix a bug in the implementation of the updated redaction format (MSC2174) where the top-level redacts field was not properly added for backwards-compatibility.
* | Re-introduce the outbound federation proxy (#15913)Eric Eastwood2023-07-1817-39/+396
| | | | | | | | | | Allow configuring the set of workers to proxy outbound federation traffic through (`outbound_federation_restricted_to`). This is useful when you have a worker setup with `federation_sender` instances responsible for sending outbound federation requests and want to make sure *all* outbound federation traffic goes through those instances. Before this change, the generic workers would still contact federation themselves for things like profile lookups, backfill, etc. This PR allows you to set more strict access controls/firewall for all workers and only allow the `federation_sender`'s to contact the outside world.
* | Mark get_user_in_directory private since only used in tests (#15884)Mathieu Velten2023-07-122-12/+12
| |
* | Unix Sockets for HTTP Replication (#15708)Jason Little2023-07-112-2/+37
| | | | | | | | | | | | | | | | | | Unix socket support for `federation` and `client` Listeners has existed now for a little while(since [1.81.0](https://github.com/matrix-org/synapse/pull/15353)), but there was one last hold out before it could be complete: HTTP Replication communication. This should finish it up. The Listeners would have always worked, but would have had no way to be talked to/at. --------- Co-authored-by: Eric Eastwood <madlittlemods@gmail.com> Co-authored-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org> Co-authored-by: Eric Eastwood <erice@element.io>
* | Add + as an allowed character for Matrix IDs (MSC4009) (#15911)Patrick Cloke2023-07-111-6/+5
|/
* Add `Server` to Access-Control-Expose-Headers header (#15908)Michael Telatynski2023-07-111-1/+1
|
* Revert "Federation outbound proxy" (#15910)Eric Eastwood2023-07-1016-299/+37
| | | | | | Revert "Federation outbound proxy (#15773)" This reverts commit b07b14b494ae1dd564b4c44f844c9a9545b3d08a.
* Fix downgrading to previous version of Synapse (#15907)Erik Johnston2023-07-101-2/+6
| | | We do this by marking the constraint as deferrable.
* Stop writing to column `user_id` of tables `profiles` and `user_filters` ↵Shay2023-07-072-157/+0
| | | | (#15787)
* Remove `worker_replication_*` deprecated settings, with helpful errors on ↵Jason Little2023-07-072-29/+4
| | | | | | startup (#15860) Co-authored-by: reivilibre <oliverw@matrix.org>
* Federation outbound proxy (#15773)Eric Eastwood2023-07-0516-37/+299
| | | | | | | Allow configuring the set of workers to proxy outbound federation traffic through (`outbound_federation_restricted_to`). This is useful when you have a worker setup with `federation_sender` instances responsible for sending outbound federation requests and want to make sure *all* outbound federation traffic goes through those instances. Before this change, the generic workers would still contact federation themselves for things like profile lookups, backfill, etc. This PR allows you to set more strict access controls/firewall for all workers and only allow the `federation_sender`'s to contact the outside world. The original code is from @erikjohnston's branches which I've gotten in-shape to merge.
* Remove support for Python 3.7 (#15851)Eric Eastwood2023-07-051-8/+2
| | | Fix https://github.com/matrix-org/synapse/issues/15836
* Add basic read/write lock (#15782)Erik Johnston2023-07-051-1/+282
|
* Add foreign key constraint to `event_forward_extremities`. (#15751)Erik Johnston2023-07-052-4/+258
|
* Add not_user_type param to the list accounts admin API (#15844)Michael Weimann2023-07-041-0/+78
| | | Signed-off-by: Michael Weimann <michaelw@element.io>
* Add login spam checker API (#15838)Erik Johnston2023-06-261-2/+106
|
* Fix forgotten rooms missing in initial sync (#15815)Nicolas Werner2023-06-211-0/+21
| | | | | | | | If you leave a room and forget it, then rejoin it, the room would be missing from the next initial sync. fixes #13262 Signed-off-by: Nicolas Werner <n.werner@famedly.com>
* Allow for the configuration of max request retries and min/max retry delays ↵Mathieu Velten2023-06-211-1/+19
| | | | in the matrix federation client (#15783)
* Switch from `matrix://` to `matrix-federation://` scheme for internal ↵Eric Eastwood2023-06-202-19/+23
| | | | | | | Synapse routing of outbound federation traffic (#15806) `matrix://` is a registered specced scheme nowadays and doesn't make sense for our internal to Synapse use case anymore. ([discussion] (https://github.com/matrix-org/synapse/pull/15773#discussion_r1227598679))
* Remove experimental MSC2716 implementation to incrementally import history ↵Eric Eastwood2023-06-162-513/+0
| | | | | | | | | | | | into existing rooms (#15748) Context for why we're removing the implementation: - https://github.com/matrix-org/matrix-spec-proposals/pull/2716#issuecomment-1487441010 - https://github.com/matrix-org/matrix-spec-proposals/pull/2716#issuecomment-1504262734 Anyone wanting to continue MSC2716, should also address these leftover tasks: https://github.com/matrix-org/synapse/issues/10737 Closes https://github.com/matrix-org/synapse/issues/10737 in the fact that it is not longer necessary to track those things.
* push rules: fix internal conversion from _type to value (#15781)Mathieu Velten2023-06-161-0/+67
| | | Also fix wrong rule names for `is_user_mention` and `is_room_mention`.
* Regularly try to wake up dests instead of waiting for next PDU/EDU (#15743)Mathieu Velten2023-06-161-13/+9
|
* Fix unsafe hotserving behaviour for non-multimedia uploads. (#15680)Josh Qou2023-06-152-16/+16
| | | | | | | | | | | | | | | | | | | | | | | | | | | * Fix unsafe hotserving behaviour for non-multimedia uploads. * invert disposition assert * test_media_storage.py: run lint * test_base.py: /inline/attachment/s * Only return attachment for disposition type, update tests * Update synapse/media/_base.py Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com> * Update changelog.d/15680.bugfix Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com> * add attribution * Update changelog. --------- Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
* Merge branch 'release-v1.86' into developMathieu Velten2023-06-141-19/+1
|\
| * Revert "Allow for the configuration of max request retries and min/max retry ↵Mathieu Velten2023-06-141-19/+1
| | | | | | | | | | | | delays in the matrix federation client (#12504)" This reverts commit d84e66144dc12dacf71c987a2ba802dd59c0b68e.
* | Prefill events after invalidate not before when persisting events (#15758)Jason Little2023-06-141-0/+49
| | | | | | | | | | Fixes #15757
* | Replace `EventContext` fields `prev_group` and `delta_ids` with field ↵Shay2023-06-133-6/+13
|/ | | | `state_group_deltas` (#15233)
* Allow for the configuration of max request retries and min/max retry delays ↵Shay2023-06-091-1/+19
| | | | | | | in the matrix federation client (#12504) Co-authored-by: Mathieu Velten <mathieuv@matrix.org> Co-authored-by: Erik Johnston <erik@matrix.org>
* Clear event caches when we purge history (#15609)Erik Johnston2023-06-083-8/+5
| | | | | | | This should help a little with #13476 --------- Co-authored-by: Patrick Cloke <patrickc@matrix.org>
* Check required power levels earlier in createRoom handler. (#15695)Grant McLean2023-06-071-0/+37
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Check required power levels earlier in createRoom handler. - If a server was configured to reject the creation of rooms with E2EE enabled (by specifying an unattainably high power level for "m.room.encryption" in default_power_level_content_override), the 403 error was not being triggered until after the room was created and before the "m.room.power_levels" was sent. This allowed a user to access the partially-configured room and complete the setup of E2EE and power levels manually. - This change causes the power level overrides to be checked earlier and the request to be rejected before the user gains access to the room. - A new `_validate_room_config` method is added to contain checks that should be run before a room is created. - The new test case confirms that a user request is rejected by the new validation method. Signed-off-by: Grant McLean <grant@catalyst.net.nz> * Add a changelog file. * Formatting fix for black. * Remove unneeded line from test. --------- Signed-off-by: Grant McLean <grant@catalyst.net.nz>
* Trace functions which return `Awaitable` (#15650)Eric Eastwood2023-06-061-11/+32
|
* Stabilize support for MSC3952: Intentional mentions. (#15520)Patrick Cloke2023-06-061-21/+13
|
* `N + 3`: Read from column `full_user_id` rather than `user_id` of tables ↵Shay2023-06-025-54/+26
| | | | `profiles` and `user_filters` (#15649)
* Add a catch-all * to the supported relation types when redacting (#15705)Mathieu Velten2023-06-021-3/+101
| | | This is an update to MSC3912 implementation
* Log when events are (unexpectedly) filtered out of responses in tests (#14213)Eric Eastwood2023-06-011-0/+12
| | | | | See https://github.com/matrix-org/synapse/pull/14095#discussion_r990335492 This is useful because when see that a relevant event is an `outlier` or `soft-failed`, then that's a good unexpected indicator explaining why it's not showing up. `filter_events_for_client` is used in `/sync`, `/messages`, `/context` which are all common end-to-end assertion touch points (also notifications, relations).
* Implement stable support for MSC3882 to allow an existing device/session to ↵Hugh Nimmo-Smith2023-06-014-20/+106
| | | | | | | | generate a login token for use on a new device/session (#15388) Implements stable support for MSC3882; this involves updating Synapse's support to match the MSC / the spec says. Continue to support the unstable version to allow clients to transition.
* Add stubs package for lxml. (#15697)Patrick Cloke2023-05-314-4/+20
| | | | | | | The stubs have some issues so this has some generous cast and ignores in it, but it is better than not having stubs. Note that confusing that Element is a function which creates _Element instances (and similarly for Comment).
* Add stricter mypy options (#15694)Patrick Cloke2023-05-311-1/+1
| | | | Enable warn_unused_configs, strict_concatenate, disallow_subclassing_any, and disallow_incomplete_defs.
* Enforce that an admin token also has the basic Matrix API scopeQuentin Gliech2023-05-301-1/+25
|
* Make the config tests spawn the homeserver only when neededQuentin Gliech2023-05-301-194/+154
|
* Reject tokens with multiple device scopesQuentin Gliech2023-05-301-1/+28
|
* Handle errors when introspecting tokensQuentin Gliech2023-05-302-2/+37
| | | | | This returns a proper 503 when the introspection endpoint is not working for some reason, which should avoid logging out clients in those cases.
* Disable incompatible Admin API endpointsQuentin Gliech2023-05-301-0/+19
|
* Make AS tokens work & allow ASes to /registerQuentin Gliech2023-05-301-2/+2
|
* Tests for JWKS endpointHugh Nimmo-Smith2023-05-302-11/+212
|
* Refactor config to be an experimental featureHugh Nimmo-Smith2023-05-303-13/+221
| | | | Also enforce you can't combine it with incompatible config options
* Test MSC2965 implementation: well-known discovery documentHugh Nimmo-Smith2023-05-301-0/+38
|
* Disable account related endpoints when using OAuth delegationQuentin Gliech2023-05-301-3/+177
|
* Actually enforce guest + return www-authenticate headerHugh Nimmo-Smith2023-05-301-3/+40
|
* Initial tests for OAuth delegationHugh Nimmo-Smith2023-05-301-0/+345
|
* Save the scopes in the requesterQuentin Gliech2023-05-301-0/+2
|
* Make the api.auth.Auth a ProtocolQuentin Gliech2023-05-303-6/+6
|
* Merge branch 'master' into developOlivier Wilkinson (reivilibre)2023-05-265-4/+44
|\
| * Fix a bug introduced in Synapse v1.84.0 where workers do not start up when ↵reivilibre2023-05-265-4/+44
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | no `instance_map` was provided. (#15672) * Fix #15669: always populate instance map even if it was empty * Fix some tests * Fix more tests * Newsfile Signed-off-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org> * CI fix: don't forget to update apt repository sources before installing olddeps deps * Add test testing the backwards compatibility --------- Signed-off-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org>
* | Process previously failed backfill events in the background (#15585)Eric Eastwood2023-05-242-0/+132
| | | | | | | | | | | | | | | | | | Process previously failed backfill events in the background because they are bound to fail again and we don't need to waste time holding up the request for something that is bound to fail again. Fix https://github.com/matrix-org/synapse/issues/13623 Follow-up to https://github.com/matrix-org/synapse/issues/13621 and https://github.com/matrix-org/synapse/issues/13622 Part of making `/messages` faster: https://github.com/matrix-org/synapse/issues/13356
* | Add requesting user id parameter to key claim methods in ↵Shay2023-05-241-1/+15
| | | | | | | | `TransportLayerClient` (#15663)
* | Add type hints to test_descriptors. (#15659)Patrick Cloke2023-05-241-93/+104
| | | | | | Require type hints in test_descriptors and add missing ones.
* | Improve type hints for cached decorator. (#15658)Patrick Cloke2023-05-242-59/+34
| | | | | | | | | | The cached decorators always return a Deferred, which was not properly propagated. It was close enough when wrapping coroutines, but failed if a bare function was wrapped.
* | Consolidate logic to check for deactivated users. (#15634)Patrick Cloke2023-05-231-38/+21
| | | | | | | | | | | | | | This moves the deactivated user check to the method which all login types call. Additionally updates the application service tests to be more realistic by removing invalid tests and fixing server names.
* | Remove duplicate timestamp from test logs (`_trial_temp/test.log`) (#15636)Eric Eastwood2023-05-221-4/+3
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Fix https://github.com/matrix-org/synapse/issues/15618 ### Before ``` 2023-05-17 22:51:36-0500 [-] 2023-05-17 22:51:36,889 - synapse.server - 338 - INFO - sentinel - Finished setting up. ``` ### After ``` 2023-05-19 18:16:20-0500 [-] synapse.server - 338 - INFO - sentinel - Finished setting up. ``` ### Dev notes The `Twisted.Logger` controls the `2023-05-19 18:16:20-0500 [-]` prefix, see : [`twisted/twisted` -> `src/twisted/logger/_format.py#L362-L374`](https://github.com/twisted/twisted/blob/34b161e66bc7c9f9efbb95e82c770a863933e498/src/twisted/logger/_format.py#L362-L374) And we delegate our logs to the Twisted Logger for the tests which puts it in `_trial_temp/test.log`
* | Properly parse event_fields in filters (#15607)Patrick Cloke2023-05-222-6/+39
| | | | | | | | | | | | | | | | The event_fields property in filters should use the proper escape rules, namely backslashes can be escaped with an additional backslash. This adds tests (adapted from matrix-js-sdk) and implements the logic to properly split the event_fields strings.
* | Limit the size of the `HomeServerConfig` cache in trial test runs (#15646)Sean Quah2023-05-221-16/+7
| | | | | | | | | | | | | | | | | | | | | | | | ...to try to control memory usage. `HomeServerConfig`s hold on to many Jinja2 objects, which come out to over 0.5 MiB per config. Over the course of a full test run, the cache grows to ~360 entries. Limit it to 8 entries. Part of #15622. Signed-off-by: Sean Quah <seanq@matrix.org>
* | Remove old R30 because R30v2 supercedes it (#10428)reivilibre2023-05-191-154/+0
| | | | | | | | | | R30v2 has been out since 2021-07-19 (https://github.com/matrix-org/synapse/pull/10332) and we started collecting stats on 2021-08-16. Since it's been over a year now (almost 2 years), this is enough grace period for us to now rip it out.
* | Rename blacklist/whitelist internally. (#15620)Patrick Cloke2023-05-1910-76/+70
| | | | | | | | Avoid renaming configuration settings for now and rename internal code to use blocklist and allowlist instead.
* | Do not allow deactivated users to login with JWT. (#15624)Patrick Cloke2023-05-191-2/+18
| | | | | | | | | | To improve the organization of this code it moves the JWT login checks to a separate handler and then fixes the bug (and a deprecation warning).
* | Remove experimental configuration flags & unstable values for faster joins ↵Patrick Cloke2023-05-191-32/+3
| | | | | | | | | | | | | | (#15625) Synapse will no longer send (or respond to) the unstable flags for faster joins. These were only available behind a configuration flag and handled in parallel with the stable flags.
* | Fix `HomeServer`s leaking during `trial` test runs (#15630)Sean Quah2023-05-191-2/+9
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This change fixes two memory leaks during `trial` test runs. Garbage collection is disabled during each test case and a gen-0 GC is run at the end of each test. However, when the gen-0 GC is run, the `TestCase` object usually still holds references to the `HomeServer` used during the test. As a result, the `HomeServer` gets promoted to gen-1 and then never garbage collected. Fix this by periodically running full GCs. Additionally, fix `HomeServer`s leaking after tests that touch inbound federation due to `FederationRateLimiter`s adding themselves to a global set, by turning the set into a `WeakSet`. Resolves #15622. Signed-off-by: Sean Quah <seanq@matrix.org>
* | Handle missing previous read marker event. (#15464)Nick Mills-Barrett2023-05-181-0/+147
| | | | | | | | | | If the previous read marker is pointing to an event that no longer exists (e.g. due to retention) then assume that the newly given read marker is newer.
* | Update Mutual Rooms (MSC2666) implementation (#15621)Jonathan de Jong2023-05-181-2/+4
| | | | | | | | | | | | | | | | To track changes in MSC2666: - The change from `/mutual_rooms/{user_id}` to `/mutual_rooms?user_id={user_id}`. - The addition of `next_batch_token` (and logic). - Unstable flag now being `uk.half-shot.msc2666.query_mutual_rooms`. - The error code when your own user is requested.
* | Apply url_preview_url_blacklist to oEmbed and pre-cached images (#15601)Patrick Cloke2023-05-162-4/+303
| | | | | | | | | | | | | | | | There are two situations which were previously not properly checked: 1. If the requested URL was replaced with an oEmbed URL, then the oEmbed URL was not checked against url_preview_url_blacklist. 2. Follow-up URLs (either via autodiscovery of oEmbed or to pre-cache images) were not checked against url_preview_url_blacklist.
* | Update code to refer to "workers". (#15606)Patrick Cloke2023-05-165-28/+15
| | | | | | | | A bunch of comments and variables are out of date and use obsolete terms.
* | Add not null constraint to column `full_user_id` of tables `profiles` and ↵Shay2023-05-162-0/+157
| | | | | | | | `user_filters` (#15537)
* | Fix subscriptable type usage in Python <3.9 (#15604)Eric Eastwood2023-05-161-1/+1
|/ | | | | | | | | | | | Fix the following `mypy` errors when running `mypy` with Python 3.7: ``` synapse/storage/controllers/stats.py:58: error: "Counter" is not subscriptable, use "typing.Counter" instead [misc] tests/test_state.py:267: error: "dict" is not subscriptable, use "typing.Dict" instead [misc] ``` Part of https://github.com/matrix-org/synapse/issues/15603 In Python 3.9, `typing` is deprecated and the types are subscriptable (generics) by default, https://peps.python.org/pep-0585/#implementation
* Revert "Fix subscriptable dict type"Eric Eastwood2023-05-151-1/+1
| | | | This reverts commit 55b08534a412f462251753f67308405ca4d02ebe.
* Fix subscriptable dict typeEric Eastwood2023-05-151-1/+1
| | | | | | | | | | Fix: ``` tests/test_state.py:267: error: "dict" is not subscriptable, use "typing.Dict" instead [misc] ``` In Python 3.9, `typing` is deprecated and the types are subscriptable (generics) by default, https://peps.python.org/pep-0585/#implementation
* Implement MSC3821 to update redaction rules (`third_party_invite.signed`) ↵Patrick Cloke2023-05-151-1/+74
| | | | | | (#15563) Updates the redaction rules to protect enough information that the event can still be properly verified.
* Implement MSC3389 to protect relations from redaction. (#15565)Patrick Cloke2023-05-151-0/+90
| | | | | | MSC3389 proposes protecting the relation type & parent event ID from redaction. This keeps the relation information intact after redaction which helps with some UX flaws (e.g. deleting an event causes it to no longer be in a thread, which is confusing).
* Remove `worker_replication_*` settings (#15491)Jason Little2023-05-115-10/+5
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Add master to the instance_map as part of Complement, have ReplicationEndpoint look at instance_map for master. * Fix typo in drive by. * Remove unnecessary worker_replication_* bits from unit tests and add master to instance_map(hopefully in the right place) * Several updates: 1. Switch from master to main for naming the main process in the instance_map. Add useful constants for easier adjustment of names in the future. 2. Add backwards compatibility for worker_replication_* to allow time to transition to new style. Make sure to prioritize declaring main directly on the instance_map. 3. Clean up old comments/commented out code. 4. Adjust unit tests to match with new code. 5. Adjust Complement setup infrastructure to only add main to the instance_map if workers are used and remove now unused options from the worker.yaml template. * Initial Docs upload * Changelog * Missed some commented out code that can go now * Remove TODO comment that no longer holds true. * Fix links in docs * More docs * Remove debug logging * Apply suggestions from code review Co-authored-by: reivilibre <olivier@librepush.net> * Apply suggestions from code review Co-authored-by: reivilibre <olivier@librepush.net> * Update version to latest, include completeish before/after examples in upgrade notes. * Fix up and docs too --------- Co-authored-by: reivilibre <olivier@librepush.net>
* HTTP Replication Client (#15470)Jason Little2023-05-091-0/+1
| | | | | | Separate out a HTTP client for replication in preparation for also supporting using UNIX sockets. The major difference from the base class is that this does not use treq to handle HTTP requests.
* Add config option to prevent media downloads from listed domains. (#15197)Travis Ralston2023-05-091-0/+139
| | | | | | | This stops media (and thumbnails) from being accessed from the listed domains. It does not delete any already locally cached media, but will prevent accessing it. Note that admin APIs are unaffected by this change.
* Check appservices for devices during a /user/devices query. (#15539)Patrick Cloke2023-05-051-1/+134
| | | | | | | | MSC3984 proxies /keys/query requests to appservices, but servers will can also requests devices / keys from the /user/devices endpoint. The formats are close enough that we can "proxy" that /user/devices to appservices (by calling /keys/query) and then change the format of the returned data before returning it over federation.
* Implement MSC4009 to widen the allowed Matrix ID grammar (#15536)Patrick Cloke2023-05-051-0/+13
| | | | | Behind a configuration flag this adds + to the list of allowed characters in Matrix IDs. The main feature this enables is using full E.164 phone numbers as Matrix IDs.
* Factor out an `is_mine_server_name` method (#15542)Sean Quah2023-05-051-2/+14
| | | | | | | | | | | | Add an `is_mine_server_name` method, similar to `is_mine_id`. Ideally we would use this consistently, instead of sometimes comparing against `hs.hostname` and other times reaching into `hs.config.server.server_name`. Also fix a bug in the tests where `hs.hostname` would sometimes differ from `hs.config.server.server_name`. Signed-off-by: Sean Quah <seanq@matrix.org>
* Move ThirdPartyEventRules into module_api/callbacks (#15535)Andrew Morgan2023-05-042-22/+38
|
* Merge branch 'release-v1.83' into developSean Quah2023-05-032-4/+4
|\
| * Revert "Reduce the size of the HTTP connection pool for non-pushers" (#15530) v1.83.0rc1Sean Quah2023-05-032-4/+4
| | | | | | | | | | | | | | | | | | #15514 introduced a regression where Synapse would encounter `PartialDownloadError`s when fetching OpenID metadata for certain providers on startup. Due to #8088, this prevents Synapse from starting entirely. Revert the change while we decide what to do about the regression.
* | Add config option to forget rooms automatically when users leave them (#15224)Sean Quah2023-05-031-0/+11
| | | | | | | | | | This is largely based off the stats and user directory updater code. Signed-off-by: Sean Quah <seanq@matrix.org>
* | Remove references to supporting per-user flag for msc2654 (#15522)Shay2023-05-021-6/+2
| |
* | Allow adding random delay to push (#15516)Erik Johnston2023-05-021-0/+37
|/ | | This is to discourage timing based profiling on the push gateways.
* Reduce the size of the HTTP connection pool for non-pushers. (#15514)Patrick Cloke2023-05-022-4/+4
| | | | | | | | | | | | | | Pushers tend to make many connections to the same HTTP host (e.g. a new event comes in, causes events to be pushed, and then the homeserver connects to the same host many times). Due to this the per-host HTTP connection pool size was increased, but this does not make sense for other SimpleHttpClients. Add a parameter for the connection pool and override it for pushers (making a separate SimpleHttpClient for pushers with the increased configuration). This returns the HTTP connection pool settings to the default Twisted ones for non-pusher HTTP clients.
* Initial implementation of MSC3981: recursive relations API (#15315)Patrick Cloke2023-05-021-0/+120
| | | | | | | | | | | Adds an optional keyword argument to the /relations API which will recurse a limited number of event relationships. This will cause the API to return not just the events related to the parent event, but also events related to those related to the parent event, etc. This is disabled by default behind an experimental configuration flag and is currently implemented using prefixed parameters.
* Add an admin API endpoint to support per-user feature flags (#15344)Shay2023-04-281-0/+127
|
* Add support for claiming multiple OTKs at once. (#15468)Patrick Cloke2023-04-272-24/+19
| | | | | | | MSC3983 provides a way to request multiple OTKs at once from appservices, this extends this concept to the Client-Server API. Note that this will likely be spit out into a separate MSC, but is currently part of MSC3983.
* Add column `full_user_id` to tables `profiles` and `user_filters`. (#15458)Shay2023-04-266-44/+56
|
* Add unstable /keys/claim endpoint which always returns fallback keys. (#15462)Patrick Cloke2023-04-251-6/+235
| | | | | | | | | | | | | It can be useful to always return the fallback key when attempting to claim keys. This adds an unstable endpoint for `/keys/claim` which always returns fallback keys in addition to one-time-keys. The fallback key(s) are not marked as "used" unless there are no corresponding OTKs. This is currently defined in MSC3983 (although likely to be split out to a separate MSC). The endpoint shape may change or be requested differently (i.e. a keyword parameter on the current endpoint), but the core logic should be reasonable.
* Finish type hints for federation client HTTP code. (#15465)Patrick Cloke2023-04-242-8/+8
|
* Remove legacy code of single user device resync api (#15418)Alok Kumar Singh2023-04-211-1/+3
| | | | | * Removed single-user resync usage and updated it to use multi-user counterpart Signed-off-by: Alok Kumar Singh alokaks601@gmail.com
* Modify StoreKeyFetcher to read from server_keys_json. (#15417)Patrick Cloke2023-04-204-48/+58
| | | | | | | | | | | | | | | | | | | | | | | | | | | Before this change: * `PerspectivesKeyFetcher` and `ServerKeyFetcher` write to `server_keys_json`. * `PerspectivesKeyFetcher` also writes to `server_signature_keys`. * `StoreKeyFetcher` reads from `server_signature_keys`. After this change: * `PerspectivesKeyFetcher` and `ServerKeyFetcher` write to `server_keys_json`. * `PerspectivesKeyFetcher` also writes to `server_signature_keys`. * `StoreKeyFetcher` reads from `server_keys_json`. This results in `StoreKeyFetcher` now using the results from `ServerKeyFetcher` in addition to those from `PerspectivesKeyFetcher`, i.e. keys which are directly fetched from a server will now be pulled from the database instead of refetched. An additional minor change is included to avoid creating a `PerspectivesKeyFetcher` (and checking it) if no `trusted_key_servers` are configured. The overall impact of this should be better usage of cached results: * If a server has no trusted key servers configured then it should reduce how often keys are fetched. * if a server's trusted key server does not have a requested server's keys cached then it should reduce how often keys are directly fetched.
* Speedup tests by caching HomeServerConfig instances (#15284)Val Lorentz2023-04-181-2/+60
| | | | | | | | | | | | | | | | | | | These two lines: ``` config_obj = HomeServerConfig() config_obj.parse_config_dict(config, "", "") ``` are called many times with the exact same value for `config`. As the test suite is CPU-bound and non-negligeably time is spent in `parse_config_dict`, this saves ~5% on the overall runtime of the Trial test suite (tested with both `-j2` and `-j12` on a 12t CPU). This is sadly rather limited, as the cache cannot be shared between processes (it contains at least jinja2.Template and RLock objects which aren't pickleable), and Trial tends to run close tests in different processes.
* Move Spam Checker callbacks to a dedicated file (#15453)Andrew Morgan2023-04-184-10/+22
|
* Delete pushers after calling on_logged_out module hook on device delete (#15410)Mathieu Velten2023-04-141-1/+50
|
* Convert async to normal tests in `TestSSOHandler` (#15433)Dirk Klimpel2023-04-141-4/+4
| | | | | * Convert async to normal tests in `TestSSOHandler` * newsfile
* Improve robustness when handling a perspective key response by deduplicating ↵reivilibre2023-04-133-21/+17
| | | | | | | | | | | | | | | received server keys. (#15423) * Change `store_server_verify_keys` to take a `Mapping[(str, str), FKR]` This is because we already can't handle duplicate keys — leads to cardinality violation * Newsfile Signed-off-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org> --------- Signed-off-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org>
* Implement MSC2174: move redacts to a content property. (#15395)Patrick Cloke2023-04-132-4/+47
| | | | | | | This moves `redacts` from being a top-level property to a `content` property in a new room version. MSC2176 (which was previously implemented) states to not `redact` this property.
* Throw if the appservice config list is the wrong type (#15425)Will Hunt2023-04-121-0/+40
| | | | | | | | | | | * raise a ConfigError on an invalid app_service_config_files * changelog * Move config check to read_config * Add test * Ensure list also contains strings
* Implement MSC3989 to redact the origin field. (#15393)Patrick Cloke2023-04-051-0/+7
| | | | This will be done in a future room version, for now an unstable room version is added which redacts the origin field.
* Delete server-side backup keys when deactivating an account. (#15181)Shay2023-04-041-0/+157
|
* Call appservices on modern paths, falling back to legacy paths. (#15317)Patrick Cloke2023-04-031-2/+55
| | | | | This uses the specced /_matrix/app/v1/... paths instead of the "legacy" paths. If the homeserver receives an error it will retry using the legacy path.
* Revert pruning of old devices (#15360)Erik Johnston2023-03-314-52/+3
| | | | | | | | | * Revert "Fix registering a device on an account with lots of devices (#15348)" This reverts commit f0d8f66eaaacfa75bed65bc5d0c602fbc5339c85. * Revert "Delete stale non-e2e devices for users, take 3 (#15183)" This reverts commit 78cdb72cd6b0e007c314d9fed9f629dfc5b937a6.
* to_device updates could be dropped when consuming the replication stream ↵Mathieu Velten2023-03-303-5/+93
| | | | | (#15349) Co-authored-by: reivilibre <oliverw@matrix.org>
* Speed up SQLite unit test CI (#15334)Erik Johnston2023-03-302-3/+36
| | | Tests now take 40% of the time.
* Implement MSC3984 to proxy /keys/query requests to appservices. (#15321)Patrick Cloke2023-03-301-1/+120
| | | | | If enabled, for users which are exclusively owned by an application service then the appservice will be queried for devices in addition to any information stored in the Synapse database.
* Fix spinloop during partial state sync when a prev event is in backoff (#15351)Sean Quah2023-03-301-4/+9
| | | | | | | | Previously, we would spin in a tight loop until `update_state_for_partial_state_event` stopped raising `FederationPullAttemptBackoffError`s. Replace the spinloop with a wait until the backoff period has expired. Signed-off-by: Sean Quah <seanq@matrix.org>
* Add the ability to enable/disable registrations when in the OIDC flow (#14978)Warren Bailey2023-03-301-1/+16
| | | Signed-off-by: Warren Bailey <warren@warrenbailey.net>
* Fix registering a device on an account with lots of devices (#15348)Erik Johnston2023-03-291-0/+47
| | | Fixes up #15183
* Delete stale non-e2e devices for users, take 3 (#15183)Erik Johnston2023-03-293-3/+5
| | | | | This should help reduce the number of devices e.g. simple bots the repeatedly login rack up. We only delete non-e2e devices as they should be safe to delete, whereas if we delete e2e devices for a user we may accidentally break their ability to receive e2e keys for a message.
* Implement MSC3983 to proxy /keys/claim queries to appservices. (#15314)Patrick Cloke2023-03-282-1/+134
| | | | | | Experimental support for MSC3983 is behind a configuration flag. If enabled, for users which are exclusively owned by an application service then the appservice will be queried for one-time keys *if* there are none uploaded to Synapse.
* Bump ruff from 0.0.252 to 0.0.259 (#15328)dependabot[bot]2023-03-282-3/+9
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Bump ruff from 0.0.252 to 0.0.259 Bumps [ruff](https://github.com/charliermarsh/ruff) from 0.0.252 to 0.0.259. - [Release notes](https://github.com/charliermarsh/ruff/releases) - [Changelog](https://github.com/charliermarsh/ruff/blob/main/BREAKING_CHANGES.md) - [Commits](https://github.com/charliermarsh/ruff/compare/v0.0.252...v0.0.259) --- updated-dependencies: - dependency-name: ruff dependency-type: direct:development update-type: version-update:semver-patch ... Signed-off-by: dependabot[bot] <support@github.com> * Fix new warnings * Mypy * Newsfile --------- Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Erik Johnston <erik@matrix.org>
* Make cleaning up pushers depend on the device_id instead of the token_id ↵Quentin Gliech2023-03-244-32/+28
| | | | | | | | | | | | | | (#15280) This makes it so that we rely on the `device_id` to delete pushers on logout, instead of relying on the `access_token_id`. This ensures we're not removing pushers on token refresh, and prepares for a world without access token IDs (also known as the OIDC). This actually runs the `set_device_id_for_pushers` background update, which was forgotten in #13831. Note that for backwards compatibility it still deletes pushers based on the `access_token` until the background update finishes.
* Reject mentions on the C-S API which are invalid. (#15311)Patrick Cloke2023-03-241-38/+56
| | | | | | Invalid mentions data received over the Client-Server API should be rejected with a 400 error. This will hopefully stop clients from sending invalid data, although does not help with data received over federation.
* Use immutabledict instead of frozendict (#15113)David Robertson2023-03-225-41/+43
| | | | | | | | | | Additionally: * Consistently use `freeze()` in test --------- Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com> Co-authored-by: 6543 <6543@obermui.de>
* Make `POST /_matrix/client/v3/rooms/{roomId}/report/{eventId}` endpoint ↵Andrew Morgan2023-03-211-0/+37
| | | | return 404 if event exists, but the user lacks access (#15300)
* Make `EventHandler.get_event` return `None` when the requested event is not ↵Andrew Morgan2023-03-211-0/+5
| | | | found (#15298)
* Fix error when sending message into deleted room. (#15235)Erik Johnston2023-03-211-0/+15
| | | | | | | | When a room is deleted in Synapse we remove the event forward extremities in the room, so if (say a bot) tries to send a message into the room we error out due to not being able to calculate prev events for the new event *before* we check if the sender is in the room. Fixes #8094
generated by cgit-pink 1.4.1 (git 2.36.1) at 2024-11-15 12:48:50 +0000