summary refs log tree commit diff
path: root/tests/storage (follow)
Commit message (Collapse)AuthorAgeFilesLines
* Correctly mention previous copyright (#16820)Erik Johnston2024-01-2344-0/+45
| | | | | During the migration the automated script to update the copyright headers accidentally got rid of some of the existing copyright lines. Reinstate them.
* Port `EventInternalMetadata` class to Rust (#16782)Erik Johnston2024-01-083-5/+7
| | | | | | | | | | | | | There are a couple of things we need to be careful of here: 1. The current python code does no validation when loading from the DB, so we need to be careful to ignore such errors (at least on jki.re there are some old events with internal metadata fields of the wrong type). 2. We want to be memory efficient, as we often have many hundreds of thousands of events in the cache at a time. --------- Co-authored-by: Quentin Gliech <quenting@element.io>
* Update license headersPatrick Cloke2023-11-2148-483/+768
|
* Revert "Add a Postgres `REPLICA IDENTITY` to tables that do not have an ↵Erik Johnston2023-11-161-84/+1
| | | | | implicit one. This should allow use of Postgres logical replication. (#16456)" (#16651) This reverts commit 69afe3f7a0d89f3422ddbd3aa16bc9bbc01056eb.
* Fix test not detecting tables with missing primary keys and missing replica ↵reivilibre2023-11-161-3/+4
| | | | | | | | | | | | | | | identities, then add more replica identities. (#16647) * Fix the CI query that did not detect all cases of missing primary keys * Add more missing REPLICA IDENTITY entries * Newsfile Signed-off-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org> --------- Signed-off-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org>
* Add an Admin API to temporarily grant the ability to update an existing ↵David Robertson2023-11-151-0/+121
| | | | cross-signing key without UIA (#16634)
* Use full GitHub links instead of bare issue numbers. (#16637)Patrick Cloke2023-11-154-13/+14
|
* Add a Postgres `REPLICA IDENTITY` to tables that do not have an implicit ↵reivilibre2023-11-131-1/+83
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | one. This should allow use of Postgres logical replication. (#16456) * Add Postgres replica identities to tables that don't have an implicit one Fixes #16224 * Newsfile Signed-off-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org> * Move the delta to version 83 as we missed the boat for 82 * Add a test that all tables have a REPLICA IDENTITY * Extend the test to include when indices are deleted * isort * black * Fully qualify `oid` as it is a 'hidden attribute' in Postgres 11 * Update tests/storage/test_database.py Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com> * Add missed tables --------- Signed-off-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org> Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
* Convert simple_select_one_txn and simple_select_one to return tuples. (#16612)Patrick Cloke2023-11-092-12/+5
|
* Bulk-invalidate e2e cached queries after claiming keys (#16613)David Robertson2023-11-091-0/+117
| | | | Co-authored-by: Patrick Cloke <patrickc@matrix.org>
* Avoid updating the same rows multiple times with simple_update_many_txn. ↵Patrick Cloke2023-11-071-2/+2
| | | | | | (#16609) simple_update_many_txn had a bug in it which would cause each update to be applied twice.
* Avoid executing no-op queries. (#16583)Patrick Cloke2023-11-071-20/+5
| | | | | | If simple_{insert,upsert,update}_many_txn is called without any data to modify then return instead of executing the query. This matches the behavior of simple_{select,delete}_many_txn.
* More tests for the simple_* methods. (#16596)Patrick Cloke2023-11-071-18/+628
| | | | Expand tests for the simple_* database methods, additionally test against both PostgreSQL and SQLite variants.
* Remove remaining usage of cursor_to_dict. (#16564)Patrick Cloke2023-10-312-10/+5
|
* Remove more usages of cursor_to_dict. (#16551)Patrick Cloke2023-10-263-10/+10
| | | Mostly to improve type safety.
* Convert simple_select_list and simple_select_list_txn to return lists of ↵Patrick Cloke2023-10-268-235/+253
| | | | | tuples (#16505) This should use fewer allocations and improves type hints.
* Fix bug where a new writer advances their token too quickly (#16473)Erik Johnston2023-10-231-12/+124
| | | | | | | | | | | | | | | | | | | * Fix bug where a new writer advances their token too quickly When starting a new writer (for e.g. persisting events), the `MultiWriterIdGenerator` doesn't have a minimum token for it as there are no rows matching that new writer in the DB. This results in the the first stream ID it acquired being announced as persisted *before* it actually finishes persisting, if another writer gets and persists a subsequent stream ID. This is due to the logic of setting the minimum persisted position to the minimum known position of across all writers, and the new writer starts off not being considered. * Fix sending out POSITIONs when our token advances without update Broke in #14820 * For replication HTTP requests, only wait for minimal position
* Convert DeviceLastConnectionInfo to attrs. (#16507)Patrick Cloke2023-10-171-67/+70
| | | To improve type safety & memory usage.
* Convert simple_select_many_batch, simple_select_many_txn to tuples. (#16444)Patrick Cloke2023-10-111-26/+38
|
* Return an immutable value from get_latest_event_ids_in_room. (#16326)Patrick Cloke2023-09-181-7/+7
|
* Refactor `get_user_by_id` (#16316)Erik Johnston2023-09-141-24/+24
|
* Refactor storing of server keys (#16261)Erik Johnston2023-09-121-137/+0
|
* Update ruff config (#16283)Patrick Cloke2023-09-086-12/+12
| | | Enable additional checks & clean-up unneeded configuration.
* Add last_seen_ts to the admin users API (#16218)Mathieu Velten2023-09-041-0/+1
|
* Log the details of background update failures (#16212)David Robertson2023-09-011-1/+23
|
* Fix rare bug that broke looping calls (#16210)Erik Johnston2023-08-301-0/+2
| | | | | | | | | | * Fix rare bug that broke looping calls We can't interact with the reactor from the main thread via looping call. Introduced in v1.90.0 / #15791. * Newsfile
* Bump mypy-zope & mypy. (#16188)Patrick Cloke2023-08-292-7/+7
|
* Prepare unit tests for Python 3.12 (#16099)V024602023-08-254-33/+37
|
* Replace simple_async_mock with AsyncMock (#16180)Patrick Cloke2023-08-251-3/+2
| | | | Python 3.8 has a native AsyncMock, use it instead of a custom implementation.
* Replace make_awaitable with AsyncMock (#16179)Patrick Cloke2023-08-245-30/+23
| | | | Python 3.8 provides a native AsyncMock, we can replace the homegrown version we have.
* Filter out unwanted user_agents from udv. (#16124)Neil Johnson2023-08-231-0/+65
|
* Speed up state res in rare case we don't have all events (#16116)Erik Johnston2023-08-181-48/+193
| | | | | | | If we don't have all the auth events in a room then not all state events will have a chain cover index. Even so, we can still use the chain cover index on the events that do have it, rather than bailing and using the slower functions. This situation should not arise for newly persisted rooms, as we check we have the full auth chain for each event, but can happen for existing rooms. c.f. #15245
* Fix database performance of read/write worker locks (#16061)Erik Johnston2023-08-171-3/+4
| | | | | | | We were seeing serialization errors when taking out multiple read locks. The transactions were retried, so isn't causing any failures. Introduced in #15782.
* Run pyupgrade for python 3.7 & 3.8. (#16110)Patrick Cloke2023-08-153-8/+8
|
* Implements admin API to lock an user (MSC3939) (#15870)Mathieu Velten2023-08-101-0/+1
|
* Allow config of the backoff algorithm for the federation client. (#15754)Mathieu Velten2023-08-031-2/+7
| | | | | | | | | | | Adds three new configuration variables: * destination_min_retry_interval is identical to before (10mn). * destination_retry_multiplier is now 2 instead of 5, the maximum value will be reached slower. * destination_max_retry_interval is one day instead of (essentially) infinity. Capping this will cause destinations to continue to be retried sometimes instead of being lost forever. The previous value was 2 ^ 62 milliseconds.
* Add ability to wait for locks and add locks to purge history / room deletion ↵Erik Johnston2023-07-311-0/+52
| | | | | (#15791) c.f. #13476
* Merge branch 'master' into developOlivier Wilkinson (reivilibre)2023-07-182-0/+157
|\
| * Revert "Stop writing to column `user_id` of tables `profiles` and ↵Shay2023-07-182-0/+157
| | | | | | | | | | | | | | | | | | `user_filters`. (#15953) * Revert "Stop writing to column `user_id` of tables `profiles` and `user_filters` (#15787)" This reverts commit f25b0f88081bb436bef914983cff7087b54eba5f. * newsfragement
* | Re-introduce the outbound federation proxy (#15913)Eric Eastwood2023-07-183-5/+3
|/ | | | | Allow configuring the set of workers to proxy outbound federation traffic through (`outbound_federation_restricted_to`). This is useful when you have a worker setup with `federation_sender` instances responsible for sending outbound federation requests and want to make sure *all* outbound federation traffic goes through those instances. Before this change, the generic workers would still contact federation themselves for things like profile lookups, backfill, etc. This PR allows you to set more strict access controls/firewall for all workers and only allow the `federation_sender`'s to contact the outside world.
* Revert "Federation outbound proxy" (#15910)Eric Eastwood2023-07-103-3/+5
| | | | | | Revert "Federation outbound proxy (#15773)" This reverts commit b07b14b494ae1dd564b4c44f844c9a9545b3d08a.
* Fix downgrading to previous version of Synapse (#15907)Erik Johnston2023-07-101-2/+6
| | | We do this by marking the constraint as deferrable.
* Stop writing to column `user_id` of tables `profiles` and `user_filters` ↵Shay2023-07-072-157/+0
| | | | (#15787)
* Federation outbound proxy (#15773)Eric Eastwood2023-07-053-5/+3
| | | | | | | Allow configuring the set of workers to proxy outbound federation traffic through (`outbound_federation_restricted_to`). This is useful when you have a worker setup with `federation_sender` instances responsible for sending outbound federation requests and want to make sure *all* outbound federation traffic goes through those instances. Before this change, the generic workers would still contact federation themselves for things like profile lookups, backfill, etc. This PR allows you to set more strict access controls/firewall for all workers and only allow the `federation_sender`'s to contact the outside world. The original code is from @erikjohnston's branches which I've gotten in-shape to merge.
* Add basic read/write lock (#15782)Erik Johnston2023-07-051-1/+282
|
* Add foreign key constraint to `event_forward_extremities`. (#15751)Erik Johnston2023-07-052-4/+258
|
* Remove experimental MSC2716 implementation to incrementally import history ↵Eric Eastwood2023-06-161-211/+0
| | | | | | | | | | | | into existing rooms (#15748) Context for why we're removing the implementation: - https://github.com/matrix-org/matrix-spec-proposals/pull/2716#issuecomment-1487441010 - https://github.com/matrix-org/matrix-spec-proposals/pull/2716#issuecomment-1504262734 Anyone wanting to continue MSC2716, should also address these leftover tasks: https://github.com/matrix-org/synapse/issues/10737 Closes https://github.com/matrix-org/synapse/issues/10737 in the fact that it is not longer necessary to track those things.
* Prefill events after invalidate not before when persisting events (#15758)Jason Little2023-06-141-0/+49
| | | | | Fixes #15757
* Replace `EventContext` fields `prev_group` and `delta_ids` with field ↵Shay2023-06-131-1/+4
| | | | `state_group_deltas` (#15233)
* Clear event caches when we purge history (#15609)Erik Johnston2023-06-081-4/+4
| | | | | | | This should help a little with #13476 --------- Co-authored-by: Patrick Cloke <patrickc@matrix.org>
* `N + 3`: Read from column `full_user_id` rather than `user_id` of tables ↵Shay2023-06-021-12/+5
| | | | `profiles` and `user_filters` (#15649)
* Merge branch 'master' into developOlivier Wilkinson (reivilibre)2023-05-261-0/+1
|\
| * Fix a bug introduced in Synapse v1.84.0 where workers do not start up when ↵reivilibre2023-05-261-0/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | no `instance_map` was provided. (#15672) * Fix #15669: always populate instance map even if it was empty * Fix some tests * Fix more tests * Newsfile Signed-off-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org> * CI fix: don't forget to update apt repository sources before installing olddeps deps * Add test testing the backwards compatibility --------- Signed-off-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org>
* | Process previously failed backfill events in the background (#15585)Eric Eastwood2023-05-241-0/+37
| | | | | | | | | | | | | | | | | | Process previously failed backfill events in the background because they are bound to fail again and we don't need to waste time holding up the request for something that is bound to fail again. Fix https://github.com/matrix-org/synapse/issues/13623 Follow-up to https://github.com/matrix-org/synapse/issues/13621 and https://github.com/matrix-org/synapse/issues/13622 Part of making `/messages` faster: https://github.com/matrix-org/synapse/issues/13356
* | Improve type hints for cached decorator. (#15658)Patrick Cloke2023-05-241-6/+5
| | | | | | | | | | The cached decorators always return a Deferred, which was not properly propagated. It was close enough when wrapping coroutines, but failed if a bare function was wrapped.
* | Add not null constraint to column `full_user_id` of tables `profiles` and ↵Shay2023-05-162-0/+157
|/ | | | `user_filters` (#15537)
* Add column `full_user_id` to tables `profiles` and `user_filters`. (#15458)Shay2023-04-262-16/+8
|
* Modify StoreKeyFetcher to read from server_keys_json. (#15417)Patrick Cloke2023-04-201-8/+8
| | | | | | | | | | | | | | | | | | | | | | | | | | | Before this change: * `PerspectivesKeyFetcher` and `ServerKeyFetcher` write to `server_keys_json`. * `PerspectivesKeyFetcher` also writes to `server_signature_keys`. * `StoreKeyFetcher` reads from `server_signature_keys`. After this change: * `PerspectivesKeyFetcher` and `ServerKeyFetcher` write to `server_keys_json`. * `PerspectivesKeyFetcher` also writes to `server_signature_keys`. * `StoreKeyFetcher` reads from `server_keys_json`. This results in `StoreKeyFetcher` now using the results from `ServerKeyFetcher` in addition to those from `PerspectivesKeyFetcher`, i.e. keys which are directly fetched from a server will now be pulled from the database instead of refetched. An additional minor change is included to avoid creating a `PerspectivesKeyFetcher` (and checking it) if no `trusted_key_servers` are configured. The overall impact of this should be better usage of cached results: * If a server has no trusted key servers configured then it should reduce how often keys are fetched. * if a server's trusted key server does not have a requested server's keys cached then it should reduce how often keys are directly fetched.
* Improve robustness when handling a perspective key response by deduplicating ↵reivilibre2023-04-131-9/+9
| | | | | | | | | | | | | | | received server keys. (#15423) * Change `store_server_verify_keys` to take a `Mapping[(str, str), FKR]` This is because we already can't handle duplicate keys — leads to cardinality violation * Newsfile Signed-off-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org> --------- Signed-off-by: Olivier Wilkinson (reivilibre) <oliverw@matrix.org>
* Revert pruning of old devices (#15360)Erik Johnston2023-03-311-3/+1
| | | | | | | | | * Revert "Fix registering a device on an account with lots of devices (#15348)" This reverts commit f0d8f66eaaacfa75bed65bc5d0c602fbc5339c85. * Revert "Delete stale non-e2e devices for users, take 3 (#15183)" This reverts commit 78cdb72cd6b0e007c314d9fed9f629dfc5b937a6.
* Fix spinloop during partial state sync when a prev event is in backoff (#15351)Sean Quah2023-03-301-4/+9
| | | | | | | | Previously, we would spin in a tight loop until `update_state_for_partial_state_event` stopped raising `FederationPullAttemptBackoffError`s. Replace the spinloop with a wait until the backoff period has expired. Signed-off-by: Sean Quah <seanq@matrix.org>
* Delete stale non-e2e devices for users, take 3 (#15183)Erik Johnston2023-03-291-1/+3
| | | | | This should help reduce the number of devices e.g. simple bots the repeatedly login rack up. We only delete non-e2e devices as they should be safe to delete, whereas if we delete e2e devices for a user we may accidentally break their ability to receive e2e keys for a message.
* Use immutabledict instead of frozendict (#15113)David Robertson2023-03-221-18/+22
| | | | | | | | | | Additionally: * Consistently use `freeze()` in test --------- Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com> Co-authored-by: 6543 <6543@obermui.de>
* Fix ICU tests on alpine / macOS. (#15177)65432023-03-031-0/+2
| | | | The word boundary behaviour is slightly different, consider it acceptable for the tests.
* Fix a long-standing bug where an initial sync would not respond to changes ↵reivilibre2023-02-281-0/+22
| | | | to the list of ignored users if there was an initial sync cached. (#15163)
* Batch up storing state groups when creating new room (#14918)Shay2023-02-242-2/+130
|
* Improve handling of non-ASCII characters in user directory search (#15143)Sean Quah2023-02-241-0/+133
| | | | | | | | | | | | | | | * Fix a long-standing bug where non-ASCII characters in search terms, including accented letters, would not match characters in a different case. * Fix a long-standing bug where search terms using combining accents would not match display names using precomposed accents and vice versa. To fully take effect, the user directory must be rebuilt after this change. Fixes #14630. Signed-off-by: Sean Quah <seanq@matrix.org>
* Bump black from 22.12.0 to 23.1.0 (#15103)dependabot[bot]2023-02-2210-27/+18
|
* Remove unused `room_alias` field from `/createRoom` response (#15093)David Robertson2023-02-223-10/+11
| | | | | | | | | | | | | | * Change `create_room` return type * Don't return room alias from /createRoom * Update other callsites * Fix up mypy complaints It looks like new_room_user_id is None iff new_room_id is None. It's a shame we haven't expressed this in a way that mypy can understand. * Changelog
* Fix a bug introduced in Synapse v1.74.0 where searching with colons when ↵reivilibre2023-02-201-1/+62
| | | | | | using ICU for search term tokenisation would fail with an error. (#15079) Co-authored-by: David Robertson <davidr@element.io>
* Add final type hint to tests.unittest. (#15072)Patrick Cloke2023-02-1410-43/+63
| | | | Adds a return type to HomeServerTestCase.make_homeserver and deal with any variables which are no longer Any.
* Add a class UnpersistedEventContext to allow for the batching up of storing ↵Shay2023-02-092-7/+21
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | state groups (#14675) * add class UnpersistedEventContext * modify create new client event to create unpersistedeventcontexts * persist event contexts after creation * fix tests to persist unpersisted event contexts * cleanup * misc lints + cleanup * changelog + fix comments * lints * fix batch insertion? * reduce redundant calculation * add unpersisted event classes * rework compute_event_context, split into function that returns unpersisted event context and then persists it * use calculate_context_info to create unpersisted event contexts * update typing * $%#^&* * black * fix comments and consolidate classes, use attr.s for class * requested changes * lint * requested changes * requested changes * refactor to be stupidly explicit * clearer renaming and flow * make partial state non-optional * update docstrings --------- Co-authored-by: Erik Johnston <erik@matrix.org>
* Attempt to delete more duplicate rows in receipts_linearized table. (#14915)Patrick Cloke2023-02-011-1/+3
| | | | | | | | | The previous assumption was that the stream_id column was unique (for a room ID, receipt type, user ID tuple), but this turned out to be incorrect. Now find the max stream ID, then map this back to a database-specific row identifier and delete other rows which match the (room ID, receipt type, user ID) tuple, but *not* the row ID.
* Make sqlite database migrations transactional again, part two (#14926)Sean Quah2023-01-311-0/+96
| | | | | | | | | | | | | | | | #14910 fixed the regression introduced by #13873 where sqlite database migrations would no longer run inside a transaction. However, it committed the transaction before Synapse updated its bookkeeping of which migrations have been run, which means that migrations may be run again after they have completed successfully. Leave the transaction open at the end of `executescript`, to restore the old, correct behaviour. Also make the PostgreSQL behaviour consistent with SQLite. Fixes #14909. Signed-off-by: Sean Quah <seanq@matrix.org>
* Fix type hints in typing edu unit tests (#14886)Andrew Morgan2023-01-261-1/+4
|
* Skip processing stats for broken rooms. (#14873)Patrick Cloke2023-01-231-34/+54
| | | | | | | * Skip processing stats for broken rooms. * Newsfragment * Use a custom exception.
* Always notify replication when a stream advances (#14877)Erik Johnston2023-01-201-0/+4
| | | This ensures that all other workers are told about stream updates in a timely manner, without having to remember to manually poke replication.
* Wait for streams to catch up when processing HTTP replication. (#14820)Erik Johnston2023-01-181-11/+9
| | | | This should hopefully mitigate a class of races where data gets out of sync due a HTTP replication request racing with the replication streams.
* Re-enable some linting (#14821)Erik Johnston2023-01-121-3/+3
| | | | | | | * Re-enable some linting * Newsfile * Remove comment
* Allow selecting "prejoin" events by state keys (#14642)David Robertson2022-12-131-622/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Declare new config * Parse new config * Read new config * Don't use trial/our TestCase where it's not needed Before: ``` $ time trial tests/events/test_utils.py > /dev/null real 0m2.277s user 0m2.186s sys 0m0.083s ``` After: ``` $ time trial tests/events/test_utils.py > /dev/null real 0m0.566s user 0m0.508s sys 0m0.056s ``` * Helper to upsert to event fields without exceeding size limits. * Use helper when adding invite/knock state Now that we allow admins to include events in prejoin room state with arbitrary state keys, be a good Matrix citizen and ensure they don't accidentally create an oversized event. * Changelog * Move StateFilter tests should have done this in #14668 * Add extra methods to StateFilter * Use StateFilter * Ensure test file enforces typed defs; alphabetise * Workaround surprising get_current_state_ids * Whoops, fix mypy
* Move `StateFilter` to `synapse.types` (#14668)David Robertson2022-12-121-1/+1
| | | | | * Move `StateFilter` to `synapse.types` * Changelog
* Revert the deletion of stale devices due to performance issues. (#14662)reivilibre2022-12-121-3/+1
|
* Add optional ICU support for user search (#14464)Brendan Abolivier2022-12-121-0/+43
| | | | | | | Fixes #13655 This change uses ICU (International Components for Unicode) to improve boundary detection in user search. This change also adds a new dependency on libicu-dev and pkg-config for the Debian packages, which are available in all supported distros.
* Require types in tests.storage. (#14646)Patrick Cloke2022-12-0933-330/+483
| | | | Adds missing type hints to `tests.storage` package and does not allow untyped definitions.
* Delete stale non-e2e devices for users, take 2 (#14595)Erik Johnston2022-12-091-1/+3
| | | | | This should help reduce the number of devices e.g. simple bots the repeatedly login rack up. We only delete non-e2e devices as they should be safe to delete, whereas if we delete e2e devices for a user we may accidentally break their ability to receive e2e keys for a message.
* Fix a long-standing bug where the user directory would return 1 more row ↵reivilibre2022-12-071-0/+6
| | | | than requested. (#14631)
* Merge branch 'release-v1.73' into developDavid Robertson2022-12-011-3/+1
|\
| * Revert "POC delete stale non-e2e devices for users (#14038)" (#14582)David Robertson2022-11-291-3/+1
| |
* | Aggregate unread notif count query for badge count calculation (#14255)Nick Mills-Barrett2022-11-301-8/+39
|/ | | | | | Fetch the unread notification counts used by the badge counts in push notifications for all rooms at once (instead of fetching them per room).
* POC delete stale non-e2e devices for users (#14038)Erik Johnston2022-11-291-1/+3
| | | | | | | | This should help reduce the number of devices e.g. simple bots the repeatedly login rack up. We only delete non-e2e devices as they should be safe to delete, whereas if we delete e2e devices for a user we may accidentally break their ability to receive e2e keys for a message. Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com> Co-authored-by: Sean Quah <8349537+squahtx@users.noreply.github.com>
* Remove legacy Prometheus metrics names. They were deprecated in Synapse ↵reivilibre2022-11-241-3/+4
| | | | v1.69.0 and disabled by default in Synapse v1.71.0. (#14538)
* Track unconverted device list outbound pokes using a position instead (#14516)Sean Quah2022-11-221-2/+1
| | | | | | | | | | | | | | | | | | | | | When a local device list change is added to `device_lists_changes_in_room`, the `converted_to_destinations` flag is set to `FALSE` and the `_handle_new_device_update_async` background process is started. This background process looks for unconverted rows in `device_lists_changes_in_room`, copies them to `device_lists_outbound_pokes` and updates the flag. To update the `converted_to_destinations` flag, the database performs a `DELETE` and `INSERT` internally, which fragments the table. To avoid this, track unconverted rows using a `(stream ID, room ID)` position instead of the flag. From now on, the `converted_to_destinations` column indicates rows that need converting to outbound pokes, but does not indicate whether the conversion has already taken place. Closes #14037. Signed-off-by: Sean Quah <seanq@matrix.org>
* Reintroduce #14376, with bugfix for monoliths (#14468)David Robertson2022-11-161-10/+152
| | | | | | | | | | | | | | | | | | | | | | * Add tests for StreamIdGenerator * Drive-by: annotate all defs * Revert "Revert "Remove slaved id tracker (#14376)" (#14463)" This reverts commit d63814fd736fed5d3d45ff3af5e6d3bfae50c439, which in turn reverted 36097e88c4da51fce6556a58c49bd675f4cf20ab. This restores the latter. * Fix StreamIdGenerator not handling unpersisted IDs Spotted by @erikjohnston. Closes #14456. * Changelog Co-authored-by: Nick Mills-Barrett <nick@fizzadar.com> Co-authored-by: Erik Johnston <erik@matrix.org>
* Fix background updates failing to add unique indexes on receipts (#14453)Sean Quah2022-11-161-0/+209
| | | | | | | | | | | | | | | | | | | | | | As part of the database migration to support threaded receipts, there is a possible window in between `73/08thread_receipts_non_null.sql.postgres` removing the original unique constraints on `receipts_linearized` and `receipts_graph` and the `reeipts_linearized_unique_index` and `receipts_graph_unique_index` background updates from `72/08thread_receipts.sql` completing where the unique constraints on `receipts_linearized` and `receipts_graph` are missing. Any emulated upserts on these tables must therefore be performed with a lock held, otherwise duplicate rows can end up in the tables when there are concurrent emulated upserts. Fix the missing lock. Note that emulated upserts no longer happen by default on sqlite, since the minimum supported version of sqlite supports native upserts by default now. Finally, clean up any duplicate receipts that may have crept in before trying to create the `receipts_graph_unique_index` and `receipts_linearized_unique_index` unique indexes. Signed-off-by: Sean Quah <seanq@matrix.org>
* Drop support for Postgres 10 in full text search code. (#14397)Patrick Cloke2022-11-091-52/+17
|
* Fix tests for change in PostgreSQL 14 behavior change. (#14310)Patrick Cloke2022-10-271-4/+12
| | | | | | | PostgreSQL 14 changed the behavior of `websearch_to_tsquery` to improve some behaviour. The tests were hitting those edge-cases about handling of hanging double quotes. This fixes the tests to take into account the PostgreSQL version.
* Unified search query syntax using the full-text search capabilities of the ↵James Salter2022-10-251-0/+213
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | underlying DB. (#11635) Support a unified search query syntax which leverages more of the full-text search of each database supported by Synapse. Supports, with the same syntax across Postgresql 11+ and Sqlite: - quoted "search terms" - `AND`, `OR`, `-` (negation) operators - Matching words based on their stem, e.g. searches for "dog" matches documents containing "dogs". This is achieved by - If on postgresql 11+, pass the user input to `websearch_to_tsquery` - If on sqlite, manually parse the query and transform it into the sqlite-specific query syntax. Note that postgresql 10, which is close to end-of-life, falls back to using `phraseto_tsquery`, which only supports a subset of the features. Multiple terms separated by a space are implicitly ANDed. Note that: 1. There is no escaping of full-text syntax that might be supported by the database; e.g. `NOT`, `NEAR`, `*` in sqlite. This runs the risk that people might discover this as accidental functionality and depend on something we don't guarantee. 2. English text is assumed for stemming. To support other languages, either the target language needs to be known at the time of indexing the message (via room metadata, or otherwise), or a separate index for each language supported could be created. Sqlite docs: https://www.sqlite.org/fts3.html#full_text_index_queries Postgres docs: https://www.postgresql.org/docs/11/textsearch-controls.html
* Remove `_get_events_cache` check optimisation from `_have_seen_events_dict` ↵Andrew Morgan2022-10-181-12/+0
| | | | (#14161)
* Support filtering the /messages API by relation type (MSC3874). (#14148)Patrick Cloke2022-10-171-33/+85
| | | Gated behind an experimental configuration flag.
* Stop getting missing `prev_events` after we already know their signature is ↵Eric Eastwood2022-10-151-0/+64
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | invalid (#13816) While https://github.com/matrix-org/synapse/pull/13635 stops us from doing the slow thing after we've already done it once, this PR stops us from doing one of the slow things in the first place. Related to - https://github.com/matrix-org/synapse/issues/13622 - https://github.com/matrix-org/synapse/pull/13635 - https://github.com/matrix-org/synapse/issues/13676 Part of https://github.com/matrix-org/synapse/issues/13356 Follow-up to https://github.com/matrix-org/synapse/pull/13815 which tracks event signature failures. With this PR, we avoid the call to the costly `_get_state_ids_after_missing_prev_event` because the signature failure will count as an attempt before and we filter events based on the backoff before calling `_get_state_ids_after_missing_prev_event` now. For example, this will save us 156s out of the 185s total that this `matrix.org` `/messages` request. If you want to see the full Jaeger trace of this, you can drag and drop this `trace.json` into your own Jaeger, https://gist.github.com/MadLittleMods/4b12d0d0afe88c2f65ffcc907306b761 To explain this exact scenario around `/messages` -> backfill, we call `/backfill` and first check the signatures of the 100 events. We see bad signature for `$luA4l7QHhf_jadH3mI-AyFqho0U2Q-IXXUbGSMq6h6M` and `$zuOn2Rd2vsC7SUia3Hp3r6JSkSFKcc5j3QTTqW_0jDw` (both member events). Then we process the 98 events remaining that have valid signatures but one of the events references `$luA4l7QHhf_jadH3mI-AyFqho0U2Q-IXXUbGSMq6h6M` as a `prev_event`. So we have to do the whole `_get_state_ids_after_missing_prev_event` rigmarole which pulls in those same events which fail again because the signatures are still invalid. - `backfill` - `outgoing-federation-request` `/backfill` - `_check_sigs_and_hash_and_fetch` - `_check_sigs_and_hash_and_fetch_one` for each event received over backfill - ❗ `$luA4l7QHhf_jadH3mI-AyFqho0U2Q-IXXUbGSMq6h6M` fails with `Signature on retrieved event was invalid.`: `unable to verify signature for sender domain xxx: 401: Failed to find any key to satisfy: _FetchKeyRequest(...)` - ❗ `$zuOn2Rd2vsC7SUia3Hp3r6JSkSFKcc5j3QTTqW_0jDw` fails with `Signature on retrieved event was invalid.`: `unable to verify signature for sender domain xxx: 401: Failed to find any key to satisfy: _FetchKeyRequest(...)` - `_process_pulled_events` - `_process_pulled_event` for each validated event - ❗ Event `$Q0iMdqtz3IJYfZQU2Xk2WjB5NDF8Gg8cFSYYyKQgKJ0` references `$luA4l7QHhf_jadH3mI-AyFqho0U2Q-IXXUbGSMq6h6M` as a `prev_event` which is missing so we try to get it - `_get_state_ids_after_missing_prev_event` - `outgoing-federation-request` `/state_ids` - ❗ `get_pdu` for `$luA4l7QHhf_jadH3mI-AyFqho0U2Q-IXXUbGSMq6h6M` which fails the signature check again - ❗ `get_pdu` for `$zuOn2Rd2vsC7SUia3Hp3r6JSkSFKcc5j3QTTqW_0jDw` which fails the signature check
* Accept threaded receipts for events related to the root event. (#14174)Patrick Cloke2022-10-141-0/+111
| | | | | | | | | The root node of a thread (and events related to it) are considered "part of a thread" when validating receipts. This allows clients which show the root node in both the main timeline and the threaded timeline to easily send receipts in either. Note that threaded notifications are not created for these events, these events created notifications on the main timeline.
* Use threaded receipts when fetching events for push. (#13878)Patrick Cloke2022-10-041-18/+39
| | | | Update the HTTP and email pushers to consider threaded read receipts when fetching unread events.
* Recursively fetch the thread for receipts & notifications. (#13824)Patrick Cloke2022-10-041-0/+100
| | | | | | Consider an event to be part of a thread if you can follow a chain of relations up to a thread root. Part of MSC3773 & MSC3771.
* Mark events as read using threaded read receipts from MSC3771. (#13877)Patrick Cloke2022-10-041-1/+188
| | | | Applies the proper logic for unthreaded and threaded receipts to either apply to all events in the room or only events in the same thread, respectively.
* Track notification counts per thread (implement MSC3773). (#13776)Patrick Cloke2022-10-041-2/+167
| | | | | | | | When retrieving counts of notifications segment the results based on the thread ID, but choose whether to return them as individual threads or as a single summed field by letting the client opt-in via a sync flag. The summarization code is also updated to be per thread, instead of per room.
* Update mypy and mypy-zope, attempt 3 (#13993)David Robertson2022-09-301-1/+6
| | | Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
* Revert "Update mypy and mypy-zope (#13925)"David Robertson2022-09-301-6/+0
| | | | This reverts commit 6d543d6d9f56e39199b7e460d0081b02d61f12be.
* Update mypy and mypy-zope (#13925)David Robertson2022-09-301-0/+6
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Update mypy and mypy-zope * Unignore assigning to LogRecord attributes Presumably https://github.com/python/typeshed/pull/8064 makes this ok Cherry-picked from #13521 * Remove unused ignores due to mypy ParamSpec fixes https://github.com/python/mypy/pull/12668 Cherry-picked from #13521 * Remove additional unused ignores * Fix new mypy complaints related to `assertGreater` Presumably due to https://github.com/python/typeshed/pull/8077 * Changelog * Reword changelog Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com> Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
* Fix overflows in /messages backfill calculation (#13936)David Robertson2022-09-301-12/+49
| | | | | | | | * Reproduce bug * Compute `least_function` first * Substitute `least_function` with an f-string * Bugfix: avoid overflow Co-authored-by: Eric Eastwood <erice@element.io>
* Allow admins to require a manual approval process before new accounts can be ↵Brendan Abolivier2022-09-291-1/+101
| | | | used (using MSC3866) (#13556)
* Clarify that a method returns only unthreaded receipts. (#13937)Patrick Cloke2022-09-291-36/+38
| | | | | By renaming it and updating the docstring. Additionally, refactors a method which is used only by tests.
* Limit and filter the number of backfill points to get from the database (#13879)Eric Eastwood2022-09-281-22/+58
| | | | | | | | | There is no need to grab thousands of backfill points when we only need 5 to make the `/backfill` request with. We need to grab a few extra in case the first few aren't visible in the history. Previously, we grabbed thousands of backfill points from the database, then sorted and filtered them in the app. Fetching the 4.6k backfill points for `#matrix:matrix.org` from the database takes ~50ms - ~570ms so it's not like this saves a lot of time 🤷. But it might save us more time now that `get_backfill_points_in_room`/`get_insertion_event_backward_extremities_in_room` are more complicated after https://github.com/matrix-org/synapse/pull/13635 This PR moves the filtering and limiting to the SQL query so we just have less data to work with in the first place. Part of https://github.com/matrix-org/synapse/issues/13356
* Persist CreateRoom events to DB in a batch (#13800)Shay2022-09-281-2/+6
|
* Fix `have_seen_event` cache not being invalidated (#13863)Eric Eastwood2022-09-271-48/+104
| | | | | | | | | | | | | | | | | | | | | Fix https://github.com/matrix-org/synapse/issues/13856 Fix https://github.com/matrix-org/synapse/issues/13865 > Discovered while trying to make Synapse fast enough for [this MSC2716 test for importing many batches](https://github.com/matrix-org/complement/pull/214#discussion_r741678240). As an example, disabling the `have_seen_event` cache saves 10 seconds for each `/messages` request in that MSC2716 Complement test because we're not making as many federation requests for `/state` (speeding up `have_seen_event` itself is related to https://github.com/matrix-org/synapse/issues/13625) > > But this will also make `/messages` faster in general so we can include it in the [faster `/messages` milestone](https://github.com/matrix-org/synapse/milestone/11). > > *-- https://github.com/matrix-org/synapse/issues/13856* ### The problem `_invalidate_caches_for_event` doesn't run in monolith mode which means we never even tried to clear the `have_seen_event` and other caches. And even in worker mode, it only runs on the workers, not the master (AFAICT). Additionally there was bug with the key being wrong so `_invalidate_caches_for_event` never invalidates the `have_seen_event` cache even when it does run. Because we were using the `@cachedList` wrong, it was putting items in the cache under keys like `((room_id, event_id),)` with a `set` in a `set` (ex. `(('!TnCIJPKzdQdUlIyXdQ:test', '$Iu0eqEBN7qcyF1S9B3oNB3I91v2o5YOgRNPwi_78s-k'),)`) and we we're trying to invalidate with just `(room_id, event_id)` which did nothing.
* Improve tests for get_unread_push_actions_for_user_in_range_*. (#13893)Patrick Cloke2022-09-261-16/+72
| | | | | * Adds a docstring. * Reduces a small amount of duplicated code. * Improves tests.
* Only try to backfill event if we haven't tried before recently (#13635)Eric Eastwood2022-09-231-4/+477
| | | | | | | | | | Only try to backfill event if we haven't tried before recently (exponential backoff). No need to keep trying the same backfill point that fails over and over. Fix https://github.com/matrix-org/synapse/issues/13622 Fix https://github.com/matrix-org/synapse/issues/8451 Follow-up to https://github.com/matrix-org/synapse/pull/13589 Part of https://github.com/matrix-org/synapse/issues/13356
* Accept & store thread IDs for receipts (implement MSC3771). (#13782)Patrick Cloke2022-09-232-8/+29
| | | | Updates the `/receipts` endpoint and receipt EDU handler to parse a `thread_id` from the body and insert it in the database.
* Require SQLite >= 3.27.0 (#13760)David Robertson2022-09-091-1/+0
|
* Add timestamp to user's consent (#13741)Dirk Klimpel2022-09-081-7/+26
| | | Co-authored-by: reivilibre <olivier@librepush.net>
* Rename the `EventFormatVersions` enum values so that they line up with room ↵reivilibre2022-09-072-2/+2
| | | | version numbers. (#13706)
* Disable calculating unread counts unless the config flag is enabled. (#13694)Patrick Cloke2022-09-011-22/+20
| | | | | | | | This avoids doing work that will never be used (since the resulting unread counts will never be sent in a /sync response). The negative of doing this is that unread counts will be incorrect when the feature is initially enabled.
* Remove support for unstable private read receipts (#13653)Šimon Brandner2022-09-011-22/+12
| | | Signed-off-by: Šimon Brandner <simon.bra.ag@gmail.com>
* Fix that user cannot `/forget` rooms after the last member has left (#13546)Dirk Klimpel2022-08-301-2/+2
|
* Add forgotten status to Room Details API (#13503)Dirk Klimpel2022-08-171-0/+70
|
* Strengthen tests about deleted old push actions. (#13471)Patrick Cloke2022-08-091-0/+15
|
* Support stable identifiers for MSC2285: private read receipts. (#13273)Šimon Brandner2022-08-051-17/+38
| | | | | This adds support for the stable identifiers of MSC2285 while continuing to support the unstable identifiers behind the configuration flag. These will be removed in a future version.
* Faster room joins: avoid blocking when pulling events with missing prevs ↵Sean Quah2022-07-261-1/+6
| | | | | | | | | (#13355) Avoid blocking on full state in `_resolve_state_at_missing_prevs` and return a new flag indicating whether the resolved state is partial. Thread that flag around so that it makes it into the event context. Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
* Remove old empty/redundant slaved stores. (#13349)Nick Mills-Barrett2022-07-211-0/+261
|
* Make DictionaryCache have better expiry properties (#13292)Erik Johnston2022-07-211-5/+5
|
* Use a real room in the notification rotation tests. (#13260)Patrick Cloke2022-07-151-116/+79
| | | | Instead of manually inserting fake data. This fixes some issues with having to manually calculate stream orderings and other oddities.
* Async get event cache prep (#13242)Nick Mills-Barrett2022-07-152-5/+5
| | | | | Some experimental prep work to enable external event caching based on #9379 & #12955. Doesn't actually move the cache at all, just lays the groundwork for async implemented caches. Signed off by Nick @ Beeper (@Fizzadar)
* Don't pull out state in `compute_event_context` for unconflicted state (#13267)Erik Johnston2022-07-141-55/+0
|
* Fix notification count after a highlighted message (#13223)Erik Johnston2022-07-081-0/+7
| | | | | Fixes #13196 Broke by #13005
* Remove obsolete RoomEventsStoreTestCase (#13200)Petr Vaněk2022-07-071-69/+0
| | | | | | | | | | All tests are prefixed with `STALE_` and therefore they are silently skipped. They were moved to `STALE_` in version `v0.5.0` in commit 2fcce3b3c508 - `Remove stale tests`. Tests from `RoomEventsStoreTestCase` class are not used for last 8 years, I believe the best would be to remove them entirely. Signed-off-by: Petr Vaněk <arkamar@atlas.cz>
* Merge tag 'v1.62.0rc3' into developAndrew Morgan2022-07-041-5/+5
|\ | | | | | | | | | | | | | | | | | | | | Synapse 1.62.0rc3 (2022-07-04) ============================== Bugfixes -------- - Update the version of the [ldap3 plugin](https://github.com/matrix-org/matrix-synapse-ldap3/) included in the `matrixdotorg/synapse` DockerHub images and the Debian packages hosted on `packages.matrix.org` to 0.2.1. This fixes [a bug](https://github.com/matrix-org/matrix-synapse-ldap3/pull/163) with usernames containing uppercase characters. ([\#13156](https://github.com/matrix-org/synapse/issues/13156)) - Fix a bug introduced in Synapse 1.62.0rc1 affecting unread counts for users on small servers. ([\#13168](https://github.com/matrix-org/synapse/issues/13168))
| * Fix stuck notification counts on small servers (#13168)Erik Johnston2022-07-041-5/+5
| |
* | Merge remote-tracking branch 'origin/release-v1.62' into developPatrick Cloke2022-06-301-7/+5
|\|
| * Fix unread counts on large servers (#13140)Erik Johnston2022-06-301-7/+5
| |
* | Implement MSC3827: Filtering of `/publicRooms` by room type (#13031)Šimon Brandner2022-06-291-0/+69
| | | | | | | | Signed-off-by: Šimon Brandner <simon.bra.ag@gmail.com>
* | Improve performance of getting unread counts in rooms (#13119)Erik Johnston2022-06-291-0/+2
|/
* Fix serialization errors when rotating notifications (#13118)Erik Johnston2022-06-281-5/+30
|
* Add type hints to event push actions tests. (#13099)Patrick Cloke2022-06-171-11/+17
|
* Speed up `get_unread_event_push_actions_by_room` (#13005)Erik Johnston2022-06-151-3/+21
| | | | | | | | | | | Fixes #11887 hopefully. The core change here is that `event_push_summary` now holds a summary of counts up until a much more recent point, meaning that the range of rows we need to count in `event_push_actions` is much smaller. This needs two major changes: 1. When we get a receipt we need to recalculate `event_push_summary` rather than just delete it 2. The logic for deleting `event_push_actions` is now divorced from calculating `event_push_summary`. In future it would be good to calculate `event_push_summary` while we persist a new event (it should just be a case of adding one to the relevant rows in `event_push_summary`), as that will further simplify the get counts logic and remove the need for us to periodically update `event_push_summary` in a background job.
* Reduce the amount of state we pull from the DB (#12811)Erik Johnston2022-06-063-13/+21
|
* Wait for lazy join to complete when getting current state (#12872)Erik Johnston2022-06-011-1/+58
|
* Remove remaining bits of groups code. (#12936)Patrick Cloke2022-06-011-1/+0
| | | | | | * Update worker docs to remove group endpoints. * Removes an unused parameter to `ApplicationService`. * Break dependency between media repo and groups. * Avoid copying `m.room.related_groups` state events during room upgrades.
* Rename storage classes (#12913)Erik Johnston2022-05-317-22/+31
|
* LockStore: fix acquiring a lock via `LockStore.try_acquire_lock` (#12832)Sumner Evans2022-05-301-0/+54
| | | Signed-off-by: Sumner Evans <sumner@beeper.com>
* Additional constants for EDU types. (#12884)Patrick Cloke2022-05-271-2/+5
| | | Instead of hard-coding strings in many places.
* Add storage and module API methods to get monthly active users and their ↵Matt C2022-05-271-0/+83
| | | | appservices (#12838)
* Pull out less state when handling gaps mk2 (#12852)Erik Johnston2022-05-261-15/+28
|
* Add some type hints to tests files (#12833)Dirk Klimpel2022-05-232-2/+2
| | | Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>
* Remove code which updates `application_services_state.last_txn` (#12680)Richard van der Hoff2022-05-171-21/+6
| | | | This column is unused as of #12209, so let's stop writing to it.
* Track in memory events using weakrefs (#10533)Erik Johnston2022-05-171-0/+25
|
* Refactor `EventContext` (#12689)Erik Johnston2022-05-101-1/+1
| | | | | | | | | | Refactor how the `EventContext` class works, with the intention of reducing the amount of state we fetch from the DB during event processing. The idea here is to get rid of the cached `current_state_ids` and `prev_state_ids` that live in the `EventContext`, and instead defer straight to the database (and its caching). One change that may have a noticeable effect is that we now no longer prefill the `get_current_state_ids` cache on a state change. However, that query is relatively light, since its just a case of reading a table from the DB (unlike fetching state at an event which is more heavyweight). For deployments with workers this cache isn't even used. Part of #12684
* Stop writing to `event_reference_hashes` (#12679)Richard van der Hoff2022-05-101-9/+0
| | | | | This table is never read, since #11794. We stop writing to it; in future we can drop it altogether.
* Update mypy to 0.950 and fix complaints (#12650)David Robertson2022-05-061-5/+5
|
* Add a consistency check on events read from the database (#12620)Richard van der Hoff2022-05-031-22/+37
| | | | | | | I've seen a few errors which can only plausibly be explained by the calculated event id for an event being different from the ID of the event in the database. It should be cheap to check this, so let's do so and raise an exception.
* Prefer `make_awaitable` over `defer.succeed` in tests (#12505)Sean Quah2022-04-271-5/+4
| | | | | | | | | | | When configuring the return values of mocks, prefer awaitables from `make_awaitable` over `defer.succeed`. `Deferred`s are only awaitable once, so it is inappropriate for a mock to return the same `Deferred` multiple times. Also update `run_in_background` to support functions that return arbitrary awaitables. Signed-off-by: Sean Quah <seanq@element.io>
* Mark remote device list updates as already handled (#12557)Erik Johnston2022-04-261-1/+1
|
* Handle cancellation in `EventsWorkerStore._get_events_from_cache_or_db` (#12529)Sean Quah2022-04-251-2/+119
| | | | | | | | Multiple calls to `EventsWorkerStore._get_events_from_cache_or_db` can reuse the same database fetch, which is initiated by the first call. Ensure that cancelling the first call doesn't cancel the other calls sharing the same database fetch. Signed-off-by: Sean Quah <seanq@element.io>
* Await un-partial-stating after a partial-state join (#12399)Richard van der Hoff2022-04-212-0/+130
| | | | | | When we join a room via the faster-joins mechanism, we end up with "partial state" at some points on the event DAG. Many parts of the codebase need to wait for the full state to load. So, we implement a mechanism to keep track of which events have partial state, and wait for them to be fully-populated.
* Only send out device list updates for our own users (#12465)Erik Johnston2022-04-141-3/+3
| | | Broke in #12365
* Process device list updates asynchronously (#12365)Erik Johnston2022-04-121-20/+27
|
* Optimise `_update_client_ips_batch_txn` to batch together database ↵reivilibre2022-04-081-16/+57
| | | | | operations. (#12252) Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
* Generate historic pagination token for `/messages` when no `?from` token ↵Eric Eastwood2022-04-061-1/+3
| | | | provided (#12370)
* Add type hints to some tests files (#12371)Dirk Klimpel2022-04-053-36/+50
|
* Track device list updates per room. (#12321)Erik Johnston2022-04-041-4/+10
| | | | | | | | | | | | | | This is a first step in dealing with #7721. The idea is basically that rather than calculating the full set of users a device list update needs to be sent to up front, we instead simply record the rooms the user was in at the time of the change. This will allow a few things: 1. we can defer calculating the set of remote servers that need to be poked about the change; and 2. during `/sync` and `/keys/changes` we can avoid also avoid calculating users who share rooms with other users, and instead just look at the rooms that have changed. However, care needs to be taken to correctly handle server downgrades. As such this PR writes to both `device_lists_changes_in_room` and the `device_lists_outbound_pokes` table synchronously. In a future release we can then bump the database schema compat version to `69` and then we can assume that the new `device_lists_changes_in_room` exists and is handled. There is a temporary option to disable writing to `device_lists_outbound_pokes` synchronously, allowing us to test the new code path does work (and by implication upgrading to a future release and downgrading to this one will work correctly). Note: Ideally we'd do the calculation of room to servers on a worker (e.g. the background worker), but currently only master can write to the `device_list_outbound_pokes` table.
* Add type hints for `tests/unittest.py`. (#12347)Richard van der Hoff2022-04-012-4/+5
| | | In particular, add type hints for get_success and friends, which are then helpful in a bunch of places.
* Remove redundant `get_success` calls in test code (#12346)Richard van der Hoff2022-04-013-51/+27
| | | There are a bunch of places we call get_success on an immediate value, which is unnecessary. Let's rip them out, and remove the redundant functionality in get_success and friends.
* Default to `private` room visibility rather than `public` when a client does ↵reivilibre2022-04-011-1/+3
| | | | not specify one, according to spec. (#12350)
* Use a sequence to generate AS transaction IDs, drop `last_txn` AS state (#12209)Nick Mills-Barrett2022-04-011-69/+2
| | | | | | | | Switching to a sequence means there's no need to track `last_txn` on the AS state table to generate new TXN IDs. This also means that there is no longer contention between the AS scheduler and AS handler on updates to the `application_services_state` table, which will prevent serialization errors during the complete AS txn transaction.
* Move single-use methods out of `TestCase` (#12348)Richard van der Hoff2022-04-011-0/+16
| | | | These methods are only used by a single testcase, so they shouldn't be cluttering up the base `TestCase` class.
* Send device list updates to application services (MSC3202) - part 1 (#11881)Andrew Morgan2022-03-301-4/+13
| | | Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
* Start application service stream token tracking from 1 (#12193)Andrew Morgan2022-03-301-2/+2
| | | Co-authored-by: Erik Johnston <erik@matrix.org>
* Refuse to start if DB has an unsafe locale (#12262)Shay2022-03-231-0/+46
|
* Add type hints to tests files. (#12256)Dirk Klimpel2022-03-212-60/+68
|
* Handle cancellation in `DatabasePool.runInteraction()` (#12199)Sean Quah2022-03-161-0/+58
| | | | | | | | | | | | To handle cancellation, we ensure that `after_callback`s and `exception_callback`s are always run, since the transaction will complete on another thread regardless of cancellation. We also wait until everything is done before releasing the `CancelledError`, so that logging contexts won't get used after they have been finished. Signed-off-by: Sean Quah <seanq@element.io>
* Use the ignored_users table to test event visibility & sync. (#12225)Patrick Cloke2022-03-151-0/+17
| | | | | Instead of fetching the raw account data and re-parsing it. The ignored_users table is a denormalised version of the account data for quick searching.
* Add tests for database transaction callbacks (#12198)Sean Quah2022-03-151-1/+103
| | | | Signed-off-by: Sean Quah <seanq@element.io>
* Add config settings for background update parameters (#11980)Shay2022-03-111-16/+237
|
* Support stable identifiers for MSC3440: Threading (#12151)Patrick Cloke2022-03-101-11/+9
| | | | The unstable identifiers are still supported if the experimental configuration flag is enabled. The unstable identifiers will be removed in a future release.
* Remove dead code in `tests/storage/test_database.py` (#12197)Sean Quah2022-03-091-16/+0
| | | Signed-off-by: Sean Quah <seanq@element.io>
* Fix a bug in background updates wherein background updates are never run ↵Shay2022-03-071-2/+2
| | | | using the default batch size (#12157)
* Back out in-flight state caching changes. (#12126)reivilibre2022-03-021-454/+0
|
* Order in-flight state group queries in biggest-first order (#11610)reivilibre2022-03-011-1/+103
| | | Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
* Replace assertEquals and friends with non-deprecated versions. (#12092)Patrick Cloke2022-02-2811-69/+69
|
* Add support for MSC3202: sending one-time key counts and fallback key usage ↵reivilibre2022-02-241-4/+4
| | | | | states to Application Services. (#11617) Co-authored-by: Erik Johnston <erik@matrix.org>
* Fix non-strings in the `event_search` table (#12037)Sean Quah2022-02-241-2/+115
| | | | | | | Don't attempt to add non-string `value`s to `event_search` and add a background update to clear out bad rows from `event_search` when using sqlite. Signed-off-by: Sean Quah <seanq@element.io>
* Remove `HomeServer.get_datastore()` (#12031)Richard van der Hoff2022-02-2334-52/+52
| | | | | | | The presence of this method was confusing, and mostly present for backwards compatibility. Let's get rid of it. Part of #11733
* Cap the number of in-flight requests for state from a single group (#11608)reivilibre2022-02-221-0/+69
|
* Add more tests for in-flight state query duplication. (#12033)reivilibre2022-02-221-21/+171
|
* Track and deduplicate in-flight requests to `_get_state_for_groups`. (#10870)reivilibre2022-02-181-0/+133
| | | Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
* Fix bug in `StateFilter.return_expanded()` and add some tests. (#12016)reivilibre2022-02-181-0/+109
|
* Fix incorrect `get_rooms_for_user` for remote user (#11999)Erik Johnston2022-02-151-0/+107
| | | | | | | When the server leaves a room the `get_rooms_for_user` cache is not correctly invalidated for the remote users in the room. This means that subsequent calls to `get_rooms_for_user` for the remote users would incorrectly include the room (it shouldn't be included because the server no longer knows anything about the room).
* Implement MSC3706: partial state in `/send_join` response (#11967)Richard van der Hoff2022-02-121-4/+4
| | | | | | | | | | | | * Make `get_auth_chain_ids` return a Set It has a set internally, and a set is often useful where it gets used, so let's avoid converting to an intermediate list. * Minor refactors in `on_send_join_request` A little bit of non-functional groundwork * Implement MSC3706: partial state in /send_join response
* Expose the registered device ID from the `register_appservice_user` test ↵reivilibre2022-02-021-1/+3
| | | | helper. (#11615)
* Send to-device messages to application services (#11215)Andrew Morgan2022-02-011-9/+17
| | | Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
* Merge tag 'v1.51.0rc2' into developAndrew Morgan2022-01-241-6/+24
|\ | | | | | | | | | | | | | | | | | | Synapse 1.51.0rc2 (2022-01-24) ============================== Bugfixes -------- - Fix a bug introduced in Synapse 1.40.0 that caused Synapse to fail to process incoming federation traffic after handling a large amount of events in a v1 room. ([\#11806](https://github.com/matrix-org/synapse/issues/11806))
| * Fix logic for dropping old events in fed queue (#11806)Andrew Morgan2022-01-241-6/+24
| | | | | | | | Co-authored-by: Brendan Abolivier <babolivier@matrix.org> Co-authored-by: Richard van der Hoff <richard@matrix.org>
* | Add `state_key` and `rejection_reason` to `events` (#11792)Richard van der Hoff2022-01-211-1/+4
|/ | | | ... and start populating them for new events
* Merge branch 'release-v1.50' into developOlivier Wilkinson (reivilibre)2022-01-141-1/+159
|\
| * Fix a bug introduced in Synapse v1.50.0rc1 whereby outbound federation could ↵reivilibre2022-01-131-1/+111
| | | | | | | | | | fail because too many EDUs were produced for device updates. (#11730) Co-authored-by: David Robertson <davidr@element.io>
| * Fix a bug introduced in Synapse v1.0.0 whereby device list updates would not ↵reivilibre2022-01-121-1/+49
| | | | | | | | | | be sent to remote homeservers if there were too many to send at once. (#11729) Co-authored-by: Brendan Abolivier <babolivier@matrix.org>
* | Replace uses of simple_insert_many with simple_insert_many_values. (#11742)Patrick Cloke2022-01-131-10/+16
|/ | | | This should be (slightly) more efficient and it is simpler to have a single method for inserting multiple values.
* Fix a type annotation in `test_account_data.py` and remove it from the Mypy ↵reivilibre2021-12-291-2/+2
| | | | | exclusion list. (#11657) Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
* Refactor `tests.util.setup_test_homeserver` and ↵reivilibre2021-12-212-2/+3
| | | | `tests.server.setup_test_homeserver`. (#11503)
* Add type hints to event_push_actions. (#11594)Patrick Cloke2021-12-211-5/+7
|
* Use mock from standard library (#11588)V024602021-12-201-9/+8
| | | Instead of the backported version.
* Add type hints to `synapse/storage/databases/main/e2e_room_keys.py` (#11549)Sean Quah2021-12-141-1/+3
|
* Remove redundant `COALESCE()`s around `COUNT()`s in database queries (#11570)Sean Quah2021-12-141-1/+1
| | | | | `COUNT()` never returns `NULL`. A `COUNT(*)` over 0 rows is 0 and a `COUNT(NULL)` is also 0.
* Revert accidental commits to develop.Olivier Wilkinson (reivilibre)2021-12-032-3/+2
|
* Move `tests.utils.setup_test_homeserver` to `tests.server`Olivier Wilkinson (reivilibre)2021-12-032-2/+3
| | | | | | | It had no users. We have just taken the identity of a previous function but don't provide the same behaviour, so we need to fix this in the next commit...
* Clean up tests.storage.test_appservice (#11492)reivilibre2021-12-021-177/+168
|
* Clean up `tests.storage.test_main` to remove use of legacy code. (#11493)reivilibre2021-12-021-17/+10
|
* Add type annotations to `tests.storage.test_appservice`. (#11488)reivilibre2021-12-021-48/+92
|
* Add a license header and comment. (#11479)Patrick Cloke2021-12-011-0/+15
|
* Make background updates controllable via a plugin (#11306)Erik Johnston2021-11-293-16/+97
| | | Co-authored-by: Brendan Abolivier <babolivier@matrix.org>
* Track ongoing event fetches correctly (again) (#11376)Sean Quah2021-11-261-1/+138
| | | | | | | | | | | | | The previous fix for the ongoing event fetches counter (8eec25a1d9d656905db18a2c62a5552e63db2667) was both insufficient and incorrect. When the database is unreachable, `_do_fetch` never gets run and so `_event_fetch_ongoing` is never decremented. The previous fix also moved the `_event_fetch_ongoing` decrement outside of the `_event_fetch_lock` which allowed race conditions to corrupt the counter.
* Improve performance of `remove_{hidden,deleted}_devices_from_device_inbox` ↵Brendan Abolivier2021-11-251-2/+2
| | | | | (#11421) Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
* Lower minumum batch size to 1 for background updates (#11422)Brendan Abolivier2021-11-241-4/+4
| | | Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
* Fix incorrect return value in tests. (#11359)Patrick Cloke2021-11-161-1/+1
|
* Merge tag 'v1.47.0rc3' into developDavid Robertson2021-11-161-1/+51
|\ | | | | | | | | | | | | | | | | | | | | Synapse 1.47.0rc3 (2021-11-16) ============================== Bugfixes -------- - Fix a bug introduced in 1.47.0rc1 which caused worker processes to not halt startup in the presence of outstanding database migrations. ([\#11346](https://github.com/matrix-org/synapse/issues/11346)) - Fix a bug introduced in 1.47.0rc1 which prevented the 'remove deleted devices from `device_inbox` column' background process from running when updating from a recent Synapse version. ([\#11303](https://github.com/matrix-org/synapse/issues/11303), [\#11353](https://github.com/matrix-org/synapse/issues/11353))
| * Run _upgrade_existing_database on workers if at current schema_version (#11346)Andrew Morgan2021-11-151-1/+51
| | | | | | Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
* | Database storage profile passes mypy (#11342)David Robertson2021-11-151-3/+6
| | | | | | | | | | | | It already seems to pass mypy. I wonder what changed, given that it was on the exclusion list. So this commit consists of me ensuring `--disallow-untyped-defs` passes and a minor fixup to a function that returned either `True` or `None`.
* | Change display names/avatar URLs to None if they contain null bytes before ↵Shay2021-11-121-0/+48
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | storing in DB (#11230) * change display names/avatar URLS to None if they contain null bytes * add changelog * add POC test, requested changes * add a saner test and remove old one * update test to verify that display name has been changed to None * make test less fragile
* | Support filtering by relations per MSC3440 (#11236)Patrick Cloke2021-11-091-0/+207
|/ | | | Adds experimental support for `relation_types` and `relation_senders` fields for filters.
* Fix rolling back when using workers (#11255)Erik Johnston2021-11-051-0/+69
| | | Fixes #11252
* Delete messages for hidden devices from `device_inbox` (#11199)Dirk Klimpel2021-11-021-0/+74
|
* Test that `ClientIpStore` combines database and in-memory data correctly ↵Sean Quah2021-11-011-0/+206
| | | | (#11179)
* Delete messages from `device_inbox` table when deleting device (#10969)Dirk Klimpel2021-10-271-0/+90
| | | Fixes: #9346
* Correctly exclude users when making a room public or private (#11075)David Robertson2021-10-151-44/+33
| | | Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
* Stop user directory from failing if it encounters users not in the `users` ↵David Robertson2021-10-133-3/+570
| | | | | | | | | | | | | | | | | | | | | | | table. (#11053) The following scenarios would halt the user directory updater: - user joins room - user leaves room - user present in room which switches from private to public, or vice versa. for two classes of users: - appservice senders - users missing from the user table. If this happened, the user directory would be stuck, unable to make forward progress. Exclude both cases from the user directory, so that we ignore them. Co-authored-by: Eric Eastwood <erice@element.io> Co-authored-by: reivilibre <oliverw@matrix.org> Co-authored-by: Sean Quah <8349537+squahtx@users.noreply.github.com> Co-authored-by: Brendan Abolivier <babolivier@matrix.org>
* Require direct references to configuration variables. (#10985)Patrick Cloke2021-10-062-2/+2
| | | | | | This removes the magic allowing accessing configurable variables directly from the config object. It is now required that a specific configuration class is used (e.g. `config.foo` must be replaced with `config.server.foo`).
* Don't drop user dir deltas when server leaves room (#10982)David Robertson2021-10-066-60/+10
| | | | | | | | | Fix a long-standing bug where a batch of user directory changes would be silently dropped if the server left a room early in the batch. * Pull out `wait_for_background_update` in tests Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com> Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
* Fix potential leak of per-room profiles when the user dir is rebuilt. (#10981)David Robertson2021-10-051-16/+78
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | There are two steps to rebuilding the user directory: 1. a scan over rooms, followed by 2. a scan over local users. The former reads avatars and display names from the `room_memberships` table and therefore contains potentially private avatars and display names. The latter reads from the the `profiles` table which only contains public data; moreover it will overwrite any private profiles that the rooms scan may have written to the user directory. This means that the rebuild could leak private user while the rebuild was in progress, only to later cover up the leaks once the rebuild had completed. This change skips over local users when writing user_directory rows when scanning rooms. Doing so means that it'll take longer for a rebuild to make local users searchable, which is unfortunate. I think a future PR can improve this by swapping the order of the two steps above. (And indeed there's more to do here, e.g. copying from `profiles` without going via Python.) Small tidy-ups while I'm here: * Remove duplicated code from test_initial. This was meant to be pulled into `purge_and_rebuild_user_dir`. * Move `is_public` before updating sharing tables. No functional change; it's still before the first read of `is_public`. * Don't bother creating a set from dict keys. Slightly nicer and makes the code simpler. Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
* Merge tag 'v1.44.0rc3' into developBrendan Abolivier2021-10-041-0/+34
|\ | | | | | | | | | | | | | | | | | | | | Synapse 1.44.0rc3 (2021-10-04) ============================== Bugfixes -------- - Fix a bug introduced in Synapse v1.40.0 where changing a user's display name or avatar in a restricted room would cause an authentication error. ([\#10933](https://github.com/matrix-org/synapse/issues/10933)) - Fix `/admin/whois/{user_id}` endpoint, which was broken in v1.44.0rc1. ([\#10968](https://github.com/matrix-org/synapse/issues/10968))
| * Fix error in `get_user_ip_and_agents` when fetching from the database (#10968)Sean Quah2021-10-011-0/+34
| |
* | Consistently exclude from user_directory (#10960)David Robertson2021-10-041-2/+144
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Introduce `should_include_local_users_in_dir` We exclude three kinds of local users from the user_directory tables. At present we don't consistently exclude all three in the same places. This commit introduces a new function to gather those exclusion conditions together. Because we have to handle local and remote users in different ways, I've made that function only consider the case of remote users. It's the caller's responsibility to make the local versus remote distinction clear and correct. A test fixup is required. The test now hits a path which makes db queries against the users table. The expected rows were missing, because we were using a dummy user that hadn't actually been registered. We also add new test cases to covert the exclusion logic. ---- By my reading this makes these changes: * When an app service user registers or changes their profile, they will _not_ be added to the user directory. (Previously only support and deactivated users were excluded). This is consistent with the logic that rebuilds the user directory. See also [the discussion here](https://github.com/matrix-org/synapse/pull/10914#discussion_r716859548). * When rebuilding the directory, exclude support and disabled users from room sharing tables. Previously only appservice users were excluded. * Exclude all three categories of local users when rebuilding the directory. Previously `_populate_user_directory_process_users` didn't do any exclusion. Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
* | Refactor user directory tests (#10935)David Robertson2021-09-301-4/+188
| | | | | | | | | | | | | | | | | | | | | | | | | | * Pull out GetUserDirectoryTables helper * Don't rebuild the dir in tests that don't need it In #10796 I changed registering a user to add directory entries under. This means we don't have to force a directory regbuild in to tests of the user directory search. * Move test_initial to tests/storage * Add type hints to both test_user_directory files Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>
* | Use direct references for configuration variables (part 6). (#10916)Patrick Cloke2021-09-291-7/+7
|/
* Use direct references for configuration variables (part 4). (#10893)Patrick Cloke2021-09-231-1/+0
|
* Use direct references for some configuration variables (part 3) (#10885)Patrick Cloke2021-09-232-10/+6
| | | | | | | | This avoids the overhead of searching through the various configuration classes by directly referencing the class that the attributes are in. It also improves type hints since mypy can now resolve the types of the configuration variables.
* Treat "\u0000" as "\u0020" for the purposes of message search (message ↵Hillery Shay2021-09-221-0/+74
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | indexing) (#10820) * add test to check if null code points are being inserted * add logic to detect and replace null code points before insertion into db * lints * add license to test * change approach to null substitution * add type hint for SearchEntry * Add changelog entry Signed-off-by: H.Shay <shaysquared@gmail.com> * updated changelog * update chanelog message * remove duplicate changelog * Update synapse/storage/databases/main/events.py remove extra space Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com> * rename and move test file, update tests, delete old test file * fix typo in comments * update _find_highlights_in_postgres to replace null byte with space * replace null byte in sqlite search insertion * beef up and reorganize test for this pr * update changelog * add type hints and update docstring * check db engine directly vs using env variable * refactor tests to be less repetetive * move rplace logic into seperate function * requested changes * Fix typo. * Update synapse/storage/databases/main/search.py Co-authored-by: reivilibre <olivier@librepush.net> * Update changelog.d/10820.misc Co-authored-by: Aaron Raimist <aaron@raim.ist> Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com> Co-authored-by: reivilibre <olivier@librepush.net> Co-authored-by: Aaron Raimist <aaron@raim.ist>
* Make StateFilter frozen so we can hash it (#10816)reivilibre2021-09-141-17/+29
| | | Also enables Mypy for related tests.
* Skip handling of push actions for outlier events (#10780)Erik Johnston2021-09-081-0/+1
| | | | Outlier events don't ever have push actions associated with them, so we can skip some expensive queries during event persistence.
* Populate `rooms.creator` field for easy lookup (#10697)Eric Eastwood2021-09-011-0/+98
| | | | | | Part of https://github.com/matrix-org/synapse/pull/10566 - Fill in creator whenever we insert into the rooms table - Add background update to backfill any missing creator values
* Flatten the synapse.rest.client package (#10600)reivilibre2021-08-177-7/+7
|
* Improve event caching code (#10119)Erik Johnston2021-08-041-0/+50
| | | | Ensure we only load an event from the DB once when the same event is requested multiple times at once.
* Fix codestyle CI from #10440 (#10511)reivilibre2021-08-021-1/+1
| | | Co-authored-by: Erik Johnston <erik@matrix.org>
generated by cgit-pink 1.4.1 (git 2.36.1) at 2024-11-20 17:32:54 +0000